273 lines
16 KiB
Markdown
273 lines
16 KiB
Markdown
# The world pipeline: from a painted map to ground in Unreal
|
||
|
||
**Read this when you want to know what order things happen in.** It is the orientation document for
|
||
everything between "an author paints a world map" and "a player stands on it", and it owns no decisions of its
|
||
own — every one of them is argued somewhere else and linked from here.
|
||
|
||
| If you want | Read |
|
||
| --- | --- |
|
||
| Why the generator exists and how it works | [`Terrain.md`](Terrain.md) — the specification and the decision record |
|
||
| What is being worked on right now, what looks wrong | [`Terrain-Next.md`](Terrain-Next.md) — the working brief |
|
||
| Every manifest key, explained | `RawContent/World/README.md` and `Scripts/Authoring/region_manifest.py` |
|
||
| How to paint a world | `RawContent/World/Templates/README.md` |
|
||
| Why a thing is the way it is | [`Decisions.md`](Decisions.md) — D-47 onwards is all world |
|
||
|
||
All of this is **off the ladder** (D-47). No gameplay code reaches into it and nothing here blocks a step in
|
||
[`Steps.md`](Steps.md).
|
||
|
||
---
|
||
|
||
## The shape of it
|
||
|
||
```
|
||
RawContent/World/Templates/Map5.png + Map5.legend.json
|
||
a painted flat cylindrical world map what each colour means: uplift mm/yr, erodibility
|
||
|
|
||
| terrain studio paint it, 127.0.0.1:8099
|
||
| terrain plan 4 s: cut the planet into regions, solve nothing
|
||
v
|
||
+----------------+----------------+
|
||
| |
|
||
terrain bake Tools/Orogen (the browser twin)
|
||
~2 h, 8 m geology grid ~12 s, 204K-region sphere mesh
|
||
| |
|
||
v v
|
||
RawContent/World/Bake_NNN/ "Export Map"
|
||
planet_height.png RawContent/World/Orogen Gens/
|
||
map_flow / class / uplift ... orogen-heightmap-7945.png (8192 x 4096)
|
||
overlay.json, meta.json orogen-colormap / satellite / climate ...
|
||
| |
|
||
| +------------+-------------+
|
||
| | |
|
||
terrain tiles generate_region_tiles.py Tools/MapArt (Go)
|
||
5 km tiles at 2 m cuts the window into tiles `biomes` -> biome masks
|
||
| ^ `build` -> 4 map layers
|
||
| | |
|
||
| | |
|
||
X NOT WIRED v v
|
||
(this is the RawContent/World/ RawContent/World/MapArt/
|
||
future; see RegionTiles/*.png map_relief / colour / satellite / climate
|
||
"What is not | |
|
||
wired yet") create_region_world.py create_world_map.py
|
||
via build_region.sh via build_world_map.sh
|
||
| |
|
||
v v
|
||
Content/Maps/L_World Content/World/Maps/
|
||
98 landscapes DA_WorldMap_L_World + 4 textures
|
||
71.40 x 35.70 km |
|
||
\ /
|
||
\________________________/
|
||
|
|
||
the map view, in game and in the editor
|
||
```
|
||
|
||
Two generators sit side by side and they are **not** interchangeable. The Go tool in `Tools/Terrain/` is the
|
||
one that makes ground a player can stand on: an 8 m geology grid, a coastal pass, faults, craters, detail
|
||
tiles. World Orogen in `Tools/Orogen/` is a browser twin that reads *the same painting and the same legend* and
|
||
solves them on a sphere mesh in about twelve seconds — for looking at a painting's rivers on a globe and tuning
|
||
the legend's numbers before committing two hours to a bake. A 44 km cell cannot show a fault or a crater, and
|
||
it never will (D-66).
|
||
|
||
**The ground in `L_World` today came from Orogen, not from the Go tool.** That is a deliberate shortcut: it
|
||
put 900 km² of ground in the engine long before the generator's detail passes reach Unreal, and the relief you
|
||
see is therefore *art* rather than a solve. `terrain bake` makes that same continent a 116 m plain; the Orogen
|
||
export makes it 2972 m (D-69).
|
||
|
||
---
|
||
|
||
## The three routes
|
||
|
||
Only one of these is live. Knowing which is which saves an hour of confusion.
|
||
|
||
| Route | Status | What it builds | How the tiles are made |
|
||
| --- | --- | --- | --- |
|
||
| **A. Orogen → Python cutter → Unreal** | **live; this is what `L_World` is** | `Content/Maps/L_World` | `generate_region_tiles.py` cuts a window out of a whole-planet PNG. `Region.json`'s `source.kind` is `planet_map` and its `metres_per_pixel` is **a number somebody chose** |
|
||
| **B. Orogen → direct tile export → Unreal** | built, not the one in use (D-71) | the same level | Orogen's **Unreal Landscape…** export renders the tiles itself and writes `Region.generated.json`. `source.kind` is `orogen_render`, and the scale is a *consequence* rather than a guess |
|
||
| **C. `terrain bake` → `terrain tiles` → Unreal** | **not wired** | the same level, eventually | This is the point of the whole generator. `terrain tiles` already writes 5 km tiles at 2 m; nothing carries them into Unreal yet |
|
||
|
||
Route B is strictly better than A on the one thing that matters most — the scale stops being invented — and the
|
||
only reason A is what is in the level is that A came first. Route C is where this is going, and when it lands,
|
||
`generate_region_tiles.py` is the only thing that changes.
|
||
|
||
---
|
||
|
||
## Route A, step by step
|
||
|
||
This is the sequence that produced what is in the project now.
|
||
|
||
### 1. Paint the world
|
||
|
||
```bash
|
||
cd Tools/Terrain && go build -o bin/terrain.exe ./cmd/terrain
|
||
Tools/Terrain/bin/terrain.exe studio # 127.0.0.1:8099
|
||
```
|
||
|
||
The brushes *are* the legend's classes. The panel prints the hillslope angle each uplift rate buys as you type
|
||
it — read the **`typical`** column, not `divide`: almost none of a map is divide, and reading the divide angle
|
||
as the landscape is how a legend gets set two or three times too hot (D-59). `o` switches to the overlay sheet,
|
||
`ctrl+z` undoes a stroke, and **Re-roll** changes the seed, which moves the massifs, the rock, the faults and
|
||
the coastline detail without touching a painted pixel (D-58).
|
||
|
||
**Paint the uplift, never the height.** A solve handed a painted surface erodes it into something else and
|
||
throws the drainage network away, which is the reason the generator exists at all.
|
||
|
||
### 2. Check the painting before committing to a bake
|
||
|
||
```bash
|
||
Tools/Terrain/bin/terrain.exe plan # four seconds, solves nothing
|
||
```
|
||
|
||
`Plan/map_class.png` and `Plan/map_regions.png` are the two pictures that decide whether a bake is worth
|
||
starting. `plan.json` carries the class table.
|
||
|
||
### 3. Solve it — in the browser first
|
||
|
||
Serve the repo and open the import page:
|
||
|
||
```bash
|
||
npx serve Tools/Orogen # any static server
|
||
# then open /import, choose the Painted Map source, and Load from studio
|
||
```
|
||
|
||
**Load from studio** pulls the painting, the legend and `Planet.json` straight out of the running `terrain
|
||
studio` in about four seconds, strokes included (D-67). The studio answers `GET` and `HEAD` only and no
|
||
preflight, so no browser tab can ever paint, save, plan or bake.
|
||
|
||
Twelve seconds later there is a globe with rivers on it. Tune the legend's numbers in the table here, not after
|
||
a two-hour bake.
|
||
|
||
### 4. Export the planet
|
||
|
||
Press **Export Map**. That writes the whole-planet PNGs into `RawContent/World/Orogen Gens/` — the heightmap
|
||
(8192 × 4096, a fixed −5000…6000 m ramp), plus colormap, satellite, climate, landmask and the painted-layer
|
||
debug maps.
|
||
|
||
> **Do not use "Export All" for the heightmap.** Its list is
|
||
> `{biome: Satellite, koppen: Climate, landheightmap: Heightmap, landmask: Land Mask}` — the entry *labelled*
|
||
> "Heightmap" is the **land** variant, whose `landHeightmapColor` returns black for `elevation <= 0`, so every
|
||
> ocean pixel is 0 m. Import that and the sea floor sits flat at exactly sea level: z-fighting with the sea
|
||
> plane across 64 % of the world, no shelf and no shore. The file `Region.json` reads is the *absolute*
|
||
> heightmap, −5 km to +6 km, which has to be exported as a single layer. Both are 8192 x 4096 greyscale and
|
||
> neither says which it is, so the tell is the filename: `orogen-heightmap-*.png` against
|
||
> `orogen-land-heightmap-*.png`.
|
||
>
|
||
> Orogen also numbers each **export**, not each planet. `orogen-colormap-14733759.png` and
|
||
> `orogen-heightmap-7945.png` look like two different worlds and are one. Nothing inside a PNG says which
|
||
> planet it is, so if you are ever unsure, `cd Tools/MapArt && go run . check` measures land/sea agreement
|
||
> against the heightmap — 97.5 % is the same planet, 50 % is not.
|
||
|
||
### 5. Render the biome masks
|
||
|
||
The painting's classes and the Köppen climate become one blurred greyscale mask per biome, which is what the
|
||
paint layers are built from. Go, because the painting is 29 megapixels of RGB and the engine's Python cannot
|
||
decode it; blurred once globally so a tile can read it without carrying a margin the width of the blend.
|
||
|
||
```bash
|
||
cd Tools/MapArt && go run . biomes # a few seconds, into RawContent/World/Biomes/
|
||
```
|
||
|
||
It prints how far the worst pixel was from any legend colour. On the painting that is **0.0** — a painted map
|
||
is made of its legend's own colours. A number much above that means the image is a *render* of a
|
||
classification rather than the classification itself, which is the difference between data and a picture of
|
||
data, and is exactly why Orogen's own class export is not the source here.
|
||
|
||
### 6. Cut the window into landscape tiles
|
||
|
||
`RawContent/World/Region.json` is the contract: which level, how many tiles, what a heightmap value means in
|
||
metres, and — the number that matters most — `source.metres_per_pixel`, because the export carries no scale of
|
||
its own.
|
||
|
||
```bash
|
||
# measure a window and print what it holds, writing nothing. Try scales here, not by rebuilding.
|
||
D:/UE_5.8/.../python.exe Scripts/Authoring/generate_region_tiles.py --scout
|
||
D:/UE_5.8/.../python.exe Scripts/Authoring/generate_region_tiles.py # ~2 min, 208 MB, untracked
|
||
```
|
||
|
||
### 7. Build the level
|
||
|
||
```bash
|
||
Scripts/Authoring/build_region.sh # the whole grid, a few tiles per process
|
||
Scripts/Authoring/build_region.sh --append # add whatever is still missing
|
||
```
|
||
|
||
One process per batch, because a landscape of a hundred components costs about a gigabyte the editor never
|
||
gives back: thirty-six tiles in one process reached 14.7 GB by the ninth (D-69).
|
||
|
||
### 8. Build the map view
|
||
|
||
```bash
|
||
Scripts/Authoring/build_world_map.sh # renders the art, imports it, writes the definition
|
||
```
|
||
|
||
Independent of the level — it reads the same `Orogen Gens/` images and `Region.json`, and writes
|
||
`Content/World/Maps/`. See [`Spec/UI.md`](Spec/UI.md) for the map itself; `M` opens it in game, **Window → World
|
||
Map** in the editor.
|
||
|
||
---
|
||
|
||
## What each artefact is, and whether it is tracked
|
||
|
||
| Path | What | In git? |
|
||
| --- | --- | --- |
|
||
| `RawContent/World/Templates/` | The painting, its legend, its plates. **The real source.** | tracked |
|
||
| `RawContent/World/Planet.json` | The planet's manifest: circumference, cell size, pipeline constants | tracked |
|
||
| `RawContent/World/Orogen Gens/` | Orogen's whole-planet exports | tracked — they came out of a browser session and cannot be regenerated headlessly |
|
||
| `RawContent/World/Plan/` | `terrain plan` output | ignored, 4 s to rebuild |
|
||
| `RawContent/World/Bake_NNN/` | `terrain bake` output: `planet_height.png`, the data maps, `overlay.json`, `meta.json` | ignored, gigabytes, ~2 h to rebuild |
|
||
| `RawContent/World/RegionTiles/` | 98 tile sets: height + 3 weightmaps each | ignored, 208 MB, 2 min to rebuild |
|
||
| `RawContent/World/MapArt/` | `layers.json` (tracked) and the rendered map layers (ignored, 6 s) | mixed |
|
||
| `Content/Maps/L_World` | The level: 98 landscapes and ~500 external actor packages | tracked, through LFS |
|
||
| `Content/World/Maps/` | `DA_WorldMap_L_World` and four map textures | tracked, through LFS |
|
||
|
||
The rule is: **an input is tracked, a product is not.** The one exception is `Orogen Gens/`, because a browser
|
||
session is not something a script can redo.
|
||
|
||
---
|
||
|
||
## Traps
|
||
|
||
Every one of these has cost real time.
|
||
|
||
- **Never run a level build while the editor holds that level.** `--rebuild` empties the level *first* and
|
||
saves *last*, so a lock detected late is indistinguishable from data loss — that is exactly what happened on
|
||
2026-09-20, leaving twelve of ninety-eight tiles, all of them the polar ocean strip, so the level opened on
|
||
71 km of sea and read as corrupted. Both `build_region.sh` and `create_region_world.py` now probe the file
|
||
before anything is destroyed (D-71a).
|
||
- **Run the long ones detached.** `create_world.py` and `terrain bake` must never go under a tool timeout. A
|
||
killed bake is two hours.
|
||
- **A script's arguments go *inside* the quoted `-script=` value.** Anything after it is parsed by the engine
|
||
and silently never reaches Python — which looks exactly like a script that ignored its arguments.
|
||
- **`create_world.py` empties whatever level it is handed.** This is why the numpy canvas is named
|
||
`L_Canvas_Proto` and not `L_World`: a manifest still pointing at the latter would replace 98 landscapes with
|
||
a 14 km square on one run, with no prompt (D-72).
|
||
- **Tile files are named after the level.** Change `Region.json`'s `level` and all 98 tile sets look missing;
|
||
rename the PNGs or 208 MB regenerates.
|
||
- **The heights out of Orogen are art.** Fixed −5000…6000 ramp, land normalised to a browser preview's peak
|
||
setting. Only `sea_scale` corrects any of it, and only the sea.
|
||
- **A freshly opened `L_World` can look empty.** The landscapes are split into world-partition streaming
|
||
proxies and none is loaded; what you see is the sky, the fog and the sea plane at Z 0, which from above looks
|
||
convincingly like soft terrain. Load a region, or build HLODs.
|
||
- **Do not trace for the ground in a commandlet.** Landscape collision is not reliably there, the sea plane's
|
||
is, and a trace that hits the sea returns `0.0` rather than failing. Read the heightmap instead.
|
||
|
||
---
|
||
|
||
## What is not wired yet
|
||
|
||
Stated plainly so nobody goes looking for it.
|
||
|
||
- **Route C.** `terrain tiles` writes 5 km tiles of 2500 samples at 2 m with hillshade, flow, wear and deposit
|
||
maps beside each one, and **nothing carries them into Unreal**. This is the path that ends with ground worth
|
||
standing on; `generate_region_tiles.py` is the piece that changes.
|
||
- **The overlay in Unreal.** `terrain overlay` proposes woodland, settlements and roads, the studio has a
|
||
**Generate marks** button, and `overlay.json` carries every feature in world metres. `Region.json` reserves an
|
||
`overlay` block and `region_manifest.py` has the slot for a per-tile mark map — **neither is implemented**. No
|
||
forest, road or settlement is placed from one. It is left unbuilt rather than written blind because a carry
|
||
that has never carried anything is a guess about a file format.
|
||
- **Overlay marks on the map view.** They are in whole-cylinder normalised coordinates, exactly like the map
|
||
art, so they transfer by the same `u,v`. This is the cheapest useful thing left in the list.
|
||
- **Erosion-derived paint layers in the region.** Route A has no erosion pass and therefore no wear, flow or
|
||
deposit map, which is why its paint rules are slope and altitude alone. `L_Canvas_Proto` — the legacy numpy
|
||
pipeline — is still the only path that carries those maps into Unreal, and that is the only reason it is kept.
|
||
- **The coastal detail pass**, which the shelf and shore platform have just unblocked. See `Terrain-Next.md`.
|