Add Ground Zero terrain pipeline and playable assets

This commit is contained in:
2026-05-14 00:09:40 -07:00
parent 46d0e080b5
commit 6d25ff690d
77 changed files with 5770 additions and 84 deletions
+88
View File
@@ -0,0 +1,88 @@
# Ground Zero DEM Acquisition
## Purpose
This document records the official USGS 3DEP DEM product selected for the Ground
Zero MVP tile.
## Acquisition Script
```text
Scripts/acquire_ground_zero_dem.py
```
The script:
- Reads the Ground Zero tile bounds from `Data/Tiles/ground_zero_tiles.json`.
- Converts the 1 km UTM tile bounds to latitude/longitude.
- Queries the USGS TNMAccess API for `Digital Elevation Model (DEM) 1 meter`
products in GeoTIFF format.
- Stores the full TNM query response and selected product record.
- Downloads the selected GeoTIFF source.
- Updates the tile registry source record.
## Selected Product
```text
title: USGS 1 Meter 10 x54y416 CA_CaliforniaGaps_B23
dataset: Digital Elevation Model (DEM) 1 meter
format: GeoTIFF
publication_date: 2025-08-20
source: USGS TNMAccess / ScienceBase / 3DEP
```
The product is a 10 km x 10 km 1-meter DEM tile that covers the 1 km Ground Zero
MVP tile.
## Local Files
```text
Data/Terrain/Sources/gz_us_ca_pacifica_utm10n_e544_n4160/
```
Expected files:
- `gz_us_ca_pacifica_utm10n_e544_n4160_tnm_1m_dem_product.json`
- `USGS_1M_10_x54y416_CA_CaliforniaGaps_B23.tif`
- `USGS_1M_10_x54y417_CA_CaliforniaGaps_B23.tif`
Ground Zero sits on a 3DEP 10 km tile boundary, so both source GeoTIFFs are
needed to extract the full 1 km MVP tile.
## Subset Extraction
```text
Scripts/extract_ground_zero_dem_subset.py
```
This script extracts the exact 1 km Ground Zero subset from the source GeoTIFFs.
It requires `rasterio` or equivalent GDAL Python support.
The extraction output is:
```text
Data/Terrain/Extracted/gz_us_ca_pacifica_utm10n_e544_n4160/
```
Generated files:
- `gz_us_ca_pacifica_utm10n_e544_n4160_1m_dem_subset.tif`
- `gz_us_ca_pacifica_utm10n_e544_n4160_1m_dem_subset_metadata.json`
Current extracted subset:
```text
size: 1000 x 1000 pixels
pixel_size: 1 m x 1 m
crs: EPSG:26910 / NAD83 UTM zone 10N
bounds: E 544000-545000, N 4160000-4161000
```
## Notes
- The USGS product is the final MVP target elevation source.
- The older USGS EPQS 33 x 33 heightmap remains useful as a fast prototype and
sanity check, but the 1-meter GeoTIFF is the source we should use for the real
Unreal terrain tile.
- The selected products are larger than the 1 km MVP tile; this is expected for
3DEP 1-meter DEM products.
+91
View File
@@ -0,0 +1,91 @@
# Ground Zero MVP Tile
## Decision
Ground Zero for the first real-terrain MVP tile is the Linda Mar / San Pedro
Valley edge in Pacifica, California.
This is a practical first tile because it gives the terrain pipeline varied
coastal California terrain in a compact 1 km square:
- Pacific coastal influence without starting in a dense city center.
- Hills, valley slope, drainage, and vegetation transitions.
- Nearby ocean/coastline/bathymetry cases for early pipeline design.
- Real public terrain, land-cover, hydrography, and coastline datasets are
available from US sources.
- The environment supports the first survival loop: wood, fiber, water, shelter,
weather exposure, and wildlife placeholders.
## Tile Identity
The MVP tile uses a simple UTM 1 km grid ID for the first prototype. This is not
the final global indexing decision; it is the concrete local key we can build
against now.
```text
tile_id: gz_us_ca_pacifica_utm10n_e544_n4160
display_name: Ground Zero - Pacifica Linda Mar / San Pedro Valley
country: United States
region: California
nearest_place: Pacifica
utm_zone: 10N
utm_easting_min_m: 544000
utm_northing_min_m: 4160000
utm_easting_max_m: 545000
utm_northing_max_m: 4161000
nominal_center_latitude: 37.5925
nominal_center_longitude: -122.4995
tile_size_meters: 1000
```
## MVP Scope
For the first pass, this tile should prove the pipeline rather than final art
quality.
- Import real elevation data into a 1 km x 1 km Unreal terrain.
- Preserve real-world horizontal scale.
- Use a conservative vertical scale that can represent hills accurately without
making traversal unusable.
- Generate a tile metadata record and store it in the registry.
- Track source datasets and generation version.
- Add at least the eight neighboring tile records as placeholders.
- Validate that edges can be stitched to adjacent 1 km tiles.
- Infer first-pass biome and resources from terrain, land cover, water, and
coastal influence.
## Initial Biome Direction
Working biome label:
```text
coastal_california_scrub_woodland
```
Expected early resources:
- Wood from scrub/woodland patches.
- Fiber from grasses, reeds, and coastal scrub.
- Water from drainage/creek source where present in source data.
- Stone from slope and exposed terrain placeholders.
- Wildlife placeholders appropriate to coastal California foothills.
## Source Data Targets
First-pass source candidates:
- USGS 3DEP elevation data for terrain height.
- USGS National Hydrography Dataset or equivalent for streams/water.
- NOAA coastline and bathymetry datasets for coastal/ocean handling.
- USGS National Land Cover Database or equivalent for land-cover/biome hints.
- OpenStreetMap only as a secondary reference for roads/trails/landmarks.
## Open Decisions
- Whether the final global tile grid stays UTM-based by zone or moves to a
single global equal-area grid.
- Exact Unreal origin strategy for tiles outside the MVP area.
- How much coastline/ocean data belongs in the same tile package versus a
separate water/bathymetry layer.
- Whether player-built state is stored per terrain tile, per settlement region,
or in a separate spatial entity table.
+172
View File
@@ -0,0 +1,172 @@
# MVP Terrain Accuracy And Source Requirements
## Purpose
This document defines the terrain accuracy bar for the Ground Zero MVP tile and
the source data requirements for moving from the current point-sampled prototype
to a real Unreal terrain import pipeline.
The goal is practical accuracy, not survey-grade simulation. Terrain should be
recognizably based on the real place, preserve the correct horizontal scale, and
support believable traversal, water, biome, and resource placement.
## MVP Tile
```text
tile_id: gz_us_ca_pacifica_utm10n_e544_n4160
location: Linda Mar / San Pedro Valley edge, Pacifica, California
tile_size: 1 km x 1 km
prototype_grid: WGS84 / UTM zone 10N
```
## Accuracy Tiers
### Tier 0 - Prototype Complete
This is the current state.
- Point-sampled USGS elevation data is available for the tile.
- A normalized `.r16` heightmap exists.
- Source and generation metadata are recorded.
- Tile registry status can move to `generated`.
Tier 0 is enough to prove the automation path, but not enough for final MVP
terrain art or gameplay.
### Tier 1 - MVP Required
This is the minimum bar for the first playable real-terrain MVP tile.
- Horizontal scale must remain 1 Unreal kilometer per real 1 kilometer.
- Ground elevation source should be 1 meter where available.
- If 1 meter source is unavailable, 1/3 arc-second or better can be used
temporarily, but the tile must be flagged as lower-confidence.
- Terrain vertical values must preserve real elevation range within a documented
tolerance after import.
- The generated Unreal landscape must include metadata for source dataset,
source date/version, vertical datum, horizontal datum/projection, min/max
elevation, and generation version.
- Terrain should not have visible seams at tile edges when adjacent tiles use
the same source and generation version.
- Slopes should be smoothed only enough for playable traversal; smoothing must
not erase major hills, valleys, drainage cuts, or coastal features.
- Water features present in source data should be represented at least as
gameplay placeholders.
- Biome/resource placement should use source-derived hints, not purely manual
placement.
### Tier 2 - Preferred After MVP
- Use lidar-derived source DEM or point-cloud processing where available.
- Preserve drainage, road/trail cuts, ridgelines, and terrain breaks with higher
fidelity.
- Build separate masks for slope, wetness, vegetation class, water, coastline,
and traversal difficulty.
- Add automated QA maps for slope extremes, seam mismatch, missing samples, and
source confidence.
- Generate World Partition-ready landscape tiles directly from the pipeline.
## Final Ground Elevation Source Requirements
Primary source:
- USGS 3DEP 1-meter DEM or USGS Seamless 1-Meter DEM where available.
Preferred fallback order:
1. USGS 3DEP 1-meter DEM / Seamless 1-Meter DEM.
2. USGS 3DEP source DEM or lidar-derived DEM with better local coverage.
3. USGS 3DEP 1/3 arc-second DEM.
4. USGS Elevation Point Query Service only for quick validation and prototypes.
The point-query service should not be the final production importer because it
is inefficient for dense raster generation and does not provide the full source
surface metadata we need for repeatable tiles.
## Coastal And Bathymetry Requirements
The Ground Zero region is coastal, so the MVP terrain requirements must account
for ocean/coastal data even if the first Ground Zero tile is mostly land.
Minimum coastal requirements:
- Determine whether each tile contains ocean, shoreline, beach, creek mouth, or
wetland features.
- If a tile includes ocean or nearshore water, use NOAA/NCEI coastal DEM,
bathymetry, or Coastal Relief Model data where appropriate.
- Track vertical datum and unit differences between land and bathymetric sources.
- Do not use bathymetric products for navigation.
- Represent ocean depth as gameplay terrain/water depth data, not as a real
nautical chart.
Preferred source order:
1. NOAA/NCEI coastal DEM or CUDEM where available for the specific coast.
2. NOAA/NCEI Coastal Relief Model for broader topographic/bathymetric context.
3. NOAA/NCEI bathymetry data viewer products for source discovery.
4. Coarser global relief only as placeholder context, not final MVP tile data.
## Import Requirements For Unreal
The terrain pipeline should produce:
- Raw heightmap in Unreal-compatible format.
- Metadata JSON.
- Tile registry update.
- Optional preview CSV or GeoJSON for debugging.
- Source confidence report.
Required metadata:
- `tile_id`
- source dataset name
- source URI or stable identifier
- source license/usage note
- source acquisition/publication date if available
- source resolution
- horizontal datum/projection
- vertical datum
- min elevation
- max elevation
- normalized heightmap format
- Unreal landscape scale assumptions
- generation version
- pipeline version
## Acceptance Tests
Before a terrain tile can be considered MVP-ready:
- [ ] Source DEM/raster covers the full 1 km tile.
- [ ] Generated tile is exactly 1000 m x 1000 m in projected coordinates.
- [ ] Heightmap resolution and Unreal landscape scale are documented.
- [ ] Min/max elevation in metadata matches generated heightmap scaling.
- [ ] Tile can be regenerated and produces the same content hash from the same
source inputs.
- [ ] Tile registry status moves through `source_data_found`, `generated`, and
`validated` intentionally.
- [ ] Adjacent-edge sample rows/columns can be compared for seam validation.
- [ ] Water/coastline handling is documented for the tile.
- [ ] Biome/resource hints are recorded.
## Current Prototype Gap List
The current `33 x 33` USGS point-sampled heightmap is useful, but it must be
replaced or supplemented before MVP terrain lock.
Known gaps:
- It samples point elevations instead of downloading a complete DEM raster.
- It has coarse 31.25 m sample spacing.
- It does not yet preserve source vertical datum metadata.
- It does not include land-cover, hydrography, coastline, or bathymetry layers.
- It is not yet imported into an Unreal Landscape actor.
- It is not yet validated against neighboring tile seams.
## Decision
For the Ground Zero MVP tile, the target terrain source is USGS 3DEP 1-meter DEM
or the USGS Seamless 1-Meter DEM if it is available for the tile. NOAA/NCEI
coastal DEM or Coastal Relief Model data should be used for coastal and
bathymetric context where the MVP tile or its immediate neighbors intersect
shoreline/ocean features.
+73
View File
@@ -0,0 +1,73 @@
# Ground Zero Terrain Import Prototype
## Purpose
This prototype proves that the selected Ground Zero tile can be sampled from
real elevation data and converted into files Unreal can import or inspect.
It is not the final terrain pipeline. The final pipeline should use raster DEM
or lidar-derived terrain data at higher resolution. This first pass is useful
because it validates tile coordinates, source access, height normalization, and
metadata recording.
## Script
```text
Scripts/prototype_ground_zero_terrain.py
```
The script:
- Reads the Ground Zero tile from `Data/Tiles/ground_zero_tiles.json`.
- Converts UTM sample points to latitude/longitude.
- Queries the USGS Elevation Point Query Service in meters.
- Writes CSV elevation samples.
- Writes a little-endian unsigned `.r16` prototype heightmap.
- Writes terrain generation metadata.
- Marks the Ground Zero tile as `generated` in the tile registry seed.
## Default Output
```text
Data/Terrain/Generated/gz_us_ca_pacifica_utm10n_e544_n4160/
```
Expected files:
- `gz_us_ca_pacifica_utm10n_e544_n4160_elevation_samples_33.csv`
- `gz_us_ca_pacifica_utm10n_e544_n4160_heightmap_33.r16`
- `gz_us_ca_pacifica_utm10n_e544_n4160_terrain_metadata.json`
## Run
```bash
python3 Scripts/prototype_ground_zero_terrain.py
```
Optional:
```bash
python3 Scripts/prototype_ground_zero_terrain.py --grid-size 65 --workers 8
```
## Import Notes
- The `.r16` file is normalized from the sampled minimum elevation to sampled
maximum elevation.
- Use the generated metadata file to recover the real elevation range.
- Horizontal tile scale is 1000 m x 1000 m.
- The default 33 x 33 grid is intentionally small for a fast proof of concept.
- Final landscape import should use a valid Unreal Landscape resolution and a
higher-resolution DEM/lidar source.
## Current Source
Primary prototype source:
```text
USGS Elevation Point Query Service
https://epqs.nationalmap.gov/v1/json
```
The service is backed by USGS elevation data and returns point elevations in
meters for latitude/longitude coordinates.
+160
View File
@@ -0,0 +1,160 @@
# Tile Registry Schema
## Purpose
The tile registry is the authoritative operational index for terrain tiles. It
tracks what tiles exist, where they are, which source data generated them, what
package version is published, and whether the tile is ready for clients.
The first implementation only needs the Ground Zero tile and neighbors. The
schema is intentionally shaped so it can scale toward hundreds of millions of
possible 1 km tiles later.
## Prototype Grid
For the MVP, the tile key uses UTM zone 10N and the lower-left 1 km grid corner.
```text
gz_us_ca_pacifica_utm10n_e544_n4160
```
Fields encoded in the prototype ID:
- Location family: `gz_us_ca_pacifica`
- Projection: `utm10n`
- Easting kilometer: `e544`
- Northing kilometer: `n4160`
The final global grid is still a design decision. The MVP schema keeps explicit
projection and metric bounds so tiles can be migrated later if the global index
changes.
## Tile Status
Allowed status values:
- `unknown`: placeholder exists, no source work started.
- `queued`: selected for source lookup or generation.
- `source_data_found`: required source datasets are identified.
- `generated`: terrain package generated but not validated.
- `validated`: QA checks passed.
- `packaged`: client/server package created.
- `published`: package is available to clients.
- `deprecated`: superseded by a newer tile version.
- `blocked`: source or generation issue needs manual review.
## Core Tables
### `terrain_tiles`
Tracks one logical 1 km tile.
Required fields:
- `tile_id`
- `grid_scheme`
- `projection`
- `utm_zone`
- `easting_min_m`
- `northing_min_m`
- `easting_max_m`
- `northing_max_m`
- `tile_size_m`
- `center_latitude`
- `center_longitude`
- `status`
- `biome_primary`
- `generation_version`
- `package_version`
- `created_at`
- `updated_at`
### `terrain_tile_neighbors`
Tracks adjacency for stitching and prefetching.
Required fields:
- `tile_id`
- `direction`
- `neighbor_tile_id`
### `terrain_tile_sources`
Tracks datasets used or intended for each tile.
Required fields:
- `tile_id`
- `source_kind`
- `source_name`
- `source_uri`
- `license_name`
- `source_version`
- `coverage_status`
### `terrain_tile_packages`
Tracks generated downloadable packages.
Required fields:
- `package_id`
- `tile_id`
- `package_version`
- `unreal_engine_version`
- `world_partition_ready`
- `package_uri`
- `content_hash`
- `package_size_bytes`
- `created_at`
- `published_at`
### `terrain_tile_generation_jobs`
Tracks generation pipeline work.
Required fields:
- `job_id`
- `tile_id`
- `job_type`
- `status`
- `requested_at`
- `started_at`
- `finished_at`
- `log_uri`
- `error_summary`
## Separation Of Concerns
Terrain tile state should be separate from player-made world state.
Terrain registry owns:
- Source terrain and water data.
- Generated landscape package.
- Biome/resource hints.
- Tile status and package version.
- Client cache/version compatibility.
Player/world persistence owns:
- Player inventory, stats, and position.
- Placed structures.
- Resource depletion, if needed.
- Claims, settlements, containers, and ownership.
- Tile-local gameplay changes.
This separation lets us regenerate terrain tiles later without overwriting
player-built history.
## First Validation Rules
- Tile bounds must be exactly 1000 m x 1000 m in the projected coordinate system.
- Center latitude/longitude must fall inside tile bounds.
- Every published tile must have at least one elevation source.
- Every published tile must have a generation version and package version.
- Neighbor records must be reciprocal once adjacent tiles are generated.
- A tile package cannot be `published` until it is `validated`.
- Terrain package hash must change when package version changes.
+111
View File
@@ -0,0 +1,111 @@
# Ground Zero Unreal Landscape Import Plan
## Purpose
This document describes how to import the extracted Ground Zero 1-meter DEM
subset into Unreal as a Landscape heightmap.
## Source
```text
Data/Terrain/Extracted/gz_us_ca_pacifica_utm10n_e544_n4160/gz_us_ca_pacifica_utm10n_e544_n4160_1m_dem_subset.tif
```
Source properties:
- Size: 1000 x 1000 pixels.
- Pixel size: 1 m x 1 m.
- CRS: EPSG:26910 / NAD83 UTM zone 10N.
- Bounds: E 544000-545000, N 4160000-4161000.
- Elevation range: about 3.16 m to 96.51 m.
## Conversion Script
```text
Scripts/convert_ground_zero_dem_to_unreal_heightmap.py
```
The script:
- Reads the extracted 1-meter DEM subset.
- Resamples it to 1009 x 1009.
- Encodes elevation into unsigned 16-bit height values using Unreal's
landscape midpoint, so sea level is approximately Unreal Z 0.
- Writes a little-endian `.r16` heightmap.
- Writes a small grayscale preview file.
- Writes import metadata with Unreal scale values.
## Output
```text
Data/Terrain/Unreal/gz_us_ca_pacifica_utm10n_e544_n4160/
```
Files:
- `gz_us_ca_pacifica_utm10n_e544_n4160_unreal_1009.r16`
- `gz_us_ca_pacifica_utm10n_e544_n4160_unreal_1009_preview.pgm`
- `gz_us_ca_pacifica_utm10n_e544_n4160_unreal_heightmap_metadata.json`
If Pillow is installed, the script also writes:
- `gz_us_ca_pacifica_utm10n_e544_n4160_unreal_1009.png`
## Unreal Import Settings
Use the generated metadata file as the source of truth.
Current settings:
```text
heightmap resolution: 1009 x 1009
tile world size: 1000 m x 1000 m
X scale: 99.2063492063492 cm
Y scale: 99.2063492063492 cm
Z scale: 100.0 cm
Z offset: 0.0 m
height encoding: Unreal landscape midpoint 32768 = approximately sea level
```
Why 1009:
- Unreal Landscape import requires specific valid dimensions.
- 1009 x 1009 is a valid import size and close to the 1000 x 1000 source DEM.
- X/Y scale maps 1008 intervals across exactly 1000 real meters.
## Import Steps
1. Open the Unreal Editor.
2. Open or create the Ground Zero terrain test map.
3. Go to Landscape mode.
4. Choose Import from File.
5. Select `gz_us_ca_pacifica_utm10n_e544_n4160_unreal_1009.r16`.
6. Set the landscape resolution to 1009 x 1009 if Unreal does not auto-detect it.
7. Set X scale and Y scale from metadata.
8. Set Z scale from metadata.
9. Place the landscape so the tile origin maps to the project terrain origin.
10. Save the map under the project terrain test area.
The repeatable project import path is:
```text
Scripts/setup_ground_zero_terrain_map.py
Scripts/verify_ground_zero_terrain_map.py
```
The generated terrain test map is:
```text
/Game/Agrarian/Maps/L_GroundZeroTerrain_Test
```
## Validation After Import
- Landscape bounds should be 1000 m x 1000 m.
- Imported elevation range should visually match the source: low coastal/valley
terrain rising into hills.
- No extreme spikes or inverted terrain should appear.
- The north/south/east/west edges should remain suitable for seam comparison
against neighboring tiles.
- The landscape should be treated as prototype terrain until material, water,
biome, and resource layers are added.