From 1c601d2eaa74861edbe5dd9b32e3d04b832ea86b Mon Sep 17 00:00:00 2001
From: nathan <nathan@pacificao.com>
Date: Fri, 15 May 2026 01:49:05 -0700
Subject: [PATCH] Add Earth-scale tile streaming design

---
 AGRARIAN_DEVELOPMENT_ROADMAP.md               |  44 +-
 Docs/Terrain/EarthScaleTileStreamingDesign.md | 435 ++++++++++++++++++
 2 files changed, 457 insertions(+), 22 deletions(-)
 create mode 100644 Docs/Terrain/EarthScaleTileStreamingDesign.md

diff --git a/AGRARIAN_DEVELOPMENT_ROADMAP.md b/AGRARIAN_DEVELOPMENT_ROADMAP.md
index 113e507..e6745c2 100644
--- a/AGRARIAN_DEVELOPMENT_ROADMAP.md
+++ b/AGRARIAN_DEVELOPMENT_ROADMAP.md
@@ -242,11 +242,11 @@ Current tooling decisions:
 
 ## 0.3 Design Documentation
 
-- [ ] Create game design document.
-- [ ] Create technical design document.
-- [ ] Create multiplayer/networking design document.
-- [ ] Create persistence design document.
-- [ ] Create Earth-scale terrain/tile streaming design document.
+- [x] Create game design document.
+- [x] Create technical design document.
+- [x] Create multiplayer/networking design document.
+- [x] Create persistence design document.
+- [x] Create Earth-scale terrain/tile streaming design document.
 - [x] Create tile registry/database design document.
 - [x] Create real-world terrain source data evaluation document.
 - [ ] Create economy and AGR design document.
@@ -328,34 +328,34 @@ full Earth scale this implies roughly 510-520 million possible tiles, added over
 many years, streamed from servers, cached locally, scrubbed when unused, and
 redownloaded when a player returns to a region.
 
-- [ ] Define canonical tile coordinate system for 1 km x 1 km world tiles.
-- [ ] Decide how latitude/longitude maps to Unreal coordinates and World Partition cells.
-- [ ] Decide how to handle projection distortion, poles, and dateline crossing.
+- [x] Define canonical tile coordinate system for 1 km x 1 km world tiles.
+- [x] Decide how latitude/longitude maps to Unreal coordinates and World Partition cells.
+- [x] Decide how to handle projection distortion, poles, and dateline crossing.
 - [x] Define authoritative tile manifest schema.
 - [x] Define tile status states: unknown, queued, source-data-found, generated, validated, packaged, published, deprecated.
-- [ ] Define tile adjacency and stitching rules.
-- [ ] Define tile versioning rules so terrain can improve without corrupting player state.
-- [ ] Define server-side tile delivery protocol.
+- [x] Define tile adjacency and stitching rules.
+- [x] Define tile versioning rules so terrain can improve without corrupting player state.
+- [x] Define server-side tile delivery protocol.
 - [x] Launch a near-term MVP tile-serving VM on Ubuntu or another Linux distro.
 - [x] Add repeatable static Ground Zero tile-delivery package and Ubuntu nginx bootstrap scripts.
 - [x] Publish a tiny Ground Zero tile manifest and package from the tile server.
 - [x] Prove client/server tile lookup, download, local cache, and redownload flow with the Ground Zero tile and immediate-neighbor metadata.
 - [x] Add tile-serving server cost controls and shutdown/runbook notes so MVP testing stays free or near-free.
 - [x] Move the MVP tile server into a dedicated `Agrarian-TileServer` VM on Unraid.
-- [ ] Decide whether public testing uses the LAN-hosted `Agrarian-TileServer` VM or an external cloud host.
-- [ ] Define client local tile cache layout.
-- [ ] Define local cache retention policy for old or unused tiles.
-- [ ] Define local cache redownload/revalidation behavior.
+- [x] Decide whether public testing uses the LAN-hosted `Agrarian-TileServer` VM or an external cloud host.
+- [x] Define client local tile cache layout.
+- [x] Define local cache retention policy for old or unused tiles.
+- [x] Define local cache redownload/revalidation behavior.
 - [x] Define minimum metadata per tile: coordinates, biome, elevation range, water coverage, source datasets, generation version, neighbors, and publish status.
 - [x] Evaluate real terrain/elevation data sources.
 - [x] Evaluate bathymetry/ocean-depth data sources.
-- [ ] Evaluate river, lake, coastline, road, and land-cover data sources.
-- [ ] Define automated terrain import pipeline for a single 1 km tile.
-- [ ] Define automated biome and natural resource inference pipeline.
-- [ ] Define QA checks for terrain seams, water edges, slope extremes, and missing source data.
+- [x] Evaluate river, lake, coastline, road, and land-cover data sources.
+- [x] Define automated terrain import pipeline for a single 1 km tile.
+- [x] Define automated biome and natural resource inference pipeline.
+- [x] Define QA checks for terrain seams, water edges, slope extremes, and missing source data.
 - [x] Build a small tile-tracking database prototype before scaling beyond MVP.
 - [x] Build an MVP tile registry table for the Ground Zero tile and immediate neighbors.
-- [ ] Keep World Partition compatibility as a hard requirement for all terrain decisions.
+- [x] Keep World Partition compatibility as a hard requirement for all terrain decisions.
 
 # Phase 1 - Foundational Survival MVP
 
@@ -1433,7 +1433,7 @@ Earliest incomplete foundation items:
 - [x] Create the technical design document.
 - [x] Create the multiplayer/networking design document.
 - [x] Create the persistence design document.
-- [ ] Create the Earth-scale terrain/tile streaming design document.
+- [x] Create the Earth-scale terrain/tile streaming design document.
 - [x] Launch near-term MVP map-tile serving cloud VM and prove Ground Zero tile lookup/download/cache flow.
 - [ ] Create economy and AGR design document.
 - [ ] Create art direction, UX/HUD direction, coding standards, Blueprint standards, and asset/folder naming standards.
@@ -1443,4 +1443,4 @@ Earliest incomplete foundation items:
 
 Immediate next item:
 
-- [ ] Create the Earth-scale terrain/tile streaming design document.
+- [ ] Create economy and AGR design document.
diff --git a/Docs/Terrain/EarthScaleTileStreamingDesign.md b/Docs/Terrain/EarthScaleTileStreamingDesign.md
new file mode 100644
index 0000000..45b0ab7
--- /dev/null
+++ b/Docs/Terrain/EarthScaleTileStreamingDesign.md
@@ -0,0 +1,435 @@
+# Earth-Scale Terrain And Tile Streaming Design
+
+## Purpose
+
+Agrarian's long-term world is intended to grow toward Earth scale while still
+remaining playable, patchable, and affordable during the MVP years. The terrain
+system must support real-world terrain, ocean depth, water, biomes, and resource
+clues without requiring the full planet to be generated, stored, or shipped at
+once.
+
+The MVP proves the contract with the Ground Zero tile and immediate-neighbor
+metadata. The design must scale from that single 1 km x 1 km tile to hundreds of
+millions of possible tiles over time.
+
+## Core Principles
+
+- The authoritative unit is a 1 km x 1 km logical terrain tile.
+- Tiles are generated, validated, packaged, published, cached, and retired by
+  version.
+- Client machines download only the tiles they need.
+- Player-made world state is stored separately from source terrain packages.
+- Terrain packages are immutable once published.
+- World Partition compatibility is mandatory for every terrain decision.
+- The MVP can use a simple static HTTP tile server, but the contract must not
+  block a future registry API, CDN, or regional tile service.
+
+## Scale Target
+
+Earth's surface area implies roughly 510-520 million 1 km x 1 km tile slots.
+Agrarian will not build all of those tiles up front. The intended growth model
+is incremental:
+
+- start with Ground Zero;
+- add nearby playable tiles around Ground Zero;
+- add new regions as content, testing, and player demand justify them;
+- keep unpublished regions as registry placeholders until source data and
+  generation work are scheduled;
+- allow improved source data and generation passes to publish new tile package
+  versions without destroying existing player history.
+
+## Current MVP Tile
+
+The first tile uses the existing UTM zone 10N prototype ID:
+
+```text
+gz_us_ca_pacifica_utm10n_e544_n4160
+```
+
+It represents the selected Ground Zero 1 km x 1 km MVP tile. The current package
+is published through the dedicated `Agrarian-TileServer` VM:
+
+```text
+http://maps.agrariangame.com:18080
+```
+
+Current static endpoints:
+
+- `/health`
+- `/manifest.json`
+- `/ground_zero_tiles.json`
+- `/schemas/tile_registry.schema.json`
+- `/tiles/<tile_id>/v<package_version>/...`
+
+## Canonical Tile Identity
+
+Each tile needs a stable logical identity and enough coordinate metadata to
+survive future grid changes. The MVP tile ID encodes location, projection, and
+1 km lower-left grid corner. Long term, the registry should treat the tile ID as
+an opaque key and store coordinate details in fields.
+
+Required identity fields:
+
+- `tile_id`
+- `grid_scheme`
+- `projection`
+- `projected_bounds_m`
+- `wgs84_bounds`
+- `center_latitude`
+- `center_longitude`
+- `tile_size_m`
+- `generation_version`
+- `package_version`
+
+The Ground Zero UTM format is acceptable for the MVP. Before generating many
+regions, the project should add a global grid naming layer that can coexist with
+regional projected grids. That prevents a future UTM-zone, dateline, or polar
+decision from forcing player save records to change tile IDs.
+
+## Coordinate Strategy
+
+Real-world latitude and longitude should not map directly to Unreal world
+coordinates. The registry maps real-world coordinates to logical tiles, then
+each tile package imports into Unreal using tile-local metric coordinates.
+
+MVP mapping:
+
+- Ground Zero is the local playable origin.
+- The tile is 1000 m x 1000 m in its projected coordinate system.
+- Neighbor tiles are positioned by cardinal/intercardinal 1 km offsets.
+- Unreal content uses local tile placement and World Partition loading, not
+  raw latitude/longitude transforms.
+
+Long-term mapping:
+
+- The registry resolves player/world coordinates to tile IDs.
+- Tile packages carry their projection metadata and real-world bounds.
+- The gameplay server is authoritative for active tile package versions.
+- Unreal World Partition manages loaded cells around the player.
+- Origin shifting and region-local coordinate roots are used before world
+  coordinates become too large for precision-sensitive gameplay.
+
+## Projection, Dateline, And Polar Handling
+
+No single flat projection handles the entire planet cleanly at 1 km gameplay
+resolution. Agrarian should use local projected coordinate systems for terrain
+generation and store WGS84 bounds for global lookup.
+
+Rules:
+
+- Use metric projected coordinates when generating a tile package.
+- Store the projection and source coordinate reference system in the registry.
+- Store WGS84 bounds and center coordinates for lookup, maps, and search.
+- Treat UTM zone edges, the antimeridian, and polar regions as explicit grid
+  boundary cases.
+- Do not publish a boundary tile until neighbor and edge validation rules are
+  satisfied for that grid scheme.
+
+The MVP does not need to solve polar and dateline regions, but the database
+schema and file package metadata must not assume every tile is in one UTM zone.
+
+## Tile Adjacency And Stitching
+
+Each generated tile tracks up to eight neighbors:
+
+```text
+N, NE, E, SE, S, SW, W, NW
+```
+
+Adjacency records are used for:
+
+- terrain edge validation;
+- water and shoreline continuity;
+- World Partition placement;
+- client prefetch;
+- gameplay server interest management;
+- future road, river, fence, and settlement continuity.
+
+Stitching rules:
+
+- Neighbor records must be reciprocal once both tiles exist.
+- Shared elevation edges must agree within the active generation tolerance.
+- Water polygons and shoreline masks must not create visible gaps at edges.
+- River and road features crossing tile borders must have matching continuation
+  metadata.
+- A tile may publish with an "open edge" only if the neighbor is unknown or not
+  generated yet.
+- Once a neighboring tile exists, both sides must pass edge QA before either
+  package is promoted beyond validation for that shared boundary.
+
+## Tile Versioning
+
+Terrain packages are immutable by version. A published package is never edited
+in place.
+
+Create a new package version when any of these change:
+
+- elevation or bathymetry source data;
+- coastline, river, lake, road, or land-cover source data;
+- terrain generation code;
+- landscape import settings;
+- biome or resource inference inputs;
+- collision or navigation-relevant terrain output;
+- package layout or required client version.
+
+Player and world persistence records reference:
+
+- logical `tile_id`;
+- active `package_version`;
+- tile-local positions;
+- placed object or resource state records owned by persistence, not the tile
+  package.
+
+Deprecated packages remain available until active saves have migrated or no
+supported server requires them. Save migration should be explicit and logged; it
+must not silently reinterpret player-built structures against a changed terrain
+surface.
+
+## Server Delivery Protocol
+
+The MVP tile server is a static nginx service. The contract is intentionally
+simple:
+
+1. Client requests `/manifest.json`.
+2. Manifest points to tile registry JSON and schema.
+3. Client requests the tile registry file.
+4. Client resolves a required `tile_id` and `package_version`.
+5. Client downloads listed package files.
+6. Client verifies checksums.
+7. Client marks the local package as verified.
+8. Gameplay server decides whether that tile package is valid for the current
+   session.
+
+Future server versions can replace static JSON with an API, but should preserve
+the same concepts:
+
+- manifest version;
+- registry version;
+- tile ID;
+- package version;
+- package file list;
+- content hashes;
+- minimum compatible game build;
+- deprecation status;
+- replacement package, if available.
+
+Missing tile behavior:
+
+- The tile server returns a normal HTTP missing response for unknown package
+  files.
+- The gameplay client reports the missing tile and does not invent terrain.
+- The gameplay server can block travel, keep the player in the loaded region, or
+  route them to a fallback boundary experience depending on the game mode.
+
+## Public Hosting Decision
+
+For MVP and internal closed testing, the LAN-hosted `Agrarian-TileServer` VM is
+acceptable because it proves the end-to-end tile lookup, download, cache, and
+redownload flow at no recurring cloud cost.
+
+Before broader public testing, move public tile delivery to an external cloud
+host, CDN-backed static bucket, or equivalent public edge service. The home VM
+can remain the staging publisher, but public testers should not depend on home
+uplink reliability once the test group grows.
+
+## Client Cache Layout
+
+Use a deterministic cache layout so downloads can be validated and scrubbed
+without touching player saves.
+
+Recommended layout:
+
+```text
+<cache_root>/
+  manifests/
+    manifest.<manifest_version>.json
+  registries/
+    tile_registry.<registry_version>.json
+  tiles/
+    <tile_id>/
+      v<package_version>/
+        package.json
+        files...
+        SHA256SUMS
+        .verified
+      .partial/
+        download-in-progress files
+```
+
+Rules:
+
+- Download into `.partial`.
+- Verify all hashes before promotion.
+- Promote atomically by renaming into the final version folder.
+- Write `.verified` only after checksums pass.
+- Never store player inventory, structures, claims, or depletion state inside
+  the tile package cache.
+
+## Cache Retention
+
+The cache should keep movement smooth without growing forever.
+
+MVP policy:
+
+- Always pin the active tile.
+- Pin immediate neighbors around the player.
+- Keep recently visited verified tiles until a configurable size or age limit is
+  reached.
+- Prefer deleting the oldest unpinned verified tile packages first.
+- Never delete `.partial` downloads that are currently active.
+- Remove stale `.partial` folders on startup if no download owns them.
+- Never delete server-authoritative save data or dirty local session data as
+  part of terrain cache cleanup.
+
+Initial settings can be conservative and adjusted after real package sizes are
+known:
+
+- minimum pinned set: current tile plus immediate neighbors;
+- retention age target: 7-30 days for unpinned packages;
+- retention size target: configurable per build and storage class.
+
+## Redownload And Revalidation
+
+At startup and before loading a tile:
+
+- confirm the required package version exists locally;
+- confirm `.verified` exists;
+- optionally recheck hashes for critical files;
+- redownload missing or failed files;
+- delete and rebuild corrupt package folders instead of trying to repair them in
+  place.
+
+When a manifest marks a package deprecated:
+
+- keep using it if the active server/session requires it;
+- download the replacement package in the background when possible;
+- migrate only through an explicit server-approved flow.
+
+## Source Data Direction
+
+The source-data strategy should prefer authoritative, affordable, scriptable
+datasets with clear licenses.
+
+MVP source families:
+
+- elevation: USGS DEM sources for Ground Zero and other US tiles;
+- bathymetry/ocean depth: NOAA or global bathymetry sources as coastal/ocean
+  tiles become relevant;
+- rivers, lakes, and coastline: USGS hydrography for US tiles, plus global
+  hydro/coastline candidates for non-US regions;
+- roads and human map features: OpenStreetMap-derived data where license and
+  attribution requirements can be met;
+- land cover and biome hints: NLCD for US tiles and global land-cover datasets
+  for later regions;
+- weather/climate context: online weather snapshots for current regional
+  weather, plus long-term climate normals for biome/resource inference.
+
+Detailed licensing and attribution review is required before shipping any
+source-derived data publicly.
+
+## Single-Tile Import Pipeline
+
+The automated pipeline for one 1 km tile should be:
+
+1. Select or create a `terrain_tiles` registry record.
+2. Resolve required source datasets.
+3. Download source files into an uncommitted local/source-data cache.
+4. Clip each source to the tile bounds plus edge buffer.
+5. Normalize projection and vertical units.
+6. Generate Unreal-compatible heightmap and metadata.
+7. Analyze landforms, slope, water, and shoreline.
+8. Generate biome and natural-resource hint layers.
+9. Import or update the Unreal landscape package.
+10. Run World Partition and map-check validation.
+11. Run edge, water, slope, and missing-data QA.
+12. Build an immutable tile delivery package.
+13. Publish the package only after validation passes.
+14. Update the manifest and registry.
+
+The current Ground Zero scripts are the prototype foundation for this flow:
+
+- `Scripts/acquire_ground_zero_dem.py`
+- `Scripts/extract_ground_zero_dem_subset.py`
+- `Scripts/convert_ground_zero_dem_to_unreal_heightmap.py`
+- `Scripts/prototype_ground_zero_terrain.py`
+- `Scripts/setup_ground_zero_terrain_map.py`
+- `Scripts/setup_ground_zero_demo_map.py`
+- `Scripts/analyze_ground_zero_landforms.py`
+- `Scripts/analyze_ground_zero_water.py`
+- `Scripts/verify_ground_zero_neighbor_edges.py`
+- `Scripts/build_ground_zero_tile_delivery_package.sh`
+- `Scripts/verify_tile_delivery_client.sh`
+
+## Biome And Resource Inference
+
+Biome and resource inference should produce hints, not final gameplay authority.
+The gameplay server and content rules decide what actually spawns, depletes, or
+persists.
+
+Inputs:
+
+- elevation;
+- slope and aspect;
+- water coverage;
+- shoreline distance;
+- land cover;
+- climate and weather context;
+- soil/geology/resource datasets when available;
+- human map features where relevant.
+
+Outputs:
+
+- primary and secondary biome tags;
+- water availability hints;
+- forage/wood/stone/clay/mineral likelihood hints;
+- traversal risk hints;
+- wildlife habitat hints;
+- farming suitability hints.
+
+These outputs belong in the tile registry/package metadata. Player-harvested
+resources and depleted nodes belong in persistence.
+
+## QA Gates
+
+A tile cannot publish until it passes the relevant checks:
+
+- bounds are exactly 1000 m x 1000 m in the projected coordinate system;
+- center latitude/longitude falls inside the tile bounds;
+- required source datasets are present or explicitly waived;
+- package has generation and package versions;
+- content hash exists and changes for new package versions;
+- World Partition compatibility flag is true;
+- map check passes for the generated tile map/package;
+- slopes outside accepted gameplay ranges are flagged;
+- water and shoreline masks are present where expected;
+- generated neighbor records are reciprocal;
+- shared edges match within tolerance where neighbors exist;
+- missing neighbor edges are recorded as open edges;
+- client download and checksum verification pass;
+- delete/redownload recovery passes for at least one package file.
+
+## World Partition Requirements
+
+World Partition is a hard requirement, not an optional optimization. Each tile
+package must be shaped so it can load around the player without requiring the
+entire world or region to be present.
+
+Requirements:
+
+- tile packages use stable tile-local placement;
+- adjacent tile offsets are deterministic;
+- generated content is grouped by tile;
+- large generated artifacts stay out of Git unless explicitly accepted under
+  the repository storage policy;
+- tests include at least one command-mode Unreal load/map-check path;
+- future multiplayer servers can decide which tile versions are active before
+  clients stream them.
+
+## Open Questions
+
+- Final global tile naming format beyond the Ground Zero UTM prototype.
+- Exact cache size defaults for investor/demo, closed test, and public builds.
+- External public tile hosting provider for broader closed testing.
+- Final licensed global source datasets for hydrography, land cover, roads,
+  bathymetry, and geology.
+- Save migration UX when a populated tile receives improved terrain.