AgrarianGameArchive/Docs/Terrain/EarthScaleTileStreamingDesign.md

Earth-Scale Terrain And Tile Streaming Design

Purpose

Agrarian's long-term world is intended to grow toward Earth scale while still remaining playable, patchable, and affordable during the MVP years. The terrain system must support real-world terrain, ocean depth, water, biomes, and resource clues without requiring the full planet to be generated, stored, or shipped at once.

The MVP proves the contract with the Ground Zero tile and immediate-neighbor metadata. The design must scale from that single 1 km x 1 km tile to hundreds of millions of possible tiles over time.

Core Principles

• The authoritative unit is a 1 km x 1 km logical terrain tile.
• Tiles are generated, validated, packaged, published, cached, and retired by version.
• Client machines download only the tiles they need.
• Player-made world state is stored separately from source terrain packages.
• Terrain packages are immutable once published.
• World Partition compatibility is mandatory for every terrain decision.
• The MVP can use a simple static HTTP tile server, but the contract must not block a future registry API, CDN, or regional tile service.

Scale Target

Earth's surface area implies roughly 510-520 million 1 km x 1 km tile slots. Agrarian will not build all of those tiles up front. The intended growth model is incremental:

• start with Ground Zero;
• add nearby playable tiles around Ground Zero;
• add new regions as content, testing, and player demand justify them;
• keep unpublished regions as registry placeholders until source data and generation work are scheduled;
• allow improved source data and generation passes to publish new tile package versions without destroying existing player history.

Current MVP Tile

The first tile uses the existing UTM zone 10N prototype ID:

gz_us_ca_pacifica_utm10n_e544_n4160

It represents the selected Ground Zero 1 km x 1 km MVP tile. The current package is published through the dedicated Agrarian-TileServer VM:

http://maps.agrariangame.com:18080

Current static endpoints:

• /health
• /manifest.json
• /ground_zero_tiles.json
• /schemas/tile_registry.schema.json
• /tiles/<tile_id>/v<package_version>/...

Canonical Tile Identity

Each tile needs a stable logical identity and enough coordinate metadata to survive future grid changes. The MVP tile ID encodes location, projection, and 1 km lower-left grid corner. Long term, the registry should treat the tile ID as an opaque key and store coordinate details in fields.

Required identity fields:

• tile_id
• grid_scheme
• projection
• projected_bounds_m
• wgs84_bounds
• center_latitude
• center_longitude
• tile_size_m
• generation_version
• package_version

The Ground Zero UTM format is acceptable for the MVP. Before generating many regions, the project should add a global grid naming layer that can coexist with regional projected grids. That prevents a future UTM-zone, dateline, or polar decision from forcing player save records to change tile IDs.

Coordinate Strategy

Real-world latitude and longitude should not map directly to Unreal world coordinates. The registry maps real-world coordinates to logical tiles, then each tile package imports into Unreal using tile-local metric coordinates.

MVP mapping:

• Ground Zero is the local playable origin.
• The tile is 1000 m x 1000 m in its projected coordinate system.
• Neighbor tiles are positioned by cardinal/intercardinal 1 km offsets.
• Unreal content uses local tile placement and World Partition loading, not raw latitude/longitude transforms.

Long-term mapping:

• The registry resolves player/world coordinates to tile IDs.
• Tile packages carry their projection metadata and real-world bounds.
• The gameplay server is authoritative for active tile package versions.
• Unreal World Partition manages loaded cells around the player.
• Origin shifting and region-local coordinate roots are used before world coordinates become too large for precision-sensitive gameplay.

Projection, Dateline, And Polar Handling

No single flat projection handles the entire planet cleanly at 1 km gameplay resolution. Agrarian should use local projected coordinate systems for terrain generation and store WGS84 bounds for global lookup.

Rules:

• Use metric projected coordinates when generating a tile package.
• Store the projection and source coordinate reference system in the registry.
• Store WGS84 bounds and center coordinates for lookup, maps, and search.
• Treat UTM zone edges, the antimeridian, and polar regions as explicit grid boundary cases.
• Do not publish a boundary tile until neighbor and edge validation rules are satisfied for that grid scheme.

The MVP does not need to solve polar and dateline regions, but the database schema and file package metadata must not assume every tile is in one UTM zone.

Tile Adjacency And Stitching

Each generated tile tracks up to eight neighbors:

N, NE, E, SE, S, SW, W, NW

Adjacency records are used for:

• terrain edge validation;
• water and shoreline continuity;
• World Partition placement;
• client prefetch;
• gameplay server interest management;
• future road, river, fence, and settlement continuity.

Stitching rules:

• Neighbor records must be reciprocal once both tiles exist.
• Shared elevation edges must agree within the active generation tolerance.
• Water polygons and shoreline masks must not create visible gaps at edges.
• River and road features crossing tile borders must have matching continuation metadata.
• A tile may publish with an "open edge" only if the neighbor is unknown or not generated yet.
• Once a neighboring tile exists, both sides must pass edge QA before either package is promoted beyond validation for that shared boundary.

Tile Versioning

Terrain packages are immutable by version. A published package is never edited in place.

Create a new package version when any of these change:

• elevation or bathymetry source data;
• coastline, river, lake, road, or land-cover source data;
• terrain generation code;
• landscape import settings;
• biome or resource inference inputs;
• collision or navigation-relevant terrain output;
• package layout or required client version.

Player and world persistence records reference:

• logical tile_id;
• active package_version;
• tile-local positions;
• placed object or resource state records owned by persistence, not the tile package.

Deprecated packages remain available until active saves have migrated or no supported server requires them. Save migration should be explicit and logged; it must not silently reinterpret player-built structures against a changed terrain surface.

Server Delivery Protocol

The MVP tile server is a static nginx service. The contract is intentionally simple:

1. Client requests /manifest.json.
2. Manifest points to tile registry JSON and schema.
3. Client requests the tile registry file.
4. Client resolves a required tile_id and package_version.
5. Client downloads listed package files.
6. Client verifies checksums.
7. Client marks the local package as verified.
8. Gameplay server decides whether that tile package is valid for the current session.

Future server versions can replace static JSON with an API, but should preserve the same concepts:

• manifest version;
• registry version;
• tile ID;
• package version;
• package file list;
• content hashes;
• minimum compatible game build;
• deprecation status;
• replacement package, if available.

Missing tile behavior:

• The tile server returns a normal HTTP missing response for unknown package files.
• The gameplay client reports the missing tile and does not invent terrain.
• The gameplay server can block travel, keep the player in the loaded region, or route them to a fallback boundary experience depending on the game mode.

Public Hosting Decision

For MVP and internal closed testing, the LAN-hosted Agrarian-TileServer VM is acceptable because it proves the end-to-end tile lookup, download, cache, and redownload flow at no recurring cloud cost.

Before broader public testing, move public tile delivery to an external cloud host, CDN-backed static bucket, or equivalent public edge service. The home VM can remain the staging publisher, but public testers should not depend on home uplink reliability once the test group grows.

Client Cache Layout

Use a deterministic cache layout so downloads can be validated and scrubbed without touching player saves.

Recommended layout:

<cache_root>/
  manifests/
    manifest.<manifest_version>.json
  registries/
    tile_registry.<registry_version>.json
  tiles/
    <tile_id>/
      v<package_version>/
        package.json
        files...
        SHA256SUMS
        .verified
      .partial/
        download-in-progress files

Rules:

• Download into .partial.
• Verify all hashes before promotion.
• Promote atomically by renaming into the final version folder.
• Write .verified only after checksums pass.
• Never store player inventory, structures, claims, or depletion state inside the tile package cache.

Cache Retention

The cache should keep movement smooth without growing forever.

MVP policy:

• Always pin the active tile.
• Pin immediate neighbors around the player.
• Keep recently visited verified tiles until a configurable size or age limit is reached.
• Prefer deleting the oldest unpinned verified tile packages first.
• Never delete .partial downloads that are currently active.
• Remove stale .partial folders on startup if no download owns them.
• Never delete server-authoritative save data or dirty local session data as part of terrain cache cleanup.

Initial settings can be conservative and adjusted after real package sizes are known:

• minimum pinned set: current tile plus immediate neighbors;
• retention age target: 7-30 days for unpinned packages;
• retention size target: configurable per build and storage class.

Redownload And Revalidation

At startup and before loading a tile:

• confirm the required package version exists locally;
• confirm .verified exists;
• optionally recheck hashes for critical files;
• redownload missing or failed files;
• delete and rebuild corrupt package folders instead of trying to repair them in place.

When a manifest marks a package deprecated:

• keep using it if the active server/session requires it;
• download the replacement package in the background when possible;
• migrate only through an explicit server-approved flow.

Source Data Direction

The source-data strategy should prefer authoritative, affordable, scriptable datasets with clear licenses.

MVP source families:

• elevation: USGS DEM sources for Ground Zero and other US tiles;
• bathymetry/ocean depth: NOAA or global bathymetry sources as coastal/ocean tiles become relevant;
• rivers, lakes, and coastline: USGS hydrography for US tiles, plus global hydro/coastline candidates for non-US regions;
• roads and human map features: OpenStreetMap-derived data where license and attribution requirements can be met;
• land cover and biome hints: NLCD for US tiles and global land-cover datasets for later regions;
• weather/climate context: online weather snapshots for current regional weather, plus long-term climate normals for biome/resource inference.

Detailed licensing and attribution review is required before shipping any source-derived data publicly.

Single-Tile Import Pipeline

The automated pipeline for one 1 km tile should be:

1. Select or create a terrain_tiles registry record.
2. Resolve required source datasets.
3. Download source files into an uncommitted local/source-data cache.
4. Clip each source to the tile bounds plus edge buffer.
5. Normalize projection and vertical units.
6. Generate Unreal-compatible heightmap and metadata.
7. Analyze landforms, slope, water, and shoreline.
8. Generate biome and natural-resource hint layers.
9. Import or update the Unreal landscape package.
10. Run World Partition and map-check validation.
11. Run edge, water, slope, and missing-data QA.
12. Build an immutable tile delivery package.
13. Publish the package only after validation passes.
14. Update the manifest and registry.

The current Ground Zero scripts are the prototype foundation for this flow:

• Scripts/acquire_ground_zero_dem.py
• Scripts/extract_ground_zero_dem_subset.py
• Scripts/convert_ground_zero_dem_to_unreal_heightmap.py
• Scripts/prototype_ground_zero_terrain.py
• Scripts/setup_ground_zero_terrain_map.py
• Scripts/setup_ground_zero_demo_map.py
• Scripts/analyze_ground_zero_landforms.py
• Scripts/analyze_ground_zero_water.py
• Scripts/verify_ground_zero_neighbor_edges.py
• Scripts/build_ground_zero_tile_delivery_package.sh
• Scripts/verify_tile_delivery_client.sh

Biome And Resource Inference

Biome and resource inference should produce hints, not final gameplay authority. The gameplay server and content rules decide what actually spawns, depletes, or persists.

Inputs:

• elevation;
• slope and aspect;
• water coverage;
• shoreline distance;
• land cover;
• climate and weather context;
• soil/geology/resource datasets when available;
• human map features where relevant.

Outputs:

• primary and secondary biome tags;
• water availability hints;
• forage/wood/stone/clay/mineral likelihood hints;
• traversal risk hints;
• wildlife habitat hints;
• farming suitability hints.

These outputs belong in the tile registry/package metadata. Player-harvested resources and depleted nodes belong in persistence.

QA Gates

A tile cannot publish until it passes the relevant checks:

• bounds are exactly 1000 m x 1000 m in the projected coordinate system;
• center latitude/longitude falls inside the tile bounds;
• required source datasets are present or explicitly waived;
• package has generation and package versions;
• content hash exists and changes for new package versions;
• World Partition compatibility flag is true;
• map check passes for the generated tile map/package;
• slopes outside accepted gameplay ranges are flagged;
• water and shoreline masks are present where expected;
• generated neighbor records are reciprocal;
• shared edges match within tolerance where neighbors exist;
• missing neighbor edges are recorded as open edges;
• client download and checksum verification pass;
• delete/redownload recovery passes for at least one package file.

World Partition Requirements

World Partition is a hard requirement, not an optional optimization. Each tile package must be shaped so it can load around the player without requiring the entire world or region to be present.

Requirements:

• tile packages use stable tile-local placement;
• adjacent tile offsets are deterministic;
• generated content is grouped by tile;
• large generated artifacts stay out of Git unless explicitly accepted under the repository storage policy;
• tests include at least one command-mode Unreal load/map-check path;
• future multiplayer servers can decide which tile versions are active before clients stream them.

Open Questions

• Final global tile naming format beyond the Ground Zero UTM prototype.
• Exact cache size defaults for investor/demo, closed test, and public builds.
• External public tile hosting provider for broader closed testing.
• Final licensed global source datasets for hydrography, land cover, roads, bathymetry, and geology.
• Save migration UX when a populated tile receives improved terrain.