# Earth-Scale Terrain And Tile Streaming Design

## Purpose

Agrarian's long-term world is intended to grow toward Earth scale while still
remaining playable, patchable, and affordable during the MVP years. The terrain
system must support real-world terrain, ocean depth, water, biomes, and resource
clues without requiring the full planet to be generated, stored, or shipped at
once.

The MVP proves the contract with the Ground Zero tile and immediate-neighbor
metadata. The design must scale from that single 1 km x 1 km tile to hundreds of
millions of possible tiles over time.

## Core Principles

- The authoritative unit is a 1 km x 1 km logical terrain tile.
- Tiles are generated, validated, packaged, published, cached, and retired by
  version.
- Client machines download only the tiles they need.
- Player-made world state is stored separately from source terrain packages.
- Terrain packages are immutable once published.
- World Partition compatibility is mandatory for every terrain decision.
- The MVP can use a simple static HTTP tile server, but the contract must not
  block a future registry API, CDN, or regional tile service.

## Scale Target

Earth's surface area implies roughly 510-520 million 1 km x 1 km tile slots.
Agrarian will not build all of those tiles up front. The intended growth model
is incremental:

- start with Ground Zero;
- add nearby playable tiles around Ground Zero;
- add new regions as content, testing, and player demand justify them;
- keep unpublished regions as registry placeholders until source data and
  generation work are scheduled;
- allow improved source data and generation passes to publish new tile package
  versions without destroying existing player history.

## Current MVP Tile

The first tile uses the existing UTM zone 10N prototype ID:

```text
gz_us_ca_pacifica_utm10n_e544_n4160
```

It represents the selected Ground Zero 1 km x 1 km MVP tile. The current package
is published through the dedicated `Agrarian-TileServer` VM:

```text
http://maps.agrariangame.com:18080
```

Current static endpoints:

- `/health`
- `/manifest.json`
- `/ground_zero_tiles.json`
- `/schemas/tile_registry.schema.json`
- `/tiles/<tile_id>/v<package_version>/...`

## Canonical Tile Identity

Each tile needs a stable logical identity and enough coordinate metadata to
survive future grid changes. The MVP tile ID encodes location, projection, and
1 km lower-left grid corner. Long term, the registry should treat the tile ID as
an opaque key and store coordinate details in fields.

Required identity fields:

- `tile_id`
- `grid_scheme`
- `projection`
- `projected_bounds_m`
- `wgs84_bounds`
- `center_latitude`
- `center_longitude`
- `tile_size_m`
- `generation_version`
- `package_version`

The Ground Zero UTM format is acceptable for the MVP. Before generating many
regions, the project should add a global grid naming layer that can coexist with
regional projected grids. That prevents a future UTM-zone, dateline, or polar
decision from forcing player save records to change tile IDs.

## Coordinate Strategy

Real-world latitude and longitude should not map directly to Unreal world
coordinates. The registry maps real-world coordinates to logical tiles, then
each tile package imports into Unreal using tile-local metric coordinates.

MVP mapping:

- Ground Zero is the local playable origin.
- The tile is 1000 m x 1000 m in its projected coordinate system.
- Neighbor tiles are positioned by cardinal/intercardinal 1 km offsets.
- Unreal content uses local tile placement and World Partition loading, not
  raw latitude/longitude transforms.

Long-term mapping:

- The registry resolves player/world coordinates to tile IDs.
- Tile packages carry their projection metadata and real-world bounds.
- The gameplay server is authoritative for active tile package versions.
- Unreal World Partition manages loaded cells around the player.
- Origin shifting and region-local coordinate roots are used before world
  coordinates become too large for precision-sensitive gameplay.

## Projection, Dateline, And Polar Handling

No single flat projection handles the entire planet cleanly at 1 km gameplay
resolution. Agrarian should use local projected coordinate systems for terrain
generation and store WGS84 bounds for global lookup.

Rules:

- Use metric projected coordinates when generating a tile package.
- Store the projection and source coordinate reference system in the registry.
- Store WGS84 bounds and center coordinates for lookup, maps, and search.
- Treat UTM zone edges, the antimeridian, and polar regions as explicit grid
  boundary cases.
- Do not publish a boundary tile until neighbor and edge validation rules are
  satisfied for that grid scheme.

The MVP does not need to solve polar and dateline regions, but the database
schema and file package metadata must not assume every tile is in one UTM zone.

## Tile Adjacency And Stitching

Each generated tile tracks up to eight neighbors:

```text
N, NE, E, SE, S, SW, W, NW
```

Adjacency records are used for:

- terrain edge validation;
- water and shoreline continuity;
- World Partition placement;
- client prefetch;
- gameplay server interest management;
- future road, river, fence, and settlement continuity.

Stitching rules:

- Neighbor records must be reciprocal once both tiles exist.
- Shared elevation edges must agree within the active generation tolerance.
- Water polygons and shoreline masks must not create visible gaps at edges.
- River and road features crossing tile borders must have matching continuation
  metadata.
- A tile may publish with an "open edge" only if the neighbor is unknown or not
  generated yet.
- Once a neighboring tile exists, both sides must pass edge QA before either
  package is promoted beyond validation for that shared boundary.

## Tile Versioning

Terrain packages are immutable by version. A published package is never edited
in place.

Create a new package version when any of these change:

- elevation or bathymetry source data;
- coastline, river, lake, road, or land-cover source data;
- terrain generation code;
- landscape import settings;
- biome or resource inference inputs;
- collision or navigation-relevant terrain output;
- package layout or required client version.

Player and world persistence records reference:

- logical `tile_id`;
- active `package_version`;
- tile-local positions;
- placed object or resource state records owned by persistence, not the tile
  package.

Deprecated packages remain available until active saves have migrated or no
supported server requires them. Save migration should be explicit and logged; it
must not silently reinterpret player-built structures against a changed terrain
surface.

## Server Delivery Protocol

The MVP tile server is a static nginx service. The contract is intentionally
simple:

1. Client requests `/manifest.json`.
2. Manifest points to tile registry JSON and schema.
3. Client requests the tile registry file.
4. Client resolves a required `tile_id` and `package_version`.
5. Client downloads listed package files.
6. Client verifies checksums.
7. Client marks the local package as verified.
8. Gameplay server decides whether that tile package is valid for the current
   session.

Future server versions can replace static JSON with an API, but should preserve
the same concepts:

- manifest version;
- registry version;
- tile ID;
- package version;
- package file list;
- content hashes;
- minimum compatible game build;
- deprecation status;
- replacement package, if available.

Missing tile behavior:

- The tile server returns a normal HTTP missing response for unknown package
  files.
- The gameplay client reports the missing tile and does not invent terrain.
- The gameplay server can block travel, keep the player in the loaded region, or
  route them to a fallback boundary experience depending on the game mode.

## Public Hosting Decision

For MVP and internal closed testing, the LAN-hosted `Agrarian-TileServer` VM is
acceptable because it proves the end-to-end tile lookup, download, cache, and
redownload flow at no recurring cloud cost.

Before broader public testing, move public tile delivery to an external cloud
host, CDN-backed static bucket, or equivalent public edge service. The home VM
can remain the staging publisher, but public testers should not depend on home
uplink reliability once the test group grows.

## Client Cache Layout

Use a deterministic cache layout so downloads can be validated and scrubbed
without touching player saves.

Recommended layout:

```text
<cache_root>/
  manifests/
    manifest.<manifest_version>.json
  registries/
    tile_registry.<registry_version>.json
  tiles/
    <tile_id>/
      v<package_version>/
        package.json
        files...
        SHA256SUMS
        .verified
      .partial/
        download-in-progress files
```

Rules:

- Download into `.partial`.
- Verify all hashes before promotion.
- Promote atomically by renaming into the final version folder.
- Write `.verified` only after checksums pass.
- Never store player inventory, structures, claims, or depletion state inside
  the tile package cache.

## Cache Retention

The cache should keep movement smooth without growing forever.

MVP policy:

- Always pin the active tile.
- Pin immediate neighbors around the player.
- Keep recently visited verified tiles until a configurable size or age limit is
  reached.
- Prefer deleting the oldest unpinned verified tile packages first.
- Never delete `.partial` downloads that are currently active.
- Remove stale `.partial` folders on startup if no download owns them.
- Never delete server-authoritative save data or dirty local session data as
  part of terrain cache cleanup.

Initial settings can be conservative and adjusted after real package sizes are
known:

- minimum pinned set: current tile plus immediate neighbors;
- retention age target: 7-30 days for unpinned packages;
- retention size target: configurable per build and storage class.

## Redownload And Revalidation

At startup and before loading a tile:

- confirm the required package version exists locally;
- confirm `.verified` exists;
- optionally recheck hashes for critical files;
- redownload missing or failed files;
- delete and rebuild corrupt package folders instead of trying to repair them in
  place.

When a manifest marks a package deprecated:

- keep using it if the active server/session requires it;
- download the replacement package in the background when possible;
- migrate only through an explicit server-approved flow.

## Source Data Direction

The source-data strategy should prefer authoritative, affordable, scriptable
datasets with clear licenses.

MVP source families:

- elevation: USGS DEM sources for Ground Zero and other US tiles;
- bathymetry/ocean depth: NOAA or global bathymetry sources as coastal/ocean
  tiles become relevant;
- rivers, lakes, and coastline: USGS hydrography for US tiles, plus global
  hydro/coastline candidates for non-US regions;
- roads and human map features: OpenStreetMap-derived data where license and
  attribution requirements can be met;
- land cover and biome hints: NLCD for US tiles and global land-cover datasets
  for later regions;
- weather/climate context: online weather snapshots for current regional
  weather, plus long-term climate normals for biome/resource inference.

Detailed licensing and attribution review is required before shipping any
source-derived data publicly.

## Single-Tile Import Pipeline

The automated pipeline for one 1 km tile should be:

1. Select or create a `terrain_tiles` registry record.
2. Resolve required source datasets.
3. Download source files into an uncommitted local/source-data cache.
4. Clip each source to the tile bounds plus edge buffer.
5. Normalize projection and vertical units.
6. Generate Unreal-compatible heightmap and metadata.
7. Analyze landforms, slope, water, and shoreline.
8. Generate biome and natural-resource hint layers.
9. Import or update the Unreal landscape package.
10. Run World Partition and map-check validation.
11. Run edge, water, slope, and missing-data QA.
12. Build an immutable tile delivery package.
13. Publish the package only after validation passes.
14. Update the manifest and registry.

The current Ground Zero scripts are the prototype foundation for this flow:

- `Scripts/acquire_ground_zero_dem.py`
- `Scripts/extract_ground_zero_dem_subset.py`
- `Scripts/convert_ground_zero_dem_to_unreal_heightmap.py`
- `Scripts/prototype_ground_zero_terrain.py`
- `Scripts/setup_ground_zero_terrain_map.py`
- `Scripts/setup_ground_zero_demo_map.py`
- `Scripts/analyze_ground_zero_landforms.py`
- `Scripts/analyze_ground_zero_water.py`
- `Scripts/verify_ground_zero_neighbor_edges.py`
- `Scripts/build_ground_zero_tile_delivery_package.sh`
- `Scripts/verify_tile_delivery_client.sh`

## Biome And Resource Inference

Biome and resource inference should produce hints, not final gameplay authority.
The gameplay server and content rules decide what actually spawns, depletes, or
persists.

Inputs:

- elevation;
- slope and aspect;
- water coverage;
- shoreline distance;
- land cover;
- climate and weather context;
- soil/geology/resource datasets when available;
- human map features where relevant.

Outputs:

- primary and secondary biome tags;
- water availability hints;
- forage/wood/stone/clay/mineral likelihood hints;
- traversal risk hints;
- wildlife habitat hints;
- farming suitability hints.

These outputs belong in the tile registry/package metadata. Player-harvested
resources and depleted nodes belong in persistence.

## QA Gates

A tile cannot publish until it passes the relevant checks:

- bounds are exactly 1000 m x 1000 m in the projected coordinate system;
- center latitude/longitude falls inside the tile bounds;
- required source datasets are present or explicitly waived;
- package has generation and package versions;
- content hash exists and changes for new package versions;
- World Partition compatibility flag is true;
- map check passes for the generated tile map/package;
- slopes outside accepted gameplay ranges are flagged;
- water and shoreline masks are present where expected;
- generated neighbor records are reciprocal;
- shared edges match within tolerance where neighbors exist;
- missing neighbor edges are recorded as open edges;
- client download and checksum verification pass;
- delete/redownload recovery passes for at least one package file.

## World Partition Requirements

World Partition is a hard requirement, not an optional optimization. Each tile
package must be shaped so it can load around the player without requiring the
entire world or region to be present.

Requirements:

- tile packages use stable tile-local placement;
- adjacent tile offsets are deterministic;
- generated content is grouped by tile;
- large generated artifacts stay out of Git unless explicitly accepted under
  the repository storage policy;
- tests include at least one command-mode Unreal load/map-check path;
- future multiplayer servers can decide which tile versions are active before
  clients stream them.

## Open Questions

- Final global tile naming format beyond the Ground Zero UTM prototype.
- Exact cache size defaults for investor/demo, closed test, and public builds.
- External public tile hosting provider for broader closed testing.
- Final licensed global source datasets for hydrography, land cover, roads,
  bathymetry, and geology.
- Save migration UX when a populated tile receives improved terrain.
