Add Earth-scale tile streaming design
This commit is contained in:
@@ -0,0 +1,435 @@
|
||||
# Earth-Scale Terrain And Tile Streaming Design
|
||||
|
||||
## Purpose
|
||||
|
||||
Agrarian's long-term world is intended to grow toward Earth scale while still
|
||||
remaining playable, patchable, and affordable during the MVP years. The terrain
|
||||
system must support real-world terrain, ocean depth, water, biomes, and resource
|
||||
clues without requiring the full planet to be generated, stored, or shipped at
|
||||
once.
|
||||
|
||||
The MVP proves the contract with the Ground Zero tile and immediate-neighbor
|
||||
metadata. The design must scale from that single 1 km x 1 km tile to hundreds of
|
||||
millions of possible tiles over time.
|
||||
|
||||
## Core Principles
|
||||
|
||||
- The authoritative unit is a 1 km x 1 km logical terrain tile.
|
||||
- Tiles are generated, validated, packaged, published, cached, and retired by
|
||||
version.
|
||||
- Client machines download only the tiles they need.
|
||||
- Player-made world state is stored separately from source terrain packages.
|
||||
- Terrain packages are immutable once published.
|
||||
- World Partition compatibility is mandatory for every terrain decision.
|
||||
- The MVP can use a simple static HTTP tile server, but the contract must not
|
||||
block a future registry API, CDN, or regional tile service.
|
||||
|
||||
## Scale Target
|
||||
|
||||
Earth's surface area implies roughly 510-520 million 1 km x 1 km tile slots.
|
||||
Agrarian will not build all of those tiles up front. The intended growth model
|
||||
is incremental:
|
||||
|
||||
- start with Ground Zero;
|
||||
- add nearby playable tiles around Ground Zero;
|
||||
- add new regions as content, testing, and player demand justify them;
|
||||
- keep unpublished regions as registry placeholders until source data and
|
||||
generation work are scheduled;
|
||||
- allow improved source data and generation passes to publish new tile package
|
||||
versions without destroying existing player history.
|
||||
|
||||
## Current MVP Tile
|
||||
|
||||
The first tile uses the existing UTM zone 10N prototype ID:
|
||||
|
||||
```text
|
||||
gz_us_ca_pacifica_utm10n_e544_n4160
|
||||
```
|
||||
|
||||
It represents the selected Ground Zero 1 km x 1 km MVP tile. The current package
|
||||
is published through the dedicated `Agrarian-TileServer` VM:
|
||||
|
||||
```text
|
||||
http://maps.agrariangame.com:18080
|
||||
```
|
||||
|
||||
Current static endpoints:
|
||||
|
||||
- `/health`
|
||||
- `/manifest.json`
|
||||
- `/ground_zero_tiles.json`
|
||||
- `/schemas/tile_registry.schema.json`
|
||||
- `/tiles/<tile_id>/v<package_version>/...`
|
||||
|
||||
## Canonical Tile Identity
|
||||
|
||||
Each tile needs a stable logical identity and enough coordinate metadata to
|
||||
survive future grid changes. The MVP tile ID encodes location, projection, and
|
||||
1 km lower-left grid corner. Long term, the registry should treat the tile ID as
|
||||
an opaque key and store coordinate details in fields.
|
||||
|
||||
Required identity fields:
|
||||
|
||||
- `tile_id`
|
||||
- `grid_scheme`
|
||||
- `projection`
|
||||
- `projected_bounds_m`
|
||||
- `wgs84_bounds`
|
||||
- `center_latitude`
|
||||
- `center_longitude`
|
||||
- `tile_size_m`
|
||||
- `generation_version`
|
||||
- `package_version`
|
||||
|
||||
The Ground Zero UTM format is acceptable for the MVP. Before generating many
|
||||
regions, the project should add a global grid naming layer that can coexist with
|
||||
regional projected grids. That prevents a future UTM-zone, dateline, or polar
|
||||
decision from forcing player save records to change tile IDs.
|
||||
|
||||
## Coordinate Strategy
|
||||
|
||||
Real-world latitude and longitude should not map directly to Unreal world
|
||||
coordinates. The registry maps real-world coordinates to logical tiles, then
|
||||
each tile package imports into Unreal using tile-local metric coordinates.
|
||||
|
||||
MVP mapping:
|
||||
|
||||
- Ground Zero is the local playable origin.
|
||||
- The tile is 1000 m x 1000 m in its projected coordinate system.
|
||||
- Neighbor tiles are positioned by cardinal/intercardinal 1 km offsets.
|
||||
- Unreal content uses local tile placement and World Partition loading, not
|
||||
raw latitude/longitude transforms.
|
||||
|
||||
Long-term mapping:
|
||||
|
||||
- The registry resolves player/world coordinates to tile IDs.
|
||||
- Tile packages carry their projection metadata and real-world bounds.
|
||||
- The gameplay server is authoritative for active tile package versions.
|
||||
- Unreal World Partition manages loaded cells around the player.
|
||||
- Origin shifting and region-local coordinate roots are used before world
|
||||
coordinates become too large for precision-sensitive gameplay.
|
||||
|
||||
## Projection, Dateline, And Polar Handling
|
||||
|
||||
No single flat projection handles the entire planet cleanly at 1 km gameplay
|
||||
resolution. Agrarian should use local projected coordinate systems for terrain
|
||||
generation and store WGS84 bounds for global lookup.
|
||||
|
||||
Rules:
|
||||
|
||||
- Use metric projected coordinates when generating a tile package.
|
||||
- Store the projection and source coordinate reference system in the registry.
|
||||
- Store WGS84 bounds and center coordinates for lookup, maps, and search.
|
||||
- Treat UTM zone edges, the antimeridian, and polar regions as explicit grid
|
||||
boundary cases.
|
||||
- Do not publish a boundary tile until neighbor and edge validation rules are
|
||||
satisfied for that grid scheme.
|
||||
|
||||
The MVP does not need to solve polar and dateline regions, but the database
|
||||
schema and file package metadata must not assume every tile is in one UTM zone.
|
||||
|
||||
## Tile Adjacency And Stitching
|
||||
|
||||
Each generated tile tracks up to eight neighbors:
|
||||
|
||||
```text
|
||||
N, NE, E, SE, S, SW, W, NW
|
||||
```
|
||||
|
||||
Adjacency records are used for:
|
||||
|
||||
- terrain edge validation;
|
||||
- water and shoreline continuity;
|
||||
- World Partition placement;
|
||||
- client prefetch;
|
||||
- gameplay server interest management;
|
||||
- future road, river, fence, and settlement continuity.
|
||||
|
||||
Stitching rules:
|
||||
|
||||
- Neighbor records must be reciprocal once both tiles exist.
|
||||
- Shared elevation edges must agree within the active generation tolerance.
|
||||
- Water polygons and shoreline masks must not create visible gaps at edges.
|
||||
- River and road features crossing tile borders must have matching continuation
|
||||
metadata.
|
||||
- A tile may publish with an "open edge" only if the neighbor is unknown or not
|
||||
generated yet.
|
||||
- Once a neighboring tile exists, both sides must pass edge QA before either
|
||||
package is promoted beyond validation for that shared boundary.
|
||||
|
||||
## Tile Versioning
|
||||
|
||||
Terrain packages are immutable by version. A published package is never edited
|
||||
in place.
|
||||
|
||||
Create a new package version when any of these change:
|
||||
|
||||
- elevation or bathymetry source data;
|
||||
- coastline, river, lake, road, or land-cover source data;
|
||||
- terrain generation code;
|
||||
- landscape import settings;
|
||||
- biome or resource inference inputs;
|
||||
- collision or navigation-relevant terrain output;
|
||||
- package layout or required client version.
|
||||
|
||||
Player and world persistence records reference:
|
||||
|
||||
- logical `tile_id`;
|
||||
- active `package_version`;
|
||||
- tile-local positions;
|
||||
- placed object or resource state records owned by persistence, not the tile
|
||||
package.
|
||||
|
||||
Deprecated packages remain available until active saves have migrated or no
|
||||
supported server requires them. Save migration should be explicit and logged; it
|
||||
must not silently reinterpret player-built structures against a changed terrain
|
||||
surface.
|
||||
|
||||
## Server Delivery Protocol
|
||||
|
||||
The MVP tile server is a static nginx service. The contract is intentionally
|
||||
simple:
|
||||
|
||||
1. Client requests `/manifest.json`.
|
||||
2. Manifest points to tile registry JSON and schema.
|
||||
3. Client requests the tile registry file.
|
||||
4. Client resolves a required `tile_id` and `package_version`.
|
||||
5. Client downloads listed package files.
|
||||
6. Client verifies checksums.
|
||||
7. Client marks the local package as verified.
|
||||
8. Gameplay server decides whether that tile package is valid for the current
|
||||
session.
|
||||
|
||||
Future server versions can replace static JSON with an API, but should preserve
|
||||
the same concepts:
|
||||
|
||||
- manifest version;
|
||||
- registry version;
|
||||
- tile ID;
|
||||
- package version;
|
||||
- package file list;
|
||||
- content hashes;
|
||||
- minimum compatible game build;
|
||||
- deprecation status;
|
||||
- replacement package, if available.
|
||||
|
||||
Missing tile behavior:
|
||||
|
||||
- The tile server returns a normal HTTP missing response for unknown package
|
||||
files.
|
||||
- The gameplay client reports the missing tile and does not invent terrain.
|
||||
- The gameplay server can block travel, keep the player in the loaded region, or
|
||||
route them to a fallback boundary experience depending on the game mode.
|
||||
|
||||
## Public Hosting Decision
|
||||
|
||||
For MVP and internal closed testing, the LAN-hosted `Agrarian-TileServer` VM is
|
||||
acceptable because it proves the end-to-end tile lookup, download, cache, and
|
||||
redownload flow at no recurring cloud cost.
|
||||
|
||||
Before broader public testing, move public tile delivery to an external cloud
|
||||
host, CDN-backed static bucket, or equivalent public edge service. The home VM
|
||||
can remain the staging publisher, but public testers should not depend on home
|
||||
uplink reliability once the test group grows.
|
||||
|
||||
## Client Cache Layout
|
||||
|
||||
Use a deterministic cache layout so downloads can be validated and scrubbed
|
||||
without touching player saves.
|
||||
|
||||
Recommended layout:
|
||||
|
||||
```text
|
||||
<cache_root>/
|
||||
manifests/
|
||||
manifest.<manifest_version>.json
|
||||
registries/
|
||||
tile_registry.<registry_version>.json
|
||||
tiles/
|
||||
<tile_id>/
|
||||
v<package_version>/
|
||||
package.json
|
||||
files...
|
||||
SHA256SUMS
|
||||
.verified
|
||||
.partial/
|
||||
download-in-progress files
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Download into `.partial`.
|
||||
- Verify all hashes before promotion.
|
||||
- Promote atomically by renaming into the final version folder.
|
||||
- Write `.verified` only after checksums pass.
|
||||
- Never store player inventory, structures, claims, or depletion state inside
|
||||
the tile package cache.
|
||||
|
||||
## Cache Retention
|
||||
|
||||
The cache should keep movement smooth without growing forever.
|
||||
|
||||
MVP policy:
|
||||
|
||||
- Always pin the active tile.
|
||||
- Pin immediate neighbors around the player.
|
||||
- Keep recently visited verified tiles until a configurable size or age limit is
|
||||
reached.
|
||||
- Prefer deleting the oldest unpinned verified tile packages first.
|
||||
- Never delete `.partial` downloads that are currently active.
|
||||
- Remove stale `.partial` folders on startup if no download owns them.
|
||||
- Never delete server-authoritative save data or dirty local session data as
|
||||
part of terrain cache cleanup.
|
||||
|
||||
Initial settings can be conservative and adjusted after real package sizes are
|
||||
known:
|
||||
|
||||
- minimum pinned set: current tile plus immediate neighbors;
|
||||
- retention age target: 7-30 days for unpinned packages;
|
||||
- retention size target: configurable per build and storage class.
|
||||
|
||||
## Redownload And Revalidation
|
||||
|
||||
At startup and before loading a tile:
|
||||
|
||||
- confirm the required package version exists locally;
|
||||
- confirm `.verified` exists;
|
||||
- optionally recheck hashes for critical files;
|
||||
- redownload missing or failed files;
|
||||
- delete and rebuild corrupt package folders instead of trying to repair them in
|
||||
place.
|
||||
|
||||
When a manifest marks a package deprecated:
|
||||
|
||||
- keep using it if the active server/session requires it;
|
||||
- download the replacement package in the background when possible;
|
||||
- migrate only through an explicit server-approved flow.
|
||||
|
||||
## Source Data Direction
|
||||
|
||||
The source-data strategy should prefer authoritative, affordable, scriptable
|
||||
datasets with clear licenses.
|
||||
|
||||
MVP source families:
|
||||
|
||||
- elevation: USGS DEM sources for Ground Zero and other US tiles;
|
||||
- bathymetry/ocean depth: NOAA or global bathymetry sources as coastal/ocean
|
||||
tiles become relevant;
|
||||
- rivers, lakes, and coastline: USGS hydrography for US tiles, plus global
|
||||
hydro/coastline candidates for non-US regions;
|
||||
- roads and human map features: OpenStreetMap-derived data where license and
|
||||
attribution requirements can be met;
|
||||
- land cover and biome hints: NLCD for US tiles and global land-cover datasets
|
||||
for later regions;
|
||||
- weather/climate context: online weather snapshots for current regional
|
||||
weather, plus long-term climate normals for biome/resource inference.
|
||||
|
||||
Detailed licensing and attribution review is required before shipping any
|
||||
source-derived data publicly.
|
||||
|
||||
## Single-Tile Import Pipeline
|
||||
|
||||
The automated pipeline for one 1 km tile should be:
|
||||
|
||||
1. Select or create a `terrain_tiles` registry record.
|
||||
2. Resolve required source datasets.
|
||||
3. Download source files into an uncommitted local/source-data cache.
|
||||
4. Clip each source to the tile bounds plus edge buffer.
|
||||
5. Normalize projection and vertical units.
|
||||
6. Generate Unreal-compatible heightmap and metadata.
|
||||
7. Analyze landforms, slope, water, and shoreline.
|
||||
8. Generate biome and natural-resource hint layers.
|
||||
9. Import or update the Unreal landscape package.
|
||||
10. Run World Partition and map-check validation.
|
||||
11. Run edge, water, slope, and missing-data QA.
|
||||
12. Build an immutable tile delivery package.
|
||||
13. Publish the package only after validation passes.
|
||||
14. Update the manifest and registry.
|
||||
|
||||
The current Ground Zero scripts are the prototype foundation for this flow:
|
||||
|
||||
- `Scripts/acquire_ground_zero_dem.py`
|
||||
- `Scripts/extract_ground_zero_dem_subset.py`
|
||||
- `Scripts/convert_ground_zero_dem_to_unreal_heightmap.py`
|
||||
- `Scripts/prototype_ground_zero_terrain.py`
|
||||
- `Scripts/setup_ground_zero_terrain_map.py`
|
||||
- `Scripts/setup_ground_zero_demo_map.py`
|
||||
- `Scripts/analyze_ground_zero_landforms.py`
|
||||
- `Scripts/analyze_ground_zero_water.py`
|
||||
- `Scripts/verify_ground_zero_neighbor_edges.py`
|
||||
- `Scripts/build_ground_zero_tile_delivery_package.sh`
|
||||
- `Scripts/verify_tile_delivery_client.sh`
|
||||
|
||||
## Biome And Resource Inference
|
||||
|
||||
Biome and resource inference should produce hints, not final gameplay authority.
|
||||
The gameplay server and content rules decide what actually spawns, depletes, or
|
||||
persists.
|
||||
|
||||
Inputs:
|
||||
|
||||
- elevation;
|
||||
- slope and aspect;
|
||||
- water coverage;
|
||||
- shoreline distance;
|
||||
- land cover;
|
||||
- climate and weather context;
|
||||
- soil/geology/resource datasets when available;
|
||||
- human map features where relevant.
|
||||
|
||||
Outputs:
|
||||
|
||||
- primary and secondary biome tags;
|
||||
- water availability hints;
|
||||
- forage/wood/stone/clay/mineral likelihood hints;
|
||||
- traversal risk hints;
|
||||
- wildlife habitat hints;
|
||||
- farming suitability hints.
|
||||
|
||||
These outputs belong in the tile registry/package metadata. Player-harvested
|
||||
resources and depleted nodes belong in persistence.
|
||||
|
||||
## QA Gates
|
||||
|
||||
A tile cannot publish until it passes the relevant checks:
|
||||
|
||||
- bounds are exactly 1000 m x 1000 m in the projected coordinate system;
|
||||
- center latitude/longitude falls inside the tile bounds;
|
||||
- required source datasets are present or explicitly waived;
|
||||
- package has generation and package versions;
|
||||
- content hash exists and changes for new package versions;
|
||||
- World Partition compatibility flag is true;
|
||||
- map check passes for the generated tile map/package;
|
||||
- slopes outside accepted gameplay ranges are flagged;
|
||||
- water and shoreline masks are present where expected;
|
||||
- generated neighbor records are reciprocal;
|
||||
- shared edges match within tolerance where neighbors exist;
|
||||
- missing neighbor edges are recorded as open edges;
|
||||
- client download and checksum verification pass;
|
||||
- delete/redownload recovery passes for at least one package file.
|
||||
|
||||
## World Partition Requirements
|
||||
|
||||
World Partition is a hard requirement, not an optional optimization. Each tile
|
||||
package must be shaped so it can load around the player without requiring the
|
||||
entire world or region to be present.
|
||||
|
||||
Requirements:
|
||||
|
||||
- tile packages use stable tile-local placement;
|
||||
- adjacent tile offsets are deterministic;
|
||||
- generated content is grouped by tile;
|
||||
- large generated artifacts stay out of Git unless explicitly accepted under
|
||||
the repository storage policy;
|
||||
- tests include at least one command-mode Unreal load/map-check path;
|
||||
- future multiplayer servers can decide which tile versions are active before
|
||||
clients stream them.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Final global tile naming format beyond the Ground Zero UTM prototype.
|
||||
- Exact cache size defaults for investor/demo, closed test, and public builds.
|
||||
- External public tile hosting provider for broader closed testing.
|
||||
- Final licensed global source datasets for hydrography, land cover, roads,
|
||||
bathymetry, and geology.
|
||||
- Save migration UX when a populated tile receives improved terrain.
|
||||
Reference in New Issue
Block a user