# Backup Expectations

Agrarian needs layered backups because the project uses GitHub, Unraid project
storage, large local source data, generated Unreal files, and build VMs. Git is
not enough by itself, and VM snapshots are not backups by themselves.

## Goals

- Recover from accidental file edits or deletes within the same day.
- Recover from bad changes discovered days or weeks later.
- Recover the whole project if DevBox, a VM, or a local workstation fails.
- Keep raw terrain/source data and generated build artifacts out of GitHub while
  still protecting them.
- Keep backup cost low by using Linastorage/NAS first and paid cloud storage
  only when needed.
- Prove restores regularly so backups are known-good before an emergency.

## Storage Roles

- GitHub stores source code, configuration, scripts, docs, Unreal project
  metadata, and curated Git LFS assets.
- DevBox is the primary working storage for the Unreal project and large local
  project data.
- Linastorage is the first backup target for project snapshots, VM backups, and
  recoverable deleted files.
- A future offsite/cloud backup should protect only critical archives that
  cannot be recreated from GitHub plus Linastorage.

## Project Folder Backups

The primary project path is:

```text
/mnt/projects/AgrarianGameBulid
\\DevBox\projects\AgrarianGameBulid
```

Back up the project working folder to Linastorage incrementally throughout the
day when changes exist.

Recommended near-term cadence:

- Every 2 hours during active development, only if changes are detected.
- One daily retained snapshot.
- One weekly retained snapshot.
- One monthly retained snapshot.
- Manual snapshot before major Unreal upgrades, plugin installs, mass asset
  moves, terrain import rewrites, migration scripts, or build-pipeline changes.

Recommended retention:

- Frequent snapshots: 7 days.
- Daily snapshots: 30 days.
- Weekly snapshots: 12 weeks.
- Monthly snapshots: 12 months.
- Manual pre-change snapshots: keep until the related change has been stable for
  at least 30 days.

Deleted files should remain recoverable for at least 30 days. For important
assets, terrain source inputs, or investor-demo materials, prefer 90 days.

## What To Include

Include:

- The full Unreal project source tree.
- `.uproject`, `Config/`, `Source/`, `Content/`, `Plugins/`, `Scripts/`,
  `Docs/`, `Data/`, and roadmap files.
- Git metadata when practical, because it helps recover unpushed local work.
- Local source data intentionally excluded from Git, including raw DEM/GIS
  inputs under `Data/Terrain/Sources/`.
- Build scripts, packaging scripts, manifests, checksums, and operational docs.

Exclude or deprioritize:

- `DerivedDataCache/`
- `Intermediate/`
- `Saved/`
- Most generated `Binaries/`
- Temporary logs and editor crash dumps.
- Packaged builds that can be regenerated, unless the build was sent to
  investors/testers or used as a release candidate.

Investor/tester builds should be archived separately with version, commit hash,
build date, target platform, and checksum.

## Backup Method

Use a snapshot tool that supports incremental storage and point-in-time restore,
such as `restic`, `borg`, ZFS snapshots replicated to Linastorage, or a careful
`rsync --link-dest` snapshot layout.

For this environment, the preferred pattern is:

1. Build a complete snapshot locally or on DevBox under a temporary
   `.incomplete-*` path.
2. Write a manifest and checksums.
3. Verify checksums before touching Linastorage.
4. Copy the completed snapshot to Linastorage under `.incomplete-*`.
5. Verify checksums on Linastorage.
6. Atomically rename the Linastorage snapshot into the retained snapshot set.
7. Prune old snapshots only after the new snapshot verifies.

This avoids treating partial SMB copies as valid backups.

## VM Backups

Back up the VMs that support development, not just the project folder.

Minimum VM scope:

- `Ubuntu-Codex`
- `Windows-Builder`
- Unraid VM definitions, NVRAM/OVMF vars, and related libvirt XML.
- Unraid flash/config files needed to recreate shares, VM definitions, and
  network settings.
- Any helper scripts used by Codex or Windows-Builder.

VM image backups should not be simple live file copies of running virtual disks.
Use one of these approaches:

- Unraid VM backup tooling that quiesces or shuts down guests.
- QEMU guest-agent filesystem freeze/thaw plus a consistent snapshot.
- A scheduled maintenance window that shuts down the VM, copies the disk image,
  verifies it, then starts the VM again.

Recommended VM cadence:

- Weekly full backup for Windows-Builder and Ubuntu-Codex.
- Manual backup before GPU passthrough changes, Unreal/Visual Studio upgrades,
  major Windows updates, VM disk resizing, or network driver changes.
- Keep at least 4 weekly VM backups and 3 monthly VM backups.
- Keep manual pre-change VM backups until the changed VM has been stable for at
  least 30 days.

## Restore Testing

Backups are not considered complete until restore has been tested.

Minimum restore tests:

- Weekly: restore one small file from the latest project snapshot to a temporary
  folder and verify checksum or content.
- Monthly: restore the repository snapshot to a scratch location and confirm
  Git status/history are usable.
- Monthly: list or mount the latest VM backup enough to prove it is readable.
- Quarterly: perform a fuller restore drill of the project folder plus one VM
  definition/disk to a non-production test path.

Record restore tests in `Docs/Ops/BackupRestoreTestLog.md` with snapshot path,
date, result, evidence, and cleanup status. Handoff notes can summarize the
latest result, but the durable restore-test record belongs in the log file.

## Security

- Do not put NAS passwords, private SSH keys, GitHub tokens, or other secrets in
  the repository or handoff files.
- Store backup credentials in root-owned files or a host secret store.
- Restrict Linastorage backup shares so normal project users cannot delete all
  retained snapshots.
- Prefer append-only or snapshot-protected NAS retention if Linastorage supports
  it.
- Alert on repeated backup failures, stale latest snapshot age, or incomplete
  snapshot directories.

## Near-Term Implementation Plan

1. Confirm the exact Linastorage share/path for Agrarian backups.
2. Create a project backup script with local staging, manifest, checksum
   verification, and Linastorage atomic publish.
3. Add a systemd timer or Unraid scheduled task that runs every 2 hours during
   active development and exits quickly when no project changes are detected.
4. Add retention pruning for frequent, daily, weekly, monthly, and manual
   snapshots.
5. Add a restore-test script for one file and one repository snapshot.
6. Add VM backup jobs for Windows-Builder and Ubuntu-Codex using a quiesced or
   powered-down backup method.
7. Document the first successful restore test in the handoff log.
