From ccc750d7895cdd103d9e009c1fd25634ecdaa60d Mon Sep 17 00:00:00 2001 From: nathan Date: Thu, 14 May 2026 10:52:35 -0700 Subject: [PATCH] Document backup expectations --- AGRARIAN_DEVELOPMENT_ROADMAP.md | 17 ++- Docs/BackupExpectations.md | 176 ++++++++++++++++++++++++++++++++ 2 files changed, 188 insertions(+), 5 deletions(-) create mode 100644 Docs/BackupExpectations.md diff --git a/AGRARIAN_DEVELOPMENT_ROADMAP.md b/AGRARIAN_DEVELOPMENT_ROADMAP.md index 000fefe..a5bd718 100644 --- a/AGRARIAN_DEVELOPMENT_ROADMAP.md +++ b/AGRARIAN_DEVELOPMENT_ROADMAP.md @@ -173,7 +173,9 @@ Remaining version 0.01 cleanup before moving deeper into new gameplay: - [x] Finish branch naming conventions. - [x] Finish commit message conventions. - [x] Define GitHub/LFS free-tier storage guardrails. -- [ ] Define backup expectations for NAS and repository. +- [x] Define backup expectations for NAS and repository. +- [ ] Implement Linastorage incremental project backup job. +- [ ] Implement quiesced VM backup job for Windows-Builder and Ubuntu-Codex. - [ ] Create repeatable dedicated server build instructions. - [~] Finish required plugin documentation. - [ ] Confirm the project opens cleanly from a fresh checkout, not just the current working share. @@ -310,8 +312,11 @@ Current tooling decisions: - [x] Confirm headless editor target build succeeds. - [ ] Set up Parsec or equivalent real-GPU remote access for occasional Codex visual inspection. - [ ] Stabilize Windows-Builder network/RDP behavior under GPU passthrough. -- [ ] Decide and document VM snapshot cadence before major engine/tool changes. -- [ ] Define Unraid share backup policy. +- [x] Decide and document VM snapshot cadence before major engine/tool changes. +- [x] Define Unraid share backup policy. +- [ ] Implement Linastorage incremental project backup job with deleted-file retention. +- [ ] Implement quiesced VM backup job for Windows-Builder and Ubuntu-Codex. +- [ ] Add recurring restore-test log for project and VM backups. - [ ] Define GitHub branch protection and review rules. - [ ] Add a build log retention policy. - [ ] Add a simple operational runbook for rebooting/recovering Windows-Builder, Ubuntu-Codex, and DevBox. @@ -1410,7 +1415,9 @@ Earliest incomplete foundation items: - [x] Finish branch naming conventions. - [x] Finish commit message conventions. - [x] Define GitHub/LFS free-tier storage guardrails. -- [ ] Define backup expectations for NAS and repository. +- [x] Define backup expectations for NAS and repository. +- [ ] Implement Linastorage incremental project backup job. +- [ ] Implement quiesced VM backup job for Windows-Builder and Ubuntu-Codex. - [ ] Create repeatable dedicated server build instructions. - [~] Finish required plugin documentation. - [ ] Confirm project opens cleanly from a fresh checkout. @@ -1428,4 +1435,4 @@ Earliest incomplete foundation items: Immediate next item: -- [ ] Define backup expectations for NAS and repository. +- [ ] Implement Linastorage incremental project backup job. diff --git a/Docs/BackupExpectations.md b/Docs/BackupExpectations.md new file mode 100644 index 0000000..8049ed9 --- /dev/null +++ b/Docs/BackupExpectations.md @@ -0,0 +1,176 @@ +# Backup Expectations + +Agrarian needs layered backups because the project uses GitHub, Unraid project +storage, large local source data, generated Unreal files, and build VMs. Git is +not enough by itself, and VM snapshots are not backups by themselves. + +## Goals + +- Recover from accidental file edits or deletes within the same day. +- Recover from bad changes discovered days or weeks later. +- Recover the whole project if DevBox, a VM, or a local workstation fails. +- Keep raw terrain/source data and generated build artifacts out of GitHub while + still protecting them. +- Keep backup cost low by using Linastorage/NAS first and paid cloud storage + only when needed. +- Prove restores regularly so backups are known-good before an emergency. + +## Storage Roles + +- GitHub stores source code, configuration, scripts, docs, Unreal project + metadata, and curated Git LFS assets. +- DevBox is the primary working storage for the Unreal project and large local + project data. +- Linastorage is the first backup target for project snapshots, VM backups, and + recoverable deleted files. +- A future offsite/cloud backup should protect only critical archives that + cannot be recreated from GitHub plus Linastorage. + +## Project Folder Backups + +The primary project path is: + +```text +/mnt/projects/AgrarianGameBulid +\\DevBox\projects\AgrarianGameBulid +``` + +Back up the project working folder to Linastorage incrementally throughout the +day when changes exist. + +Recommended near-term cadence: + +- Every 2 hours during active development, only if changes are detected. +- One daily retained snapshot. +- One weekly retained snapshot. +- One monthly retained snapshot. +- Manual snapshot before major Unreal upgrades, plugin installs, mass asset + moves, terrain import rewrites, migration scripts, or build-pipeline changes. + +Recommended retention: + +- Frequent snapshots: 7 days. +- Daily snapshots: 30 days. +- Weekly snapshots: 12 weeks. +- Monthly snapshots: 12 months. +- Manual pre-change snapshots: keep until the related change has been stable for + at least 30 days. + +Deleted files should remain recoverable for at least 30 days. For important +assets, terrain source inputs, or investor-demo materials, prefer 90 days. + +## What To Include + +Include: + +- The full Unreal project source tree. +- `.uproject`, `Config/`, `Source/`, `Content/`, `Plugins/`, `Scripts/`, + `Docs/`, `Data/`, and roadmap files. +- Git metadata when practical, because it helps recover unpushed local work. +- Local source data intentionally excluded from Git, including raw DEM/GIS + inputs under `Data/Terrain/Sources/`. +- Build scripts, packaging scripts, manifests, checksums, and operational docs. + +Exclude or deprioritize: + +- `DerivedDataCache/` +- `Intermediate/` +- `Saved/` +- Most generated `Binaries/` +- Temporary logs and editor crash dumps. +- Packaged builds that can be regenerated, unless the build was sent to + investors/testers or used as a release candidate. + +Investor/tester builds should be archived separately with version, commit hash, +build date, target platform, and checksum. + +## Backup Method + +Use a snapshot tool that supports incremental storage and point-in-time restore, +such as `restic`, `borg`, ZFS snapshots replicated to Linastorage, or a careful +`rsync --link-dest` snapshot layout. + +For this environment, the preferred pattern is: + +1. Build a complete snapshot locally or on DevBox under a temporary + `.incomplete-*` path. +2. Write a manifest and checksums. +3. Verify checksums before touching Linastorage. +4. Copy the completed snapshot to Linastorage under `.incomplete-*`. +5. Verify checksums on Linastorage. +6. Atomically rename the Linastorage snapshot into the retained snapshot set. +7. Prune old snapshots only after the new snapshot verifies. + +This avoids treating partial SMB copies as valid backups. + +## VM Backups + +Back up the VMs that support development, not just the project folder. + +Minimum VM scope: + +- `Ubuntu-Codex` +- `Windows-Builder` +- Unraid VM definitions, NVRAM/OVMF vars, and related libvirt XML. +- Unraid flash/config files needed to recreate shares, VM definitions, and + network settings. +- Any helper scripts used by Codex or Windows-Builder. + +VM image backups should not be simple live file copies of running virtual disks. +Use one of these approaches: + +- Unraid VM backup tooling that quiesces or shuts down guests. +- QEMU guest-agent filesystem freeze/thaw plus a consistent snapshot. +- A scheduled maintenance window that shuts down the VM, copies the disk image, + verifies it, then starts the VM again. + +Recommended VM cadence: + +- Weekly full backup for Windows-Builder and Ubuntu-Codex. +- Manual backup before GPU passthrough changes, Unreal/Visual Studio upgrades, + major Windows updates, VM disk resizing, or network driver changes. +- Keep at least 4 weekly VM backups and 3 monthly VM backups. +- Keep manual pre-change VM backups until the changed VM has been stable for at + least 30 days. + +## Restore Testing + +Backups are not considered complete until restore has been tested. + +Minimum restore tests: + +- Weekly: restore one small file from the latest project snapshot to a temporary + folder and verify checksum or content. +- Monthly: restore the repository snapshot to a scratch location and confirm + Git status/history are usable. +- Monthly: list or mount the latest VM backup enough to prove it is readable. +- Quarterly: perform a fuller restore drill of the project folder plus one VM + definition/disk to a non-production test path. + +Record restore tests in the handoff log with snapshot path, date, and result. + +## Security + +- Do not put NAS passwords, private SSH keys, GitHub tokens, or other secrets in + the repository or handoff files. +- Store backup credentials in root-owned files or a host secret store. +- Restrict Linastorage backup shares so normal project users cannot delete all + retained snapshots. +- Prefer append-only or snapshot-protected NAS retention if Linastorage supports + it. +- Alert on repeated backup failures, stale latest snapshot age, or incomplete + snapshot directories. + +## Near-Term Implementation Plan + +1. Confirm the exact Linastorage share/path for Agrarian backups. +2. Create a project backup script with local staging, manifest, checksum + verification, and Linastorage atomic publish. +3. Add a systemd timer or Unraid scheduled task that runs every 2 hours during + active development and exits quickly when no project changes are detected. +4. Add retention pruning for frequent, daily, weekly, monthly, and manual + snapshots. +5. Add a restore-test script for one file and one repository snapshot. +6. Add VM backup jobs for Windows-Builder and Ubuntu-Codex using a quiesced or + powered-down backup method. +7. Document the first successful restore test in the handoff log.