179 lines
6.7 KiB
Markdown
179 lines
6.7 KiB
Markdown
# Backup Expectations
|
|
|
|
Agrarian needs layered backups because the project uses GitHub, Unraid project
|
|
storage, large local source data, generated Unreal files, and build VMs. Git is
|
|
not enough by itself, and VM snapshots are not backups by themselves.
|
|
|
|
## Goals
|
|
|
|
- Recover from accidental file edits or deletes within the same day.
|
|
- Recover from bad changes discovered days or weeks later.
|
|
- Recover the whole project if DevBox, a VM, or a local workstation fails.
|
|
- Keep raw terrain/source data and generated build artifacts out of GitHub while
|
|
still protecting them.
|
|
- Keep backup cost low by using Linastorage/NAS first and paid cloud storage
|
|
only when needed.
|
|
- Prove restores regularly so backups are known-good before an emergency.
|
|
|
|
## Storage Roles
|
|
|
|
- GitHub stores source code, configuration, scripts, docs, Unreal project
|
|
metadata, and curated Git LFS assets.
|
|
- DevBox is the primary working storage for the Unreal project and large local
|
|
project data.
|
|
- Linastorage is the first backup target for project snapshots, VM backups, and
|
|
recoverable deleted files.
|
|
- A future offsite/cloud backup should protect only critical archives that
|
|
cannot be recreated from GitHub plus Linastorage.
|
|
|
|
## Project Folder Backups
|
|
|
|
The primary project path is:
|
|
|
|
```text
|
|
/mnt/projects/AgrarianGameBulid
|
|
\\DevBox\projects\AgrarianGameBulid
|
|
```
|
|
|
|
Back up the project working folder to Linastorage incrementally throughout the
|
|
day when changes exist.
|
|
|
|
Recommended near-term cadence:
|
|
|
|
- Every 2 hours during active development, only if changes are detected.
|
|
- One daily retained snapshot.
|
|
- One weekly retained snapshot.
|
|
- One monthly retained snapshot.
|
|
- Manual snapshot before major Unreal upgrades, plugin installs, mass asset
|
|
moves, terrain import rewrites, migration scripts, or build-pipeline changes.
|
|
|
|
Recommended retention:
|
|
|
|
- Frequent snapshots: 7 days.
|
|
- Daily snapshots: 30 days.
|
|
- Weekly snapshots: 12 weeks.
|
|
- Monthly snapshots: 12 months.
|
|
- Manual pre-change snapshots: keep until the related change has been stable for
|
|
at least 30 days.
|
|
|
|
Deleted files should remain recoverable for at least 30 days. For important
|
|
assets, terrain source inputs, or investor-demo materials, prefer 90 days.
|
|
|
|
## What To Include
|
|
|
|
Include:
|
|
|
|
- The full Unreal project source tree.
|
|
- `.uproject`, `Config/`, `Source/`, `Content/`, `Plugins/`, `Scripts/`,
|
|
`Docs/`, `Data/`, and roadmap files.
|
|
- Git metadata when practical, because it helps recover unpushed local work.
|
|
- Local source data intentionally excluded from Git, including raw DEM/GIS
|
|
inputs under `Data/Terrain/Sources/`.
|
|
- Build scripts, packaging scripts, manifests, checksums, and operational docs.
|
|
|
|
Exclude or deprioritize:
|
|
|
|
- `DerivedDataCache/`
|
|
- `Intermediate/`
|
|
- `Saved/`
|
|
- Most generated `Binaries/`
|
|
- Temporary logs and editor crash dumps.
|
|
- Packaged builds that can be regenerated, unless the build was sent to
|
|
investors/testers or used as a release candidate.
|
|
|
|
Investor/tester builds should be archived separately with version, commit hash,
|
|
build date, target platform, and checksum.
|
|
|
|
## Backup Method
|
|
|
|
Use a snapshot tool that supports incremental storage and point-in-time restore,
|
|
such as `restic`, `borg`, ZFS snapshots replicated to Linastorage, or a careful
|
|
`rsync --link-dest` snapshot layout.
|
|
|
|
For this environment, the preferred pattern is:
|
|
|
|
1. Build a complete snapshot locally or on DevBox under a temporary
|
|
`.incomplete-*` path.
|
|
2. Write a manifest and checksums.
|
|
3. Verify checksums before touching Linastorage.
|
|
4. Copy the completed snapshot to Linastorage under `.incomplete-*`.
|
|
5. Verify checksums on Linastorage.
|
|
6. Atomically rename the Linastorage snapshot into the retained snapshot set.
|
|
7. Prune old snapshots only after the new snapshot verifies.
|
|
|
|
This avoids treating partial SMB copies as valid backups.
|
|
|
|
## VM Backups
|
|
|
|
Back up the VMs that support development, not just the project folder.
|
|
|
|
Minimum VM scope:
|
|
|
|
- `Ubuntu-Codex`
|
|
- `Windows-Builder`
|
|
- Unraid VM definitions, NVRAM/OVMF vars, and related libvirt XML.
|
|
- Unraid flash/config files needed to recreate shares, VM definitions, and
|
|
network settings.
|
|
- Any helper scripts used by Codex or Windows-Builder.
|
|
|
|
VM image backups should not be simple live file copies of running virtual disks.
|
|
Use one of these approaches:
|
|
|
|
- Unraid VM backup tooling that quiesces or shuts down guests.
|
|
- QEMU guest-agent filesystem freeze/thaw plus a consistent snapshot.
|
|
- A scheduled maintenance window that shuts down the VM, copies the disk image,
|
|
verifies it, then starts the VM again.
|
|
|
|
Recommended VM cadence:
|
|
|
|
- Weekly full backup for Windows-Builder and Ubuntu-Codex.
|
|
- Manual backup before GPU passthrough changes, Unreal/Visual Studio upgrades,
|
|
major Windows updates, VM disk resizing, or network driver changes.
|
|
- Keep at least 4 weekly VM backups and 3 monthly VM backups.
|
|
- Keep manual pre-change VM backups until the changed VM has been stable for at
|
|
least 30 days.
|
|
|
|
## Restore Testing
|
|
|
|
Backups are not considered complete until restore has been tested.
|
|
|
|
Minimum restore tests:
|
|
|
|
- Weekly: restore one small file from the latest project snapshot to a temporary
|
|
folder and verify checksum or content.
|
|
- Monthly: restore the repository snapshot to a scratch location and confirm
|
|
Git status/history are usable.
|
|
- Monthly: list or mount the latest VM backup enough to prove it is readable.
|
|
- Quarterly: perform a fuller restore drill of the project folder plus one VM
|
|
definition/disk to a non-production test path.
|
|
|
|
Record restore tests in `Docs/Ops/BackupRestoreTestLog.md` with snapshot path,
|
|
date, result, evidence, and cleanup status. Handoff notes can summarize the
|
|
latest result, but the durable restore-test record belongs in the log file.
|
|
|
|
## Security
|
|
|
|
- Do not put NAS passwords, private SSH keys, GitHub tokens, or other secrets in
|
|
the repository or handoff files.
|
|
- Store backup credentials in root-owned files or a host secret store.
|
|
- Restrict Linastorage backup shares so normal project users cannot delete all
|
|
retained snapshots.
|
|
- Prefer append-only or snapshot-protected NAS retention if Linastorage supports
|
|
it.
|
|
- Alert on repeated backup failures, stale latest snapshot age, or incomplete
|
|
snapshot directories.
|
|
|
|
## Near-Term Implementation Plan
|
|
|
|
1. Confirm the exact Linastorage share/path for Agrarian backups.
|
|
2. Create a project backup script with local staging, manifest, checksum
|
|
verification, and Linastorage atomic publish.
|
|
3. Add a systemd timer or Unraid scheduled task that runs every 2 hours during
|
|
active development and exits quickly when no project changes are detected.
|
|
4. Add retention pruning for frequent, daily, weekly, monthly, and manual
|
|
snapshots.
|
|
5. Add a restore-test script for one file and one repository snapshot.
|
|
6. Add VM backup jobs for Windows-Builder and Ubuntu-Codex using a quiesced or
|
|
powered-down backup method.
|
|
7. Document the first successful restore test in the handoff log.
|