feat(storage): sqlite online snapshot and disaster recovery rehearsal #67

Merged

yuris merged 1 commit from feat/issue-36-backup into main 2026-09-23 00:58:12 +00:00

Conversation 1 Commits 1 Files changed 7 +3238 -2

yuris commented 2026-09-23 00:56:53 +00:00

Owner

Copy link

Motivation

The journal runs in WAL mode, so a committed transaction lives in life.sqlite3-wal until a checkpoint writes it back into the main file. Copying the live life.sqlite3 alone is therefore not a backup: it loses recently committed transactions at best and yields an inconsistent database at worst. Issue #36 (scenarios D10, D11, D12) requires the supported snapshot procedure, provenance for it, and a restore gate that verifies before a restored node serves anything.

There was no backup verb, no manifest, no restore path and no rehearsal. A restore also has a second half that is easy to miss: a restored journal stopped observing at its capture time, so the node must reconcile unresolved operations, drop the sync cursors the restore invalidated, and record the interval it never observed instead of presenting restored state as continuous history (docs/technical/operations-security.md#backup-and-restore, docs/technical/timeline-consistency.md#external-reconciliation).

Changes

• src/history/backup.rs (new) — the snapshot engine, manifest type, verification and restore gate:
  • create_snapshot runs VACUUM INTO ?1 through the single DatabaseWorker that owns the process's one connection; SQLite serializes the snapshot against the live connection's committed view, so a commit made just before the snapshot is in the file. No second writer or competing connection is opened, and a staging directory inside the source data directory is refused before anything is created.
  • The produced file is verified as it is produced (PRAGMA integrity_check, PRAGMA foreign_key_check, recorded schema version) and removed when it fails.
  • SnapshotManifest records manifest format version, application + binary version, backend + engine version, snapshot file name, sha256:<hex>, capture time and journal schema version, and is written atomically (temp + rename).
  • validate_snapshot verifies manifest presence/shape, file-name agreement, capture-time plausibility, checksum, integrity/foreign keys, schema agreement, and refuses a schema newer than this binary supports (never a destructive downgrade).
  • restore_into refuses a target that is not isolated (an exclusive service lock is taken on the target, an existing target database is refused, unrelated -wal/-shm sidecars are refused), copies to a temp path, verifies the copy's checksum, and renames it into place. It returns the still-held lock so the caller rebaselines before releasing it.
  • snapshot_status reports capture age, size and whether the file still matches its manifest — a stale or failing snapshot is visible.
  • The rejected procedure (main-file-only copy) is documented in the module doc and defended by a test that shows the copy missing the WAL-resident commit.
• src/history/disaster.rs (new) — reconcile_after_restore: reuses recover_pending_operations (the one recovery pass, no remote replay), then in one transaction drops every collection's sync cursor and advances its generation (cursor = NULL, generation + 1, complete = 0, coverage incomplete), and appends one observation_gap event (confirmed_service) whose after_json.coverage covers [snapshot_capture_time, restore_started_at]. No creation or deletion event is synthesized for anything that changed across the gap; the summary reports whether the rebaseline is complete so a caller can refuse to serve.
• src/cli.rs — life backup snapshot --out <dir>, life backup status --out <dir> [--max-age-seconds <n>], life backup restore --manifest <path> [--snapshot <path>] --into <dir> [--database <path>]. Snapshot/restore take the #34 exclusive data-directory lock, so a running life serve owns the directory and is refused rather than joined: snapshotting is an explicit operator action, not a hidden daemon behaviour. Restore verifies before writing, and reconciles when --config is given; without one the rebaseline is left to the next start.
• src/api/mod.rs — serve() completes a pending restore (complete_pending_restore) before the listener accepts a request: a directory restored from a snapshot is rebaselined (same recovery + cursor invalidation + gap) and then its marker cleared. An unreadable marker or a rebaseline that cannot be recorded aborts startup, leaving the marker for explicit resolution.
• src/history/mod.rs, src/history/recovery.rs — module registration and re-exports; RecoverySummary derives Serialize so a rebaseline can report it.
• tests/disaster_recovery_rehearsal.rs (new) — 9 tests covering D10, D11, D12, the isolation refusals, the marker/daemon gate, and the full rehearsal with known markers.

Rejected procedure, recorded as required: copying life.sqlite3 alone while WAL is active is unsafe and is not a supported backup. Use life backup snapshot (SQLite VACUUM INTO through the owning connection).

Verification

• nix develop --command cargo test — all suites green (543 tests, including the 9 new ones).
• nix develop --command cargo clippy --all-targets -- -D warnings and cargo fmt --check — clean.
• Manual smoke of the real binary: life backup snapshot wrote a snapshot + manifest; life backup status reported age/size/checksum: ok; life backup restore --into <isolated dir> restored, verified, and left life.restore-pending.json for the next start; a missing manifest aborted with manifest_missing.
• D10 (d10_snapshot_carries_wal_commits_that_a_main_file_copy_loses): 50 operations and events committed with automatic checkpoints off; the -wal file is non-empty; the snapshot carries all 50 rows and passes integrity_check, while a copy of the main file alone does not hold the newest operation.
• D11 (d11_restore_against_a_differently_timed_calendar_reconciles_and_records_the_gap): a snapshot captured 15 minutes before the restore, restored against a calendar holding states the service never wrote. Integrity and foreign keys pass on the restored file; all three unresolved operations are reconciled on evidence (two to needs_reconciliation, one observed as observed_external with explicit attribution uncertainty); the cursor is dropped and the generation set; the gap event covers exactly [capture, restore]; the event kinds are exactly the seeded fact plus the observation plus the gap, so nothing was fabricated; and the backend received no write at all.
• D12 (d12_corrupt_incomplete_or_unsupported_snapshots_abort_before_restoring): flipped bytes abort with ChecksumMismatch; damage that also survives a regenerated manifest aborts with CorruptedDatabase; a missing manifest aborts with MissingManifest; a manifest describing another file aborts with InvalidManifest; a newer schema aborts with UnsupportedSchemaVersion. No target database and no marker are left behind.
• Rehearsal (rehearsal_recovers_operations_timeline_proposals_outbox_and_idempotency_keys): operations, timeline events, proposals and outbox rows come back with their marker values; replaying a restored operation key returns Replay and a different request hash under the same key is still IdempotencyConflict; the gap is reported and states that history is not complete through it.
• Daemon: daemon_completes_a_pending_restore_before_it_serves starts the real binary over a restored directory and observes the gap recorded, the cursor invalidated and the unresolved operation reconciled by the time /health/live answers, with no backend write; daemon_refuses_to_serve_a_restore_it_cannot_rebaseline shows an unreadable marker aborting startup and being preserved.

Deployment inputs deliberately left out

• Encryption, off-host destination/transfer, snapshot frequency, retention window and key custody — deployment responsibilities, as the operations contract states.
• Radicale's own backup and Hindsight's lifecycle — separate systems, not assumed here.
• Reapplying an erasure manifest before restored data becomes accessible (D12 case C): the offline privacy-erasure procedure itself is not implemented in this codebase, so there is no erasure state to replay yet. Restoration today cannot resurrect erased data it never recorded an erasure for.
• The full CalDAV enumeration that forms a new baseline: this change invalidates the cursors and establishes the generation that enumeration will run in. Enumeration belongs to the sync path, which is still unimplemented (life sync).

Related Issues

Closes #36

## Motivation The journal runs in WAL mode, so a committed transaction lives in `life.sqlite3-wal` until a checkpoint writes it back into the main file. Copying the live `life.sqlite3` alone is therefore **not** a backup: it loses recently committed transactions at best and yields an inconsistent database at worst. Issue #36 (scenarios D10, D11, D12) requires the supported snapshot procedure, provenance for it, and a restore gate that verifies before a restored node serves anything. There was no backup verb, no manifest, no restore path and no rehearsal. A restore also has a second half that is easy to miss: a restored journal stopped observing at its capture time, so the node must reconcile unresolved operations, drop the sync cursors the restore invalidated, and record the interval it never observed instead of presenting restored state as continuous history (`docs/technical/operations-security.md#backup-and-restore`, `docs/technical/timeline-consistency.md#external-reconciliation`). ## Changes - **`src/history/backup.rs` (new)** — the snapshot engine, manifest type, verification and restore gate: - `create_snapshot` runs `VACUUM INTO ?1` **through the single `DatabaseWorker`** that owns the process's one connection; SQLite serializes the snapshot against the live connection's committed view, so a commit made just before the snapshot is in the file. No second writer or competing connection is opened, and a staging directory inside the source data directory is refused before anything is created. - The produced file is verified as it is produced (`PRAGMA integrity_check`, `PRAGMA foreign_key_check`, recorded schema version) and removed when it fails. - `SnapshotManifest` records manifest format version, application + binary version, backend + engine version, snapshot file name, `sha256:<hex>`, capture time and journal schema version, and is written atomically (temp + rename). - `validate_snapshot` verifies manifest presence/shape, file-name agreement, capture-time plausibility, checksum, integrity/foreign keys, schema agreement, and refuses a schema newer than this binary supports (never a destructive downgrade). - `restore_into` refuses a target that is not isolated (an exclusive service lock is taken on the target, an existing target database is refused, unrelated `-wal`/`-shm` sidecars are refused), copies to a temp path, verifies the copy's checksum, and renames it into place. It returns the still-held lock so the caller rebaselines before releasing it. - `snapshot_status` reports capture age, size and whether the file still matches its manifest — a stale *or failing* snapshot is visible. - The rejected procedure (main-file-only copy) is documented in the module doc **and defended by a test** that shows the copy missing the WAL-resident commit. - **`src/history/disaster.rs` (new)** — `reconcile_after_restore`: reuses `recover_pending_operations` (the one recovery pass, no remote replay), then in one transaction drops every collection's sync cursor and advances its generation (`cursor = NULL`, `generation + 1`, `complete = 0`, coverage `incomplete`), and appends one `observation_gap` event (`confirmed_service`) whose `after_json.coverage` covers `[snapshot_capture_time, restore_started_at]`. No creation or deletion event is synthesized for anything that changed across the gap; the summary reports whether the rebaseline is complete so a caller can refuse to serve. - **`src/cli.rs`** — `life backup snapshot --out <dir>`, `life backup status --out <dir> [--max-age-seconds <n>]`, `life backup restore --manifest <path> [--snapshot <path>] --into <dir> [--database <path>]`. Snapshot/restore take the `#34` exclusive data-directory lock, so a running `life serve` owns the directory and is refused rather than joined: snapshotting is an explicit operator action, not a hidden daemon behaviour. Restore verifies before writing, and reconciles when `--config` is given; without one the rebaseline is left to the next start. - **`src/api/mod.rs`** — `serve()` completes a pending restore (`complete_pending_restore`) *before* the listener accepts a request: a directory restored from a snapshot is rebaselined (same recovery + cursor invalidation + gap) and then its marker cleared. An unreadable marker or a rebaseline that cannot be recorded aborts startup, leaving the marker for explicit resolution. - **`src/history/mod.rs`, `src/history/recovery.rs`** — module registration and re-exports; `RecoverySummary` derives `Serialize` so a rebaseline can report it. - **`tests/disaster_recovery_rehearsal.rs` (new)** — 9 tests covering D10, D11, D12, the isolation refusals, the marker/daemon gate, and the full rehearsal with known markers. Rejected procedure, recorded as required: **copying `life.sqlite3` alone while WAL is active is unsafe and is not a supported backup.** Use `life backup snapshot` (SQLite `VACUUM INTO` through the owning connection). ## Verification - `nix develop --command cargo test` — all suites green (543 tests, including the 9 new ones). - `nix develop --command cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` — clean. - Manual smoke of the real binary: `life backup snapshot` wrote a snapshot + manifest; `life backup status` reported age/size/`checksum: ok`; `life backup restore --into <isolated dir>` restored, verified, and left `life.restore-pending.json` for the next start; a missing manifest aborted with `manifest_missing`. - D10 (`d10_snapshot_carries_wal_commits_that_a_main_file_copy_loses`): 50 operations and events committed with automatic checkpoints off; the `-wal` file is non-empty; the snapshot carries all 50 rows and passes `integrity_check`, while a copy of the main file alone does **not** hold the newest operation. - D11 (`d11_restore_against_a_differently_timed_calendar_reconciles_and_records_the_gap`): a snapshot captured 15 minutes before the restore, restored against a calendar holding states the service never wrote. Integrity and foreign keys pass on the restored file; all three unresolved operations are reconciled on evidence (two to `needs_reconciliation`, one observed as `observed_external` with explicit attribution uncertainty); the cursor is dropped and the generation set; the gap event covers exactly `[capture, restore]`; the event kinds are exactly the seeded fact plus the observation plus the gap, so nothing was fabricated; and the backend received no write at all. - D12 (`d12_corrupt_incomplete_or_unsupported_snapshots_abort_before_restoring`): flipped bytes abort with `ChecksumMismatch`; damage that also survives a regenerated manifest aborts with `CorruptedDatabase`; a missing manifest aborts with `MissingManifest`; a manifest describing another file aborts with `InvalidManifest`; a newer schema aborts with `UnsupportedSchemaVersion`. No target database and no marker are left behind. - Rehearsal (`rehearsal_recovers_operations_timeline_proposals_outbox_and_idempotency_keys`): operations, timeline events, proposals and outbox rows come back with their marker values; replaying a restored operation key returns `Replay` and a different request hash under the same key is still `IdempotencyConflict`; the gap is reported and states that history is not complete through it. - Daemon: `daemon_completes_a_pending_restore_before_it_serves` starts the real binary over a restored directory and observes the gap recorded, the cursor invalidated and the unresolved operation reconciled by the time `/health/live` answers, with no backend write; `daemon_refuses_to_serve_a_restore_it_cannot_rebaseline` shows an unreadable marker aborting startup and being preserved. ### Deployment inputs deliberately left out - Encryption, off-host destination/transfer, snapshot frequency, retention window and key custody — deployment responsibilities, as the operations contract states. - Radicale's own backup and Hindsight's lifecycle — separate systems, not assumed here. - Reapplying an erasure manifest before restored data becomes accessible (D12 case C): the offline privacy-erasure procedure itself is not implemented in this codebase, so there is no erasure state to replay yet. Restoration today cannot resurrect erased data it never recorded an erasure for. - The full CalDAV enumeration that forms a new baseline: this change invalidates the cursors and establishes the generation that enumeration will run in. Enumeration belongs to the sync path, which is still unimplemented (`life sync`). ## Related Issues Closes #36

yuris added 1 commit 2026-09-23 00:56:53 +00:00

feat(storage): sqlite online snapshot and disaster recovery rehearsal b783b05021

yuris commented 2026-09-23 00:58:11 +00:00

Author

Owner

Copy link

Coordinator review — APPROVED

Verified b783b05 myself: cargo test 566 passed / 0 failed (9 disaster-recovery tests), cargo clippy --all-targets -- -D warnings 0 errors, cargo fmt --check clean.

What I checked beyond the summary:

• The snapshot goes through SQLite, on the worker's own connection (VACUUM INTO ?1 at src/history/backup.rs:335, documented in the module header and at the call site), and the rejection of the naive copy is evidence-backed rather than asserted: with wal_autocheckpoint=0 the test commits rows that stay in -wal, then shows the snapshot holds the final row while a plain life.sqlite3 copy does not. That is exactly the D10 pair, and it is the difference between a documented rule and a justified one.
• D11's hardest clause is asserted by absence, which is the only way to prove it. The rehearsal queries life_events for kind LIKE '%delete%' OR '%tombstone%' OR '%removed%', asserts the event-kind set is exactly {task_created, operation_recovery_observed, observation_gap}, and pins the gap's window (from = capture time, to = restore start, with a duration floor). Reconciling without fabricating creation or deletion events cannot be shown any other way.
• Reconciliation reuses rather than re-implements: reconcile_after_restore calls recover_pending_operations (so the #6 bound, deferred and unreadable semantics still apply) and then, in one transaction, invalidates sync cursors (cursor=NULL, generation+1, complete=0, coverage incomplete) and appends the single observation gap.
• Restore refuses unsafe targets before touching anything: a held service lock, an existing database, or foreign -wal/-shm sidecars abort, then the copy's checksum is verified before the rename — so a half-restore cannot land. serve() runs complete_pending_restore before admitting a request, which is what makes "verified before traffic" true rather than intended.
• The manifest makes staleness and corruption visible (life backup status reports age, size, schema and backend version, is_stale, and a recomputed checksum so a modified file reports checksum_ok=false), covering the "monitor snapshot age/failures" half of the operations contract.

Deployment inputs correctly left out (encryption, off-host destination, frequency, retention, key custody) and disclosed; both sides of the src/api/mod.rs conflict were kept with #24's /v1/next. No new dependency.

Merging. Phase 5's operational half is now complete except the agent-portability work (#31–#33).

## Coordinator review — APPROVED Verified `b783b05` myself: `cargo test` **566 passed / 0 failed** (9 disaster-recovery tests), `cargo clippy --all-targets -- -D warnings` 0 errors, `cargo fmt --check` clean. What I checked beyond the summary: - **The snapshot goes through SQLite, on the worker's own connection** (`VACUUM INTO ?1` at `src/history/backup.rs:335`, documented in the module header and at the call site), and the rejection of the naive copy is **evidence-backed rather than asserted**: with `wal_autocheckpoint=0` the test commits rows that stay in `-wal`, then shows the snapshot holds the final row while a plain `life.sqlite3` copy does not. That is exactly the D10 pair, and it is the difference between a documented rule and a justified one. - **D11's hardest clause is asserted by absence, which is the only way to prove it.** The rehearsal queries `life_events` for `kind LIKE '%delete%' OR '%tombstone%' OR '%removed%'`, asserts the event-kind set is exactly `{task_created, operation_recovery_observed, observation_gap}`, and pins the gap's window (`from` = capture time, `to` = restore start, with a duration floor). Reconciling without fabricating creation or deletion events cannot be shown any other way. - **Reconciliation reuses rather than re-implements:** `reconcile_after_restore` calls `recover_pending_operations` (so the #6 bound, `deferred` and `unreadable` semantics still apply) and *then*, in one transaction, invalidates sync cursors (`cursor=NULL`, `generation+1`, `complete=0`, coverage `incomplete`) and appends the single observation gap. - **Restore refuses unsafe targets before touching anything:** a held service lock, an existing database, or foreign `-wal`/`-shm` sidecars abort, then the copy's checksum is verified before the rename — so a half-restore cannot land. `serve()` runs `complete_pending_restore` before admitting a request, which is what makes "verified before traffic" true rather than intended. - **The manifest makes staleness and corruption visible** (`life backup status` reports age, size, schema and backend version, `is_stale`, and a recomputed checksum so a modified file reports `checksum_ok=false`), covering the "monitor snapshot age/failures" half of the operations contract. Deployment inputs correctly left out (encryption, off-host destination, frequency, retention, key custody) and disclosed; both sides of the `src/api/mod.rs` conflict were kept with #24's `/v1/next`. No new dependency. Merging. Phase 5's operational half is now complete except the agent-portability work (#31–#33).

yuris merged commit ec727086c4 into main 2026-09-23 00:58:12 +00:00

yuris referenced this pull request from a commit 2026-09-23 00:58:14 +00:00

feat(storage): sqlite online snapshot and disaster recovery rehearsal (#67)

yuris changed title from feat(storage): sqlite online snapshot and disaster recovery rehearsal (#36) to feat(storage): sqlite online snapshot and disaster recovery rehearsal 2026-09-23 05:12:01 +00:00

yuris referenced this pull request from a commit 2026-09-23 11:54:43 +00:00

feat(storage): sqlite online snapshot and disaster recovery rehearsal (#67)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service!67

No description provided.