feat(history): crash recovery scanner and operation inspection cli #48

Merged

yuris merged 2 commits from feat/issue-6-recovery into main 2026-09-22 22:01:15 +00:00

Conversation 2 Commits 2 Files changed 7 +2200 -55

yuris commented 2026-09-22 21:54:22 +00:00

Owner

Copy link

Motivation

docs/technical/cli-api.md maps life operation get ID to GET /v1/operations/{id}, but nothing implemented either half: src/cli.rs routed Commands::Operation(_) into not_implemented() and no operations route existed. Meanwhile the mutation routes already report error.operation_id on transport failures and partial envelopes for unconfirmed outcomes, so the recovery loop they point at was dead.

Crash recovery was missing too: an operation recorded prepared is potentially dispatched after a restart, so a service that restarts mid-write has no way to reconcile what the remote actually holds. This lands both halves — a startup scanner that reconciles unresolved operations from evidence, and the read side an operator or agent uses to inspect one.

Changes

• src/history/recovery.rs (new): startup scanner over operations WHERE state IN ('prepared','unknown'). For each operation it reads the stored target live (href/uid/representation from intent_json) and classifies the evidence per docs/technical/timeline-consistency.md#crash-recovery: a live state matching the intended representation appends a life_events observation with evidence_class='observed_external' and no invented actor or time (attribution stays uncertain); matching baseline, differing from both, and live absence reconcile without an observation. Every case moves the operation to needs_reconciliation — nothing is ever reported applied or failed, and no remote write is ever issued, because a readback establishes neither.
  • The comparison covers preserved resource members, not only the fields the read routes expose: the live component is compared against the stored intended representation.
  • Failures are reported in a RecoverySummary and logged with structured tracing fields (operation_id, error) — never payloads — and never take the daemon down.
• src/api/operations.rs (new) plus GET /v1/operations/{id}: returns operation_id, state, actor, prepared_at, updated_at, intent, baseline, result and recovery_requirement (present for unknown/needs_reconciliation). Unknown ID is HTTP 404 not_found; an unreadable journal is dependency_unavailable. The envelope is partial while the outcome is not established — the documented pairing of exit 6 with a partial response — and ok once it is.
• src/api/mod.rs: the route, and startup_recovery() wired into serve() before the listener accepts a request; the pass is skipped entirely when there is no journal worker or no calendar backend.
• src/cli.rs: life operation get <ID> [--json] dispatches to that route; exit 0 for applied/failed, 6 for prepared/unknown/needs_reconciliation, 2 for not_found, and transport failures keep the read-command codes (5 unreachable, 6 timeout).
• src/history/journal.rs: one shared decoder for operations rows (replacing three copies) plus JournalService::unresolved_operations(), the scanner's work list.
• src/history/recovery.rs: the pass is bounded — MAX_RECOVERY_OPERATIONS (25) and RECOVERY_BUDGET (30s), both public and checked between operations. The remainder is left exactly as it was and reported as RecoverySummary::deferred (plus a log line and the deferred field on the startup summary), so a crash loop's backlog against a slow or unreachable backend cannot hold readiness for one client timeout per operation; the next start continues where this one stopped.

Verification

• nix develop --command cargo test — 182 tests pass across 14 suites, including the new tests/crash_recovery_tests.rs (18 tests).
• New scenarios in tests/crash_recovery_tests.rs, asserted against real HTTP behaviour and the SQLite journal:
  • R05: prepared commit, crash before dispatch → needs_reconciliation, one evidence readback, zero remote writes.
  • R06: dispatch, lost response → needs_reconciliation with an observed_external observation carrying no actor and no occurred_at and provenance attribution: uncertain; the observed write is never repeated.
  • R07: remote success with a failed result commit (injected by occupying the operation's deterministic life_events dedupe key) → the response is partial with operation_state: prepared and operation_unknown, the operation ID survives, the durable row stays prepared, and recovery then reconciles it with no secondary write.
  • R08: crash after the local result commit → same-key replay returns the durable applied result (changed: false) with no second write, and recovery leaves the operation alone.
  • Evidence rows: matching baseline and live absence reconcile without an observation; an external change is preserved rather than overwritten; a difference only in a preserved member is not mistaken for the intended state.
  • life operation get exit codes 0/6/2 and 5/6 through the real binary, the 404 path, the missing-journal path, and recovery skipped without a journal worker.
• Manual smoke test of the real daemon: life serve over a journal holding a prepared operation reconciles it at startup (log line plus journal row) and keeps serving /health/live; life operation get against that daemon exits 6 with the full record and 2 for an unknown ID.
• Bounded pass: a backlog of MAX_RECOVERY_OPERATIONS + 2 prepared operations examines exactly the cap, reports the remainder deferred and still prepared, and a second pass reconciles them all; an unreachable backend with MAX_RECOVERY_OPERATIONS + 5 unresolved operations returns inside RECOVERY_BUDGET with nothing touched. Manually confirmed through the real daemon: a 30-operation backlog against a dead backend answers /health/live after 0.06s, logs scanned=25 deferred=5, and leaves all 30 operations unresolved.
• No regression to existing contracts: the #10/#11 partial + error.operation_id envelopes, deny_unknown_fields on create payloads and the If-Match preflight all still pass unchanged.

Related Issues

Closes #6

## Motivation `docs/technical/cli-api.md` maps `life operation get ID` to `GET /v1/operations/{id}`, but nothing implemented either half: `src/cli.rs` routed `Commands::Operation(_)` into `not_implemented()` and no operations route existed. Meanwhile the mutation routes already report `error.operation_id` on transport failures and `partial` envelopes for unconfirmed outcomes, so the recovery loop they point at was dead. Crash recovery was missing too: an operation recorded `prepared` is potentially dispatched after a restart, so a service that restarts mid-write has no way to reconcile what the remote actually holds. This lands both halves — a startup scanner that reconciles unresolved operations from evidence, and the read side an operator or agent uses to inspect one. ## Changes - `src/history/recovery.rs` (new): startup scanner over `operations WHERE state IN ('prepared','unknown')`. For each operation it reads the stored target live (`href`/`uid`/`representation` from `intent_json`) and classifies the evidence per `docs/technical/timeline-consistency.md#crash-recovery`: a live state matching the intended representation appends a `life_events` observation with `evidence_class='observed_external'` and no invented actor or time (attribution stays uncertain); matching baseline, differing from both, and live absence reconcile without an observation. Every case moves the operation to `needs_reconciliation` — nothing is ever reported `applied` or `failed`, and no remote write is ever issued, because a readback establishes neither. - The comparison covers preserved resource members, not only the fields the read routes expose: the live component is compared against the stored intended representation. - Failures are reported in a `RecoverySummary` and logged with structured `tracing` fields (`operation_id`, error) — never payloads — and never take the daemon down. - `src/api/operations.rs` (new) plus `GET /v1/operations/{id}`: returns `operation_id`, `state`, `actor`, `prepared_at`, `updated_at`, `intent`, `baseline`, `result` and `recovery_requirement` (present for `unknown`/`needs_reconciliation`). Unknown ID is HTTP 404 `not_found`; an unreadable journal is `dependency_unavailable`. The envelope is `partial` while the outcome is not established — the documented pairing of exit 6 with a partial response — and `ok` once it is. - `src/api/mod.rs`: the route, and `startup_recovery()` wired into `serve()` before the listener accepts a request; the pass is skipped entirely when there is no journal worker or no calendar backend. - `src/cli.rs`: `life operation get <ID> [--json]` dispatches to that route; exit 0 for `applied`/`failed`, 6 for `prepared`/`unknown`/`needs_reconciliation`, 2 for `not_found`, and transport failures keep the read-command codes (5 unreachable, 6 timeout). - `src/history/journal.rs`: one shared decoder for `operations` rows (replacing three copies) plus `JournalService::unresolved_operations()`, the scanner's work list. - `src/history/recovery.rs`: the pass is bounded — `MAX_RECOVERY_OPERATIONS` (25) and `RECOVERY_BUDGET` (30s), both public and checked between operations. The remainder is left exactly as it was and reported as `RecoverySummary::deferred` (plus a log line and the `deferred` field on the startup summary), so a crash loop's backlog against a slow or unreachable backend cannot hold readiness for one client timeout per operation; the next start continues where this one stopped. ## Verification - `nix develop --command cargo test` — 182 tests pass across 14 suites, including the new `tests/crash_recovery_tests.rs` (18 tests). - New scenarios in `tests/crash_recovery_tests.rs`, asserted against real HTTP behaviour and the SQLite journal: - R05: prepared commit, crash before dispatch → `needs_reconciliation`, one evidence readback, zero remote writes. - R06: dispatch, lost response → `needs_reconciliation` with an `observed_external` observation carrying no actor and no `occurred_at` and provenance `attribution: uncertain`; the observed write is never repeated. - R07: remote success with a failed result commit (injected by occupying the operation's deterministic `life_events` dedupe key) → the response is `partial` with `operation_state: prepared` and `operation_unknown`, the operation ID survives, the durable row stays `prepared`, and recovery then reconciles it with no secondary write. - R08: crash after the local result commit → same-key replay returns the durable `applied` result (`changed: false`) with no second write, and recovery leaves the operation alone. - Evidence rows: matching baseline and live absence reconcile without an observation; an external change is preserved rather than overwritten; a difference only in a preserved member is not mistaken for the intended state. - `life operation get` exit codes 0/6/2 and 5/6 through the real binary, the 404 path, the missing-journal path, and recovery skipped without a journal worker. - Manual smoke test of the real daemon: `life serve` over a journal holding a `prepared` operation reconciles it at startup (log line plus journal row) and keeps serving `/health/live`; `life operation get` against that daemon exits 6 with the full record and 2 for an unknown ID. - Bounded pass: a backlog of `MAX_RECOVERY_OPERATIONS + 2` prepared operations examines exactly the cap, reports the remainder deferred and still `prepared`, and a second pass reconciles them all; an unreachable backend with `MAX_RECOVERY_OPERATIONS + 5` unresolved operations returns inside `RECOVERY_BUDGET` with nothing touched. Manually confirmed through the real daemon: a 30-operation backlog against a dead backend answers `/health/live` after 0.06s, logs `scanned=25 deferred=5`, and leaves all 30 operations unresolved. - No regression to existing contracts: the #10/#11 `partial` + `error.operation_id` envelopes, `deny_unknown_fields` on create payloads and the `If-Match` preflight all still pass unchanged. ## Related Issues Closes #6

yuris added 1 commit 2026-09-22 21:54:22 +00:00

feat(history): crash recovery scanner and operation inspection cli e54f959b47

yuris commented 2026-09-22 21:57:09 +00:00

Author

Owner

Copy link

Coordinator review — CHANGES REQUESTED (one item)

Verified e54f959 by reading the diff and running it myself. The evidence discipline is exactly right:

• Recovery only reads: no PUT anywhere in src/history/recovery.rs; the only journal write is commit_operation_result(.., needs_reconciliation, ..) plus the observed_external observation.
• R06 is handled honestly — matching the intent produces an observation with explicit attribution uncertainty (Matching remote state proves an observation, not unique attribution…) and never applied.
• Every branch lands on needs_reconciliation; unreadable operations stay unresolved for the next start rather than being guessed.
• Journal failure during the pass is reported (journal_error) and the daemon keeps serving.
• GET /v1/operations/{id} returns the full record with recovery_requirement only for unknown/needs_reconciliation, 404 not_found, 503 journal_unavailable.
• life operation get exit codes: 0 settled, 6 unresolved, 2 unknown id, 5/6 transport.

Independent verification on e54f959: cargo test 182 passed / 0 failed across 14 suites including crash_recovery_tests (18); cargo clippy --all-targets -- -D warnings 0 errors; cargo fmt --check clean.

Change needed: bound the startup pass.

recover_pending_operations iterates every unresolved operation with no cap and no total deadline. Each readback is bounded only by the CalDAV client timeout (default request_timeout_seconds = 20), and startup_recovery runs before the listener accepts requests — so N unresolved operations against a slow or unreachable backend delay readiness by up to N × 20s, on precisely the path (crash + backend outage) recovery exists to handle. A crash loop accumulates exactly those rows.

Please add a bounded pass and report it:

• a cap on operations examined per pass and/or an overall deadline for the pass;
• operations left over are not touched and are reported (e.g. deferred: usize in RecoverySummary, logged) so the next start continues;
• readiness still waits only for the bounded pass.

This matches the repo's existing discipline (MAX_QUERY_SPAN_DAYS, MAX_EXPANDED_INSTANCES, MAX_HISTORY_ROWS, bounded reconcile_attempts) and keeps the daemon's startup time a function of a constant rather than of journal size.

Tests: (a) more unresolved operations than the cap → exactly the cap examined, the remainder reported as deferred and still unresolved, and the next pass picks them up; (b) an unreachable backend with many unresolved operations still returns within the pass bound. Keep everything else as-is and keep test/clippy/fmt green.

Observation (not a change for this PR)

not_found maps to exit 2 here but exit 1 for life item get (#9). docs/technical/cli-api.md defines exit 2 as "Invalid input/configuration" and exit 1 as "other failure", so exit 2 is the defensible reading and #9 is the outlier — the two issue specs disagreed. I'm filing a follow-up to align life item get's 404 to exit 2 rather than changing it here.

## Coordinator review — CHANGES REQUESTED (one item) Verified `e54f959` by reading the diff and running it myself. The evidence discipline is exactly right: - Recovery only reads: no `PUT` anywhere in `src/history/recovery.rs`; the only journal write is `commit_operation_result(.., needs_reconciliation, ..)` plus the `observed_external` observation. - R06 is handled honestly — matching the intent produces an observation with explicit attribution uncertainty (`Matching remote state proves an observation, not unique attribution…`) and never `applied`. - Every branch lands on `needs_reconciliation`; `unreadable` operations stay unresolved for the next start rather than being guessed. - Journal failure during the pass is reported (`journal_error`) and the daemon keeps serving. - `GET /v1/operations/{id}` returns the full record with `recovery_requirement` only for `unknown`/`needs_reconciliation`, 404 `not_found`, 503 `journal_unavailable`. - `life operation get` exit codes: 0 settled, 6 unresolved, 2 unknown id, 5/6 transport. Independent verification on `e54f959`: `cargo test` **182 passed / 0 failed** across 14 suites including `crash_recovery_tests` (18); `cargo clippy --all-targets -- -D warnings` 0 errors; `cargo fmt --check` clean. **Change needed: bound the startup pass.** `recover_pending_operations` iterates every unresolved operation with no cap and no total deadline. Each readback is bounded only by the CalDAV client timeout (default `request_timeout_seconds = 20`), and `startup_recovery` runs before the listener accepts requests — so N unresolved operations against a slow or unreachable backend delay readiness by up to N × 20s, on precisely the path (crash + backend outage) recovery exists to handle. A crash loop accumulates exactly those rows. Please add a bounded pass and report it: - a cap on operations examined per pass and/or an overall deadline for the pass; - operations left over are **not** touched and are reported (e.g. `deferred: usize` in `RecoverySummary`, logged) so the next start continues; - readiness still waits only for the bounded pass. This matches the repo's existing discipline (`MAX_QUERY_SPAN_DAYS`, `MAX_EXPANDED_INSTANCES`, `MAX_HISTORY_ROWS`, bounded `reconcile_attempts`) and keeps the daemon's startup time a function of a constant rather than of journal size. Tests: (a) more unresolved operations than the cap → exactly the cap examined, the remainder reported as deferred and still unresolved, and the next pass picks them up; (b) an unreachable backend with many unresolved operations still returns within the pass bound. Keep everything else as-is and keep test/clippy/fmt green. ### Observation (not a change for this PR) `not_found` maps to exit **2** here but exit **1** for `life item get` (#9). `docs/technical/cli-api.md` defines exit 2 as "Invalid input/configuration" and exit 1 as "other failure", so exit 2 is the defensible reading and #9 is the outlier — the two issue specs disagreed. I'm filing a follow-up to align `life item get`'s 404 to exit 2 rather than changing it here.

yuris added 1 commit 2026-09-22 21:59:58 +00:00

fix(history): bound the startup recovery pass 89395d0f14

yuris commented 2026-09-22 22:01:13 +00:00

Author

Owner

Copy link

Coordinator re-review — APPROVED

Verified 89395d0 myself (read the diff, ran the suite):

• MAX_RECOVERY_OPERATIONS = 25 and RECOVERY_BUDGET = 30s are checked before each operation, so startup time is now a function of constants rather than of journal size. Worst case is bounded by the budget plus one client timeout — acceptable and a large improvement on N × 20s.
• Leftovers are left untouched (still prepared/unknown) and reported as RecoverySummary::deferred plus a log line, so the next start continues rather than the evidence being guessed.
• Tests prove both halves: a backlog of cap+2 examines exactly the cap, defers the remainder and reconciles them on a second pass; an unreachable backend with cap+5 returns inside the budget with the remainder deferred.

Independent verification on 89395d0: cargo test 184 passed / 0 failed across 14 suites including crash_recovery_tests (20); cargo clippy --all-targets -- -D warnings 0 errors; cargo fmt --check clean.

Merging. The not_found exit-code divergence is tracked separately in #49 and correctly left alone here.

## Coordinator re-review — APPROVED Verified `89395d0` myself (read the diff, ran the suite): - `MAX_RECOVERY_OPERATIONS = 25` and `RECOVERY_BUDGET = 30s` are checked before each operation, so startup time is now a function of constants rather than of journal size. Worst case is bounded by the budget plus one client timeout — acceptable and a large improvement on N × 20s. - Leftovers are left **untouched** (still `prepared`/`unknown`) and reported as `RecoverySummary::deferred` plus a log line, so the next start continues rather than the evidence being guessed. - Tests prove both halves: a backlog of cap+2 examines exactly the cap, defers the remainder and reconciles them on a second pass; an unreachable backend with cap+5 returns inside the budget with the remainder deferred. Independent verification on `89395d0`: `cargo test` **184 passed / 0 failed** across 14 suites including `crash_recovery_tests` (20); `cargo clippy --all-targets -- -D warnings` 0 errors; `cargo fmt --check` clean. Merging. The `not_found` exit-code divergence is tracked separately in #49 and correctly left alone here.

yuris merged commit be5ed6d8bf into main 2026-09-22 22:01:15 +00:00

yuris referenced this pull request from a commit 2026-09-22 22:01:15 +00:00

feat(history): crash recovery scanner and operation inspection cli (#48)

yuris changed title from feat(history): crash recovery scanner and operation inspection cli (#6) to feat(history): crash recovery scanner and operation inspection cli 2026-09-23 05:12:07 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service!48

No description provided.