Crash Recovery & Operation Inspection CLI #6

New issue

Closed

opened 2026-09-22 17:58:53 +00:00 by yuris · 0 comments

yuris commented 2026-09-22 17:58:53 +00:00

Owner

Copy link

Summary

Implement startup crash recovery scanning to detect in-flight or unconfirmed operations, classify uncertain external states using evidence-based readback rules, provide the GET /v1/operations/{id} API route, and build the life operation get <ID> CLI command with exit code 6 pending/uncertainty semantics.

Background & Technical Rationale

Distributed state mutations are susceptible to process termination, host crashes, network timeouts, and power failures at distinct execution boundaries. The Life service must never blindly replay an operation or fabricate completion without observable evidence:

1. Crash After Prepared Commit, Before Dispatch (Scenario R05):
   • The operation was persisted as prepared, but the process crashed before emitting the HTTP request.
   • Upon restart, the service cannot assume the request was never sent (in a distributed system, a crash could occur during socket transmission).
   • Recovery must treat the operation as potentially dispatched, inspect the remote state, and transition to needs_reconciliation if unverified. Blind replay is prohibited.
2. Crash After Dispatch, Before Remote Response (Scenario R06):
   • The HTTP request was sent to Radicale, but the process crashed before receiving or persisting the response.
   • Upon restart, remote readback may find that the remote resource matches the intended state. However, matching remote state proves observation, not unique attribution (an external client or background process could have made that edit).
   • The service appends an observation event with attribution uncertainty and marks the operation needs_reconciliation.
3. Remote Success Followed by Local Result Transaction Failure (Scenario R07):
   • The remote write succeeded on Radicale, but SQLite failed or crashed during the local result commit.
   • The operation must NOT be reported as fully committed. The operation ID survives in operations, and recovery marks the uncertainty.
4. Crash After Local Result Commit, Before Client Response (Scenario R08):
   • The remote write succeeded, and SQLite committed operations.state = 'applied' and appended life_events. However, the process crashed before transmitting the HTTP response to the client.
   • A subsequent same-key replay by the client returns the durable applied outcome without dispatching any secondary remote mutation.
5. Operation Inspection CLI & Exit Code Semantics:
   • life operation get <ID> --json maps to GET /v1/operations/{id}.
   • It outputs the durable operation state, intended target, baseline, recorded result, linked events, and recovery requirements.
   • If an operation is in unknown, needs_reconciliation, or prepared, the CLI exits with code 6 (partial / pending), signaling to agents and operators that recovery investigation is required before any further action.

Exact Technical References & Specifications

Crash Recovery Rules Table

From docs/technical/timeline-consistency.md#crash-recovery:

"Read live remote resource using the stored trusted target and compare baseline and intended normalized semantics, including preserved resource members:

Evidence	Recovery action
Known acknowledged success persisted with result	Replay recorded outcome, no remote retry
Current state matches intended; original acknowledgment missing	Append recovered observation with attribution uncertainty; matching state proves observation, not unique attribution; retain needs_reconciliation unless policy/user resolves the execution claim
Current state matches baseline	Does not rule out write then external revert; keep needs_reconciliation without independent proof of no dispatch
Resource absent after uncertain create/delete	Absence alone does not establish whether a transient write happened; reconcile
Current state differs from both	Preserve external change; ask for reviewed repair, never overwrite
Confirmed pre-dispatch failure	Mark failed; a newly authorized operation may retry

For create, original UID/href/nonexistence condition are stable. Never allocate a new UID merely because the response was lost. State markers can strengthen evidence if preservation-compatible, but a matching marker is not unforgeable proof. Manual reconciliation appends user resolution with evidence and can close the operation as applied/failed only if the corresponding outcome is actually established; otherwise retain uncertainty and proceed with a distinct reviewed repair."

Verification Scenarios (R05, R06, R07, R08)

From docs/technical/verification.md#recovery-scenarios:

ID	Boundary / injected failure	Observable acceptance
R05	Crash after prepared commit, before dispatch	Restart treats prepared as potentially dispatched; evidence-based reconciliation, no blind replay
R06	Crash after dispatch, before remote response	unknown/reconciliation; matching remote state is observation, not unique attribution
R07	Remote success then local result transaction fails/crashes	Not reported fully committed; operation ID survives; recovery records uncertainty
R08	Crash after local result commit, before client response	Same-key replay returns durable applied result without another mutation

Illustrative Pending / Unknown Response Envelope

From docs/technical/verification.md#recovery-scenarios:

"Illustrative pending response for life task complete itm_demo --etag '\"v1\"' --operation-id op_demo_unknown --input - --json after explicit approval, with stdin {} and a lost remote response:

{"schema_version":1,"request_id":"req_demo_unknown","status":"partial","data":{"operation_id":"op_demo_unknown","operation_state":"unknown"},"warnings":[],"error":{"code":"operation_unknown","message":"Remote outcome uncertain; inspect operation before any retry","retryable":false,"operation_id":"op_demo_unknown"}}

Expected CLI exit 6. life operation get op_demo_unknown --json investigates recovery; it is not evidence of completion until the returned state/evidence establishes it. See consistency contract."

Command and Route Specifications

From docs/technical/cli-api.md#command-to-route-table:
| life operation get ID | GET /v1/operations/{id} | read |

From docs/technical/cli-api.md#command-payloads-and-results:
| Operation get | ID | state, intended target, recorded result, steps, recovery requirement and context delivery |

From docs/technical/cli-api.md#error-identifiers:
| operation_unknown, needs_reconciliation | Recover by operation ID; no blind replay |

From docs/technical/cli-api.md#transport-and-common-contract:

• Exit 6: Pending, partial or unknown outcome (HTTP 202 or 200 with partial).

Formal Section Citations

• docs/technical/timeline-consistency.md#operation-state-machine: Defines the operation lifecycle states and transitions.
• docs/technical/timeline-consistency.md#crash-recovery: Normative table of evidence classes and recovery actions.
• docs/technical/cli-api.md#command-to-route-table: Defines life operation get ID route mapping.
• docs/technical/cli-api.md#command-payloads-and-results: Specifies payload schema for operation get.
• docs/technical/cli-api.md#transport-and-common-contract: Governs exit code 6 for pending/unknown outcomes.
• docs/technical/verification.md#recovery-scenarios: Formulates normative test scenarios R05, R06, R07, and R08.

What to Build (Step-by-Step)

1. Implement Recovery Scanner (src/history/recovery.rs):

   • On startup of life serve, query operations where state IN ('prepared', 'unknown').
   • For each unresolved operation:
     • Read target resource from remote Radicale collection using stored href/UID.
     • Compare current remote representation against baseline_json and intended intent_json:
       • If remote matches intended state: insert a life_events observation row with evidence_class = 'observed_external' and update operations.state = 'needs_reconciliation' (attribution uncertainty).
       • If remote matches baseline: update operations.state = 'needs_reconciliation'.
       • If remote differs from both: update operations.state = 'needs_reconciliation'.
     • Log recovery outcome with structured tracing (without logging private item payloads).

2. Implement Operation API Route in Axum (src/api.rs):

   • Route: GET /v1/operations/{id}.
   • Query operations table by operation_id.
   • If not found: return HTTP 404 with error code not_found.
   • If found: return HTTP 200 with OperationDetails:
     • operation_id: String
     • state: prepared | applied | failed | unknown | needs_reconciliation
     • actor: String
     • prepared_at: RFC3339 UTC timestamp
     • updated_at: RFC3339 UTC timestamp
     • intent: JSON value
     • baseline: Option
     • result: Option
     • recovery_requirement: String description of required operator/agent review if state is needs_reconciliation or unknown.

3. Implement life operation get CLI Command (src/cli.rs):

   • Parse life operation get <ID> [--json].
   • Make authenticated GET request to /v1/operations/{id}.
   • Format output using standard ResponseEnvelope.
   • Evaluate operation state for exit code:
     • If state == "applied" or state == "failed": exit 0.
     • If state == "unknown" or state == "needs_reconciliation" or state == "prepared": exit 6 (partial / pending).
     • If 404: exit 2 with error code not_found.

4. Integration Fault-Injection Tests (tests/crash_recovery.rs):

   • Scenario R05: Inject crash after prepared commit, before dispatch. Verify restart detects prepared operation, performs evidence readback, and does not blindly retry remote write.
   • Scenario R06: Inject crash after dispatch, before remote response. Verify restart marks operation as needs_reconciliation with attribution uncertainty.
   • Scenario R07: Inject SQLite failure on result commit after successful remote write. Verify operation ID survives and recovery identifies unconfirmed status.
   • Scenario R08: Inject client network disconnect after local result commit. Issue same-key replay and assert durable applied result is returned without secondary remote mutation.
   • Assert life operation get <ID> exits with code 6 for all unresolved operations.

Acceptance Criteria

• A simulated process crash immediately after prepared commit (before dispatch) transitions operation to needs_reconciliation on restart without issuing a blind duplicate write (verifying Scenario R05).
• A simulated process crash after remote dispatch (before response commit) records attribution uncertainty and marks operation needs_reconciliation on restart (verifying Scenario R06).
• Remote success followed by local commit failure leaves operation in recoverable state and never reports fully committed (verifying Scenario R07).
• Process crash after local result commit allows safe replay returning durable applied result without secondary remote write (verifying Scenario R08).
• Running life operation get <ID> --json returns envelope with current operation state and recovery requirements.
• life operation get exits with code 6 when an operation is in unknown, needs_reconciliation, or prepared state.
• Formatted JSON output matches the normative pending response envelope from docs/technical/verification.md.

Notes

• Review round added a bounded startup pass: MAX_RECOVERY_OPERATIONS = 25, RECOVERY_BUDGET = 30s, leftovers reported as deferred and retried on the next start.
• not_found exit-code divergence from life item get is tracked in #49.

Blocked by

• #005: Two-Phase Commit Journaling & Idempotency Fingerprinting

## Summary Implement startup crash recovery scanning to detect in-flight or unconfirmed operations, classify uncertain external states using evidence-based readback rules, provide the `GET /v1/operations/{id}` API route, and build the `life operation get <ID>` CLI command with exit code 6 pending/uncertainty semantics. ## Background & Technical Rationale Distributed state mutations are susceptible to process termination, host crashes, network timeouts, and power failures at distinct execution boundaries. The Life service must never blindly replay an operation or fabricate completion without observable evidence: 1. **Crash After Prepared Commit, Before Dispatch (Scenario R05)**: - The operation was persisted as `prepared`, but the process crashed before emitting the HTTP request. - Upon restart, the service cannot assume the request was never sent (in a distributed system, a crash could occur during socket transmission). - Recovery must treat the operation as potentially dispatched, inspect the remote state, and transition to `needs_reconciliation` if unverified. Blind replay is prohibited. 2. **Crash After Dispatch, Before Remote Response (Scenario R06)**: - The HTTP request was sent to Radicale, but the process crashed before receiving or persisting the response. - Upon restart, remote readback may find that the remote resource matches the intended state. However, **matching remote state proves observation, not unique attribution** (an external client or background process could have made that edit). - The service appends an observation event with attribution uncertainty and marks the operation `needs_reconciliation`. 3. **Remote Success Followed by Local Result Transaction Failure (Scenario R07)**: - The remote write succeeded on Radicale, but SQLite failed or crashed during the local result commit. - The operation must NOT be reported as fully committed. The operation ID survives in `operations`, and recovery marks the uncertainty. 4. **Crash After Local Result Commit, Before Client Response (Scenario R08)**: - The remote write succeeded, and SQLite committed `operations.state = 'applied'` and appended `life_events`. However, the process crashed before transmitting the HTTP response to the client. - A subsequent same-key replay by the client returns the durable `applied` outcome without dispatching any secondary remote mutation. 5. **Operation Inspection CLI & Exit Code Semantics**: - `life operation get <ID> --json` maps to `GET /v1/operations/{id}`. - It outputs the durable operation state, intended target, baseline, recorded result, linked events, and recovery requirements. - If an operation is in `unknown`, `needs_reconciliation`, or `prepared`, the CLI exits with code 6 (`partial` / pending), signaling to agents and operators that recovery investigation is required before any further action. ## Exact Technical References & Specifications ### Crash Recovery Rules Table From `docs/technical/timeline-consistency.md#crash-recovery`: > "Read live remote resource using the stored trusted target and compare baseline and intended normalized semantics, including preserved resource members: | Evidence | Recovery action | |---|---| | Known acknowledged success persisted with result | Replay recorded outcome, no remote retry | | Current state matches intended; original acknowledgment missing | Append recovered observation with attribution uncertainty; matching state proves observation, not unique attribution; retain needs_reconciliation unless policy/user resolves the execution claim | | Current state matches baseline | Does not rule out write then external revert; keep needs_reconciliation without independent proof of no dispatch | | Resource absent after uncertain create/delete | Absence alone does not establish whether a transient write happened; reconcile | | Current state differs from both | Preserve external change; ask for reviewed repair, never overwrite | | Confirmed pre-dispatch failure | Mark failed; a newly authorized operation may retry | > For create, original UID/href/nonexistence condition are stable. Never allocate a new UID merely because the response was lost. State markers can strengthen evidence if preservation-compatible, but a matching marker is not unforgeable proof. Manual reconciliation appends user resolution with evidence and can close the operation as applied/failed only if the corresponding outcome is actually established; otherwise retain uncertainty and proceed with a distinct reviewed repair." ### Verification Scenarios (R05, R06, R07, R08) From `docs/technical/verification.md#recovery-scenarios`: | ID | Boundary / injected failure | Observable acceptance | |---|---|---| | R05 | Crash after prepared commit, before dispatch | Restart treats prepared as potentially dispatched; evidence-based reconciliation, no blind replay | | R06 | Crash after dispatch, before remote response | unknown/reconciliation; matching remote state is observation, not unique attribution | | R07 | Remote success then local result transaction fails/crashes | Not reported fully committed; operation ID survives; recovery records uncertainty | | R08 | Crash after local result commit, before client response | Same-key replay returns durable applied result without another mutation | ### Illustrative Pending / Unknown Response Envelope From `docs/technical/verification.md#recovery-scenarios`: > "Illustrative pending response for `life task complete itm_demo --etag '\"v1\"' --operation-id op_demo_unknown --input - --json` after explicit approval, with stdin `{}` and a lost remote response: > > ```json > {"schema_version":1,"request_id":"req_demo_unknown","status":"partial","data":{"operation_id":"op_demo_unknown","operation_state":"unknown"},"warnings":[],"error":{"code":"operation_unknown","message":"Remote outcome uncertain; inspect operation before any retry","retryable":false,"operation_id":"op_demo_unknown"}} > ``` > > Expected CLI exit 6. `life operation get op_demo_unknown --json` investigates recovery; it is not evidence of completion until the returned state/evidence establishes it. See [consistency contract](timeline-consistency.md#crash-recovery)." ### Command and Route Specifications From `docs/technical/cli-api.md#command-to-route-table`: `| life operation get ID | GET /v1/operations/{id} | read |` From `docs/technical/cli-api.md#command-payloads-and-results`: `| Operation get | ID | state, intended target, recorded result, steps, recovery requirement and context delivery |` From `docs/technical/cli-api.md#error-identifiers`: `| operation_unknown, needs_reconciliation | Recover by operation ID; no blind replay |` From `docs/technical/cli-api.md#transport-and-common-contract`: - Exit 6: Pending, partial or unknown outcome (HTTP 202 or 200 with partial). ### Formal Section Citations - `docs/technical/timeline-consistency.md#operation-state-machine`: Defines the operation lifecycle states and transitions. - `docs/technical/timeline-consistency.md#crash-recovery`: Normative table of evidence classes and recovery actions. - `docs/technical/cli-api.md#command-to-route-table`: Defines `life operation get ID` route mapping. - `docs/technical/cli-api.md#command-payloads-and-results`: Specifies payload schema for operation get. - `docs/technical/cli-api.md#transport-and-common-contract`: Governs exit code 6 for pending/unknown outcomes. - `docs/technical/verification.md#recovery-scenarios`: Formulates normative test scenarios R05, R06, R07, and R08. ## What to Build (Step-by-Step) 1. **Implement Recovery Scanner (`src/history/recovery.rs`)**: - On startup of `life serve`, query `operations` where `state IN ('prepared', 'unknown')`. - For each unresolved operation: - Read target resource from remote Radicale collection using stored href/UID. - Compare current remote representation against `baseline_json` and intended `intent_json`: - If remote matches intended state: insert a `life_events` observation row with `evidence_class = 'observed_external'` and update `operations.state = 'needs_reconciliation'` (attribution uncertainty). - If remote matches baseline: update `operations.state = 'needs_reconciliation'`. - If remote differs from both: update `operations.state = 'needs_reconciliation'`. - Log recovery outcome with structured tracing (without logging private item payloads). 2. **Implement Operation API Route in Axum (`src/api.rs`)**: - Route: `GET /v1/operations/{id}`. - Query `operations` table by `operation_id`. - If not found: return HTTP 404 with error code `not_found`. - If found: return HTTP 200 with `OperationDetails`: - `operation_id`: String - `state`: `prepared` | `applied` | `failed` | `unknown` | `needs_reconciliation` - `actor`: String - `prepared_at`: RFC3339 UTC timestamp - `updated_at`: RFC3339 UTC timestamp - `intent`: JSON value - `baseline`: Option<JSON value> - `result`: Option<JSON value> - `recovery_requirement`: String description of required operator/agent review if state is `needs_reconciliation` or `unknown`. 3. **Implement `life operation get` CLI Command (`src/cli.rs`)**: - Parse `life operation get <ID> [--json]`. - Make authenticated GET request to `/v1/operations/{id}`. - Format output using standard `ResponseEnvelope`. - Evaluate operation state for exit code: - If `state == "applied"` or `state == "failed"`: exit 0. - If `state == "unknown"` or `state == "needs_reconciliation"` or `state == "prepared"`: exit 6 (`partial` / pending). - If 404: exit 2 with error code `not_found`. 4. **Integration Fault-Injection Tests (`tests/crash_recovery.rs`)**: - **Scenario R05**: Inject crash after prepared commit, before dispatch. Verify restart detects prepared operation, performs evidence readback, and does not blindly retry remote write. - **Scenario R06**: Inject crash after dispatch, before remote response. Verify restart marks operation as `needs_reconciliation` with attribution uncertainty. - **Scenario R07**: Inject SQLite failure on result commit after successful remote write. Verify operation ID survives and recovery identifies unconfirmed status. - **Scenario R08**: Inject client network disconnect after local result commit. Issue same-key replay and assert durable applied result is returned without secondary remote mutation. - Assert `life operation get <ID>` exits with code 6 for all unresolved operations. ## Acceptance Criteria - [x] A simulated process crash immediately after prepared commit (before dispatch) transitions operation to `needs_reconciliation` on restart without issuing a blind duplicate write (verifying Scenario R05). - [x] A simulated process crash after remote dispatch (before response commit) records attribution uncertainty and marks operation `needs_reconciliation` on restart (verifying Scenario R06). - [x] Remote success followed by local commit failure leaves operation in recoverable state and never reports fully committed (verifying Scenario R07). - [x] Process crash after local result commit allows safe replay returning durable applied result without secondary remote write (verifying Scenario R08). - [x] Running `life operation get <ID> --json` returns envelope with current operation state and recovery requirements. - [x] `life operation get` exits with code 6 when an operation is in `unknown`, `needs_reconciliation`, or `prepared` state. - [x] Formatted JSON output matches the normative pending response envelope from `docs/technical/verification.md`. ## Notes - Review round added a bounded startup pass: `MAX_RECOVERY_OPERATIONS = 25`, `RECOVERY_BUDGET = 30s`, leftovers reported as `deferred` and retried on the next start. - `not_found` exit-code divergence from `life item get` is tracked in #49. ## Blocked by - #005: Two-Phase Commit Journaling & Idempotency Fingerprinting

yuris referenced this issue 2026-09-22 17:58:59 +00:00

Hardened Production Container & Exclusive Directory Lock #34

yuris changed title from Issue #006: Crash Recovery & Operation Inspection CLI to Crash Recovery & Operation Inspection CLI 2026-09-22 20:02:40 +00:00

yuris referenced this issue 2026-09-22 20:38:06 +00:00

Pagination: shared materialized read result #45

yuris referenced this issue from a pull request that will close it, 2026-09-22 21:54:22 +00:00

feat(history): crash recovery scanner and operation inspection cli #48

yuris referenced this issue 2026-09-22 21:57:09 +00:00

Unify not_found CLI exit code to 2 across commands #49

yuris closed this issue 2026-09-22 22:01:15 +00:00

yuris referenced this issue from a commit 2026-09-22 22:01:15 +00:00

feat(history): crash recovery scanner and operation inspection cli (#48)

yuris referenced this issue 2026-09-22 23:39:19 +00:00

feat(hindsight): durable context outbox and background delivery worker #59

yuris referenced this issue 2026-09-23 00:58:11 +00:00

feat(storage): sqlite online snapshot and disaster recovery rehearsal #67

yuris referenced this issue 2026-09-23 06:19:43 +00:00

life sync / POST /v1/sync is documented with an admin scope but has no handler #85

yuris referenced this issue 2026-09-23 06:23:14 +00:00

Implement POST /v1/sync: incremental and rebaseline collection sync #88

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service#6

No description provided.