feat(hindsight): reqwest client and deterministic document identity #55

Merged

yuris merged 1 commit from feat/issue-16-hindsight into main 2026-09-22 22:58:11 +00:00

Conversation 1 Commits 1 Files changed 6 +3317

yuris commented 2026-09-22 22:56:35 +00:00

Owner

Copy link

Motivation

Life incorporates Hindsight as its semantic context backend, but nothing in the codebase talks to it yet. This lands the direct Rust adapter for one configured bank together with the two identities that make retained memory trustworthy:

• a stable document identity, so retrying the same Life event replaces one upstream document instead of accumulating duplicate, conflicting memories;
• a stable item tag, so observations about the same recurring series consolidate across time instead of fragmenting into volatile per-event tags.

It also makes the delivery vocabulary honest before anything depends on it: an accepted retain is not completed ingestion, an unconfirmed delivery is not a delivered one, and an authentication or schema mismatch stops delivery with an operator-visible error instead of being retried away. Hindsight remains optional throughout — it owns semantic context and never current calendar or task state.

Changes

• src/hindsight/identity.rs — deterministic derivation: document_id = "life-event:<event_id>", item_tag = "life-item:" + lowercase hex SHA-256(canonical UTF-8 JSON array [collection_id,uid]), and the stable life-context tag. The canonical form is produced by the JSON serializer, so no caller formatting can change the digest. Unit tests pin independently computed vectors (printf '%s' '["personal","fictional-demo-uid"]' | shasum -a 256), order sensitivity, whitespace significance and non-collapse.
• src/hindsight/models.rs — typed upstream wire format and the delivery-state model. RetainItem has no constructor that accepts a caller-supplied document id, update mode or observation scope: the document id is derived from the source event and update_mode/observation_scopes are single-variant enums (replace / combined), so a retry can only ever replace the same document. ContentTimestamp distinguishes a real occurrence time from unset, so ingestion time is never presented as occurrence time. Metadata is string-valued and held in a BTreeMap (not HashMap) so the same event always serializes to the same bytes. Responses expose upstream memory/operation ids separately, and ContextDelivery serializes as the documented result literal ("context_delivery":"not_requested"), never leaking upstream ids into that field.
• src/hindsight/client.rs — pooled reqwest+rustls client, one per bank, credential read from the mounted [hindsight].authorization_file via read_secret_file, bounded timeouts, bounded response size, Debug redaction, and redirects refused outright so an Authorization header can never be forwarded to another origin. verify_compatibility() proves the deployed authentication scheme and operations route before retains are enabled; authentication, authorization, absence and schema failures halt delivery and stay operator-visible, while an outage stays transient and leaves delivery retry_wait. retry_delay() is the bounded backoff that honors a server Retry-After.
• Delivery states stay distinct: pending / accepted / completed / retry_wait / needs_reconciliation / failed (plus not_requested), matching the timeline outbox vocabulary. A lost acceptance response is reported as needs_reconciliation, never as accepted or completed. confirm_retain() hands the upstream operation identity to the caller before the first poll is dispatched, and refuses to poll at all if the caller cannot record it.
• tests/hindsight_client_tests.rs — 23 integration tests against an in-process mock Hindsight (and, for outage independence, the daemon plus a mock Radicale), including exact captured request bodies across repeated retains.
• No new dependency: SHA-256 uses ring::digest, already used by src/history/fingerprint.rs, and the secret is held in zeroize::Zeroizing like src/calendar/client.rs. Cargo.toml is untouched (so the authors chore is not bundled here).

Left for #17 by design: the durable outbox row, the background polling worker and retry scheduling. The outbox table and its state vocabulary already exist in the baseline schema; this issue builds the adapter, identity, wire types, delivery-state model and the direct client a caller uses.

Deliberate deviations from the issue's illustrative sketch

• secrecy::SecretString → zeroize::Zeroizing<String>, matching read_secret_file/CalDavClient (no new dependency).
• sha2::Sha256 → ring::digest::SHA256, matching src/history/fingerprint.rs (no new dependency).
• HashMap<String,String> metadata → BTreeMap, so serialization is deterministic and repeated retains produce identical bytes.
• HTTP 401 and 403 are separate variants (Unauthenticated/Forbidden) because the documented CLI vocabulary distinguishes unauthenticated from forbidden.
• HindsightError::code() is named to match the merged JevError::code() in src/jev/client.rs.

Verification

nix develop --command cargo test — all suites green (44 lib unit tests, 23 new hindsight_client_tests, and every pre-existing suite unchanged):

test result: ok. 44 passed; 0 failed   (unittests src/lib.rs)
test result: ok. 23 passed; 0 failed   (tests/hindsight_client_tests.rs)
... agenda_tasks 22, api 6, caldav_discovery 14, caldav_report 10, cli_creation 8,
cli 9, config 13, crash_recovery 20, creation 19, history_journal 5, history 8,
jev_client 24, jev_fixtures 34, lifecycle 27, recurrence 23, update 33 — all ok

cargo clippy --all-targets -- -D warnings and cargo fmt --check are clean.

What the new tests observe (all against a real mock HTTP server, never a live bank):

• I01 — two retains of evt_demo (the second with revised content) send document_id: "life-event:evt_demo", update_mode: "replace" and observation_scopes: "combined" in both captured bodies; the mock holds exactly one memory for that document, holding the revision; re-sending identical content produces byte-identical bytes (no random id, no "now" timestamp).
• Tag — the tag actually sent equals the independently computed vector, and an occurrence-specific record keeps the same series tag while carrying recurrence_kind/recurrence_key metadata; context-only records carry life-context, the source event id and no uid key.
• Wire — the serialized body is byte-equal to the normative example from docs/technical/hindsight.md (with the placeholder tag replaced by the real derivation), and unknown content is sent as "unset".
• I03 — a queued retain reports accepted and is not completed; polling pending, in_progress, completed and failed maps to accepted/accepted/completed/failed; only upstream reporting completion yields a completed delivery.
• Ordering — the mock reads the caller's durable record when the poll arrives, proving the operation id was recorded before the poll was dispatched; a record failure sends no poll at all and stays needs_reconciliation.
• Bootstrap — a healthy probe (404 for a reserved probe operation) enables retains; a 401 probe or retain, a 403, a 404 and three schema failures (unparsable body, 202 without an operation, a failure status alongside ids) all halt delivery, stop further dispatch and surface the documented error code.
• Outage — with Hindsight configured but unreachable, POST /v1/tasks still returns 201 ok with the created item, twice; the context side reports retry_wait with a context_pending warning. A disabled backend is refused at construction and reported as an explicit failed/context_unavailable, never as pending or completed.
• Redirect — a 307 to a second mock is refused and the second mock receives zero requests, so the credential is never forwarded.

Related Issues

Closes #16

## Motivation Life incorporates Hindsight as its semantic context backend, but nothing in the codebase talks to it yet. This lands the direct Rust adapter for one configured bank together with the two identities that make retained memory trustworthy: * a **stable document identity**, so retrying the same Life event replaces one upstream document instead of accumulating duplicate, conflicting memories; * a **stable item tag**, so observations about the same recurring series consolidate across time instead of fragmenting into volatile per-event tags. It also makes the delivery vocabulary honest before anything depends on it: an accepted retain is not completed ingestion, an unconfirmed delivery is not a delivered one, and an authentication or schema mismatch stops delivery with an operator-visible error instead of being retried away. Hindsight remains optional throughout — it owns semantic context and never current calendar or task state. ## Changes - `src/hindsight/identity.rs` — deterministic derivation: `document_id = "life-event:<event_id>"`, `item_tag = "life-item:" + lowercase hex SHA-256(canonical UTF-8 JSON array `[collection_id,uid]`),` and the stable `life-context` tag. The canonical form is produced by the JSON serializer, so no caller formatting can change the digest. Unit tests pin independently computed vectors (`printf '%s' '["personal","fictional-demo-uid"]' | shasum -a 256`), order sensitivity, whitespace significance and non-collapse. - `src/hindsight/models.rs` — typed upstream wire format and the delivery-state model. `RetainItem` has no constructor that accepts a caller-supplied document id, update mode or observation scope: the document id is derived from the source event and `update_mode`/`observation_scopes` are single-variant enums (`replace` / `combined`), so a retry can only ever replace the same document. `ContentTimestamp` distinguishes a real occurrence time from `unset`, so ingestion time is never presented as occurrence time. Metadata is string-valued and held in a `BTreeMap` (not `HashMap`) so the same event always serializes to the same bytes. Responses expose upstream memory/operation ids separately, and `ContextDelivery` serializes as the documented result literal (`"context_delivery":"not_requested"`), never leaking upstream ids into that field. - `src/hindsight/client.rs` — pooled `reqwest`+rustls client, one per bank, credential read from the mounted `[hindsight].authorization_file` via `read_secret_file`, bounded timeouts, bounded response size, `Debug` redaction, and redirects refused outright so an `Authorization` header can never be forwarded to another origin. `verify_compatibility()` proves the deployed authentication scheme and operations route before retains are enabled; authentication, authorization, absence and schema failures halt delivery and stay operator-visible, while an outage stays transient and leaves delivery `retry_wait`. `retry_delay()` is the bounded backoff that honors a server `Retry-After`. - Delivery states stay distinct: `pending` / `accepted` / `completed` / `retry_wait` / `needs_reconciliation` / `failed` (plus `not_requested`), matching the timeline outbox vocabulary. A lost acceptance response is reported as `needs_reconciliation`, never as accepted or completed. `confirm_retain()` hands the upstream operation identity to the caller **before** the first poll is dispatched, and refuses to poll at all if the caller cannot record it. - `tests/hindsight_client_tests.rs` — 23 integration tests against an in-process mock Hindsight (and, for outage independence, the daemon plus a mock Radicale), including exact captured request bodies across repeated retains. - No new dependency: SHA-256 uses `ring::digest`, already used by `src/history/fingerprint.rs`, and the secret is held in `zeroize::Zeroizing` like `src/calendar/client.rs`. `Cargo.toml` is untouched (so the `authors` chore is not bundled here). Left for #17 by design: the durable outbox row, the background polling worker and retry scheduling. The `outbox` table and its state vocabulary already exist in the baseline schema; this issue builds the adapter, identity, wire types, delivery-state model and the direct client a caller uses. ### Deliberate deviations from the issue's illustrative sketch - `secrecy::SecretString` → `zeroize::Zeroizing<String>`, matching `read_secret_file`/`CalDavClient` (no new dependency). - `sha2::Sha256` → `ring::digest::SHA256`, matching `src/history/fingerprint.rs` (no new dependency). - `HashMap<String,String>` metadata → `BTreeMap`, so serialization is deterministic and repeated retains produce identical bytes. - HTTP 401 and 403 are separate variants (`Unauthenticated`/`Forbidden`) because the documented CLI vocabulary distinguishes `unauthenticated` from `forbidden`. - `HindsightError::code()` is named to match the merged `JevError::code()` in `src/jev/client.rs`. ## Verification `nix develop --command cargo test` — all suites green (44 lib unit tests, 23 new `hindsight_client_tests`, and every pre-existing suite unchanged): ``` test result: ok. 44 passed; 0 failed (unittests src/lib.rs) test result: ok. 23 passed; 0 failed (tests/hindsight_client_tests.rs) ... agenda_tasks 22, api 6, caldav_discovery 14, caldav_report 10, cli_creation 8, cli 9, config 13, crash_recovery 20, creation 19, history_journal 5, history 8, jev_client 24, jev_fixtures 34, lifecycle 27, recurrence 23, update 33 — all ok ``` `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` are clean. What the new tests observe (all against a real mock HTTP server, never a live bank): - **I01** — two retains of `evt_demo` (the second with revised content) send `document_id: "life-event:evt_demo"`, `update_mode: "replace"` and `observation_scopes: "combined"` in both captured bodies; the mock holds exactly one memory for that document, holding the revision; re-sending identical content produces byte-identical bytes (no random id, no "now" timestamp). - **Tag** — the tag actually sent equals the independently computed vector, and an occurrence-specific record keeps the same series tag while carrying `recurrence_kind`/`recurrence_key` metadata; context-only records carry `life-context`, the source event id and no `uid` key. - **Wire** — the serialized body is byte-equal to the normative example from `docs/technical/hindsight.md` (with the placeholder tag replaced by the real derivation), and unknown content is sent as `"unset"`. - **I03** — a queued retain reports `accepted` and is not completed; polling `pending`, `in_progress`, `completed` and `failed` maps to accepted/accepted/completed/failed; only upstream reporting completion yields a completed delivery. - **Ordering** — the mock reads the caller's durable record when the poll arrives, proving the operation id was recorded before the poll was dispatched; a record failure sends no poll at all and stays `needs_reconciliation`. - **Bootstrap** — a healthy probe (404 for a reserved probe operation) enables retains; a 401 probe or retain, a 403, a 404 and three schema failures (unparsable body, 202 without an operation, a failure status alongside ids) all halt delivery, stop further dispatch and surface the documented error code. - **Outage** — with Hindsight configured but unreachable, `POST /v1/tasks` still returns 201 `ok` with the created item, twice; the context side reports `retry_wait` with a `context_pending` warning. A disabled backend is refused at construction and reported as an explicit `failed`/`context_unavailable`, never as pending or completed. - **Redirect** — a 307 to a second mock is refused and the second mock receives zero requests, so the credential is never forwarded. ## Related Issues Closes #16

yuris added 1 commit 2026-09-22 22:56:35 +00:00

feat(hindsight): reqwest client and deterministic document identity ... 383daba0bd

Add the direct reqwest+rustls adapter for one configured Hindsight bank,
deterministic document and item-tag identity, the typed retain/poll wire
format, and the delivery-state model (#16).

- identity: document_id is always `life-event:<event_id>` and the item tag is
  `life-item:` plus lowercase SHA-256 of the canonical UTF-8 JSON array
  [collection_id,uid]; context-only records use the stable `life-context` tag
  with the source event identity and no fabricated UID.
- models: the only constructors derive the document id from the source event
  and fix `update_mode: "replace"` and `observation_scopes: "combined"`, so a
  retry can only ever replace the same document; unknown occurrence time is
  sent as `timestamp: "unset"`; metadata is string-valued and deterministically
  ordered.
- client: pooled client, credential from the mounted secret file, bounded
  timeouts, redirects refused; `verify_compatibility` proves the deployed
  authentication scheme and operations route before retains are enabled, and
  authentication/schema failures halt delivery with an operator-visible error.
- delivery: pending / accepted / completed / retry_wait / needs_reconciliation
  / failed stay distinct (accepted is never reported as completed), a returned
  operation id is recorded before it is polled, and a lost acceptance response
  stays explicitly uncertain and is only re-submitted under the same document
  identity.
- outage independence: a disabled, unreachable or slow backend leaves calendar
  results untouched and reports context_unavailable / context_pending.

The durable outbox table, the background polling worker and retry scheduling
belong to #17.

yuris commented 2026-09-22 22:58:10 +00:00

Author

Owner

Copy link

Coordinator review — APPROVED

Verified 383daba myself. Notably, I re-derived the identity vectors independently rather than trusting the test's own comment:

printf '%s' '["personal","fictional-demo-uid"]' | shasum -a 256
391b39768920dc80ef2326ac9ae1ca88055fc12822cb9c1cdd0c3a50c4840839

which matches the hard-coded PERSONAL_DEMO_TAG, and likewise for 970038bf… and the reversed-order 96bf4ab0…. The derivation itself is right for a stronger reason than the vectors: derive_item_tag builds the canonical form with serde_json::to_string(&[collection_id, uid]) rather than string concatenation, so no caller's formatting can move the digest — and the order-sensitivity vector proves [a,b] ≠ [b,a].

What I checked beyond the summary:

• Milestones stay distinct, including the negative case. i03_accepted_and_completed_stay_distinct_milestones walks pending → in_progress → completed → failed, and an_unrecognised_operation_status_is_not_a_completion_claim refuses to treat an unknown status as done. Nothing claims exactly-once consolidation.
• I04 is tested end to end, not asserted. i04_a_hindsight_outage_never_changes_a_calendar_result runs the real in-process daemon against a mock Radicale with Hindsight down, which is the one claim that would have been easy — and wrong — to take on faith.
• Halted delivery latches. verify_compatibility probes a reserved operation id (404 = credential accepted and route present); 401/403/404/schema failures set a halt so a later retain is refused with zero POSTs, while a transient outage deliberately does not halt (an outage is not a compatibility halt) — the distinction the spec asks for.
• A lost acceptance response re-submits the same document identity rather than a fresh one, and an unrecordable operation is never polled.
• a_redirect_is_refused_without_forwarding_the_credential — the credential-forwarding boundary is enforced in the transport, not just documented.

Independent verification on 383daba: cargo test 342 passed / 0 failed (23 new Hindsight tests); cargo clippy --all-targets -- -D warnings 0 errors; cargo fmt --check clean. The rebase onto ade394a kept both pub mod hindsight; and pub mod jev;. No new dependency; Cargo.toml untouched, so the deferred authors chore stays pending.

Correctly scoped: the outbox table, polling worker and retry scheduling are left to #17, and wiring verify_compatibility into startup belongs with #17/#18.

Merging.

## Coordinator review — APPROVED Verified `383daba` myself. Notably, I re-derived the identity vectors independently rather than trusting the test's own comment: ``` printf '%s' '["personal","fictional-demo-uid"]' | shasum -a 256 391b39768920dc80ef2326ac9ae1ca88055fc12822cb9c1cdd0c3a50c4840839 ``` which matches the hard-coded `PERSONAL_DEMO_TAG`, and likewise for `970038bf…` and the reversed-order `96bf4ab0…`. The derivation itself is right for a stronger reason than the vectors: `derive_item_tag` builds the canonical form with `serde_json::to_string(&[collection_id, uid])` rather than string concatenation, so no caller's formatting can move the digest — and the order-sensitivity vector proves `[a,b]` ≠ `[b,a]`. What I checked beyond the summary: - **Milestones stay distinct, including the negative case.** `i03_accepted_and_completed_stay_distinct_milestones` walks pending → in_progress → completed → failed, and `an_unrecognised_operation_status_is_not_a_completion_claim` refuses to treat an unknown status as done. Nothing claims exactly-once consolidation. - **I04 is tested end to end, not asserted.** `i04_a_hindsight_outage_never_changes_a_calendar_result` runs the real in-process daemon against a mock Radicale with Hindsight down, which is the one claim that would have been easy — and wrong — to take on faith. - **Halted delivery latches.** `verify_compatibility` probes a reserved operation id (404 = credential accepted and route present); 401/403/404/schema failures set a halt so a later retain is refused with zero POSTs, while a transient outage deliberately does **not** halt (`an outage is not a compatibility halt`) — the distinction the spec asks for. - **A lost acceptance response re-submits the same document identity** rather than a fresh one, and an unrecordable operation is never polled. - `a_redirect_is_refused_without_forwarding_the_credential` — the credential-forwarding boundary is enforced in the transport, not just documented. Independent verification on `383daba`: `cargo test` **342 passed / 0 failed** (23 new Hindsight tests); `cargo clippy --all-targets -- -D warnings` 0 errors; `cargo fmt --check` clean. The rebase onto `ade394a` kept both `pub mod hindsight;` and `pub mod jev;`. No new dependency; `Cargo.toml` untouched, so the deferred authors chore stays pending. Correctly scoped: the outbox table, polling worker and retry scheduling are left to #17, and wiring `verify_compatibility` into startup belongs with #17/#18. Merging.

yuris merged commit 12fcf8d04d into main 2026-09-22 22:58:11 +00:00

yuris referenced this pull request from a commit 2026-09-22 22:58:11 +00:00

feat(hindsight): reqwest client and deterministic document identity (#55)

yuris referenced this pull request 2026-09-22 23:06:17 +00:00

feat(calendar): scoped recurring mutations for series and occurrences #56

yuris changed title from feat(hindsight): reqwest client and deterministic document identity (#16) to feat(hindsight): reqwest client and deterministic document identity 2026-09-23 05:12:05 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service!55

No description provided.