feat(hindsight): strict context search and supersession provenance #61

Merged

yuris merged 1 commit from feat/issue-18-context into main 2026-09-23 00:03:24 +00:00

Conversation 1 Commits 1 Files changed 12 +3614 -80

yuris commented 2026-09-23 00:01:48 +00:00

Owner

Copy link

Motivation

Issue #18 makes semantic context retrieval safe to rely on. Hindsight is semantic and bounded:
it can rank an outdated rationale first, it knows nothing about what the calendar says now, and
a recall for one item can silently return a decision that belongs to another. Life therefore has
to own the rules around it:

• Strict item scoping (I02). An item-scoped query must send exactly
  tags: ["life-item:<sha256>"] with tags_match: "all_strict", which excludes untagged and
  foreign-tagged memories. An empty result is a valid answer, and widening to a bank-wide query
  would leak unrelated private decisions across tasks.
• Live calendar authority (W11). Recalled text is a recollection, never current state. A task
  the live calendar reports COMPLETED must not be answered as "postponed" because an embedding
  still ranks that memory highly.
• Supersession provenance (I05). A correction appends a new source event; upstream does not
  forget the old one. The journal's own facts say which event supersedes which, so the recalled
  memory can be labelled rather than presented as the current preference.
• Bounded retrieval. Recall is semantic, not an enumeration: counts and coverage come from
  SQLite, never from a result set.

Changes

Adds POST /v1/context and POST /v1/context/search (life context add / life context search)
on top of the #16 adapter and the #17 outbox — no second queue, no second table, no second worker.

• src/hindsight/recall.rs — the typed recall request/response. RecallScope is part of the
  request type: an item-scoped query cannot be built without its tag and is fixed to all_strict,
  and a bank-wide read is a separate, explicit constructor. There is no expression that starts
  item-scoped and ends bank-wide.
• src/hindsight/provenance.rs — the local supersession ledger, read from life_events
  (correction_target / provenance_json.supersedes_context_id), resolved to the end of the
  supersession chain and applied to each recalled memory as is_stale + superseded_by, plus the
  journal counts (context_events, undelivered_context_events). No migration and no new table:
  the ledger is the append-only journal.
• src/hindsight/why.rs — live-calendar authority. Live state leads the answer; every recollection
  is labelled historical, and one that asserts a state contradicting the live STATUS (over a
  short, documented, deterministic vocabulary) is flagged and quoted as historical context.
• src/hindsight/context.rs — the parts captures and context-add must not diverge on: the derived
  source-event identity, the retain body (item tag vs. stable context tag) and the single
  transaction that commits the fact together with its queued delivery. src/api/captures.rs now
  uses these, so the two routes share one implementation rather than two copies (net −37 lines
  there).
• src/api/context.rs — the two routes. Add resolves an item-linked context against the live
  calendar to derive life-item:<sha256> (refusing rather than inventing a tag), records the fact
  and enqueues the delivery in one transaction, and reports the delivery state the durable row
  actually holds. Search resolves the item live for the tag and for the state it reports, sends
  one recall in one scope, labels the hits against the ledger, reads counts from the journal, and
  returns the live read never from item_projection.
• src/hindsight/client.rs — HindsightClient::recall, one request, no follow-up.
• src/cli.rs — life context add --input - [--operation-id KEY] (write, operation key required)
  and life context search [--input - | --query TEXT [--item-id ID]] (read, no key), with terminal
  rendering that puts the live state first.

Degradation is explicit in both directions. Hindsight disabled or unreachable leaves the
deterministic parts working under context_unavailable, recorded-but-undelivered context is
context_pending from the journal's rows, superseded recollections carry context_stale, and no
context failure can alter a calendar result.

Note on the recall route

docs/technical/hindsight.md's endpoint table lists recall as POST /v1/default/banks/{bank_id}/recall.
The reviewed OpenAPI 0.10.1 that same page pins compatibility to serves it at
/v1/default/banks/{bank_id}/memories/recall (its retain and operations rows match the table exactly,
so the table's recall row looks like the odd one out). This change uses the pinned deployment's route,
memories/recall, so a real deployment answers rather than 404s.

Verification

cargo test — 26 targets, all green (480 passed, 0 failed, exit 0; up from 448).
cargo clippy --all-targets -- -D warnings
and cargo fmt --check clean. The rebase onto origin/main (which now carries #34's src/api/mod.rs
changes) auto-merged cleanly; the merged router registers both sides and the suite was re-run on the
rebased tree.

tests/context_tests.rs (14 tests) drives the real router over TCP against a mock Hindsight that
records every request and implements all_strict faithfully, and a mock Radicale that serves the
live calendar, with the daemon's SQLite file read back. Per criterion:

• Strict item scoping — the captured request body is asserted, not just the filtered outcome:
  {"query":"Why was an outline chosen first?","tags":["life-item:c84962…"],"tags_match":"all_strict","budget":"mid","max_tokens":1024,"trace":false}.
  With an item-tagged, a foreign-tagged and an untagged memory present, the item query returns only
  the item's own memory.
• No widening — an item-scoped query against a bank holding only foreign/untagged memories
  returns hits: [] and issues exactly one recall request, whose body carries the tag and
  all_strict. A separate explicit bank-wide query finds all three memories, and its captured body
  carries no tags/tags_match field at all.
• Live authority — with a VTODO at STATUS:COMPLETED and a recollection saying the outline was
  postponed, the answer reports live.authority = "live_calendar", live.item.status = "COMPLETED",
  a statement beginning Item … is COMPLETED (observed live in CalDAV at …) followed by
  Historical context from evt_123 indicated a deferral (…), and the hit flagged
  contradicts_live_state: true. The test also asserts one live GET against Radicale and zero
  item_projection rows, so the state came from the live read and not the cached projection.
• Supersession — C1 then C2 (supersedes_context_id = C1's source event) are recorded through the
  route; the older recalled memory comes back is_stale: true, superseded_by: <C2 event id>, with a
  context_stale warning naming the superseding event, while C2 is not stale.
• Bounded retrieval & journal counts — with recall_max_tokens = 256 the captured body carries
  max_tokens: 256 and the response reports recall_max_tokens: 256; with one memory recalled and
  two journal context events, the response reports journal.context_events: 2 (not hits.len()).
  A supplied temporal_window is asserted on the request as a preference.
• Degradation — against a closed port the search returns partial with context_unavailable,
  empty hits (never a silent "there is no context"), and the live COMPLETED state, while
  GET /v1/items/{id} beside it returns the unaltered calendar result with no warnings. A
  Hindsight-disabled deployment reports context_unavailable and dispatches nothing, and a context
  add still records the fact and the durable outbox row.
• Add records and queues — the returned source_event_id matches the life_events row
  (kind: context_captured, collection_id/uid set, provenance containing /v1/context); the
  outbox row is pending, addressed to life-event:<event_id>, and its stored retain body carries
  tags: ["life-item:<sha256>"], metadata.source_event_id, update_mode: replace and
  observation_scopes: combined. Replaying the operation records nothing new. The queued delivery is
  then dispatched by the existing OutboxWorker (one pass → completed, one retain body
  captured with the item tag) and the item-scoped search finds the delivered memory.
• CLI contract — life context add --input - --operation-id KEY --json exits 6 with
  context_delivery: "pending"; an item-scoped life context search --input - --json exits 6 with
  the live state, the labelled hit and context_pending; a bank-wide search with nothing pending
  exits 0 with status: "ok"; an unrecordable payload exits 2 with invalid_input.

Related Issues

Closes #18

## Motivation Issue #18 makes semantic context retrieval safe to rely on. Hindsight is semantic and bounded: it can rank an outdated rationale first, it knows nothing about what the calendar says *now*, and a recall for one item can silently return a decision that belongs to another. Life therefore has to own the rules around it: - **Strict item scoping (I02).** An item-scoped query must send exactly `tags: ["life-item:<sha256>"]` with `tags_match: "all_strict"`, which excludes untagged and foreign-tagged memories. An empty result is a valid answer, and widening to a bank-wide query would leak unrelated private decisions across tasks. - **Live calendar authority (W11).** Recalled text is a recollection, never current state. A task the live calendar reports `COMPLETED` must not be answered as "postponed" because an embedding still ranks that memory highly. - **Supersession provenance (I05).** A correction appends a new source event; upstream does not forget the old one. The journal's own facts say which event supersedes which, so the recalled memory can be labelled rather than presented as the current preference. - **Bounded retrieval.** Recall is semantic, not an enumeration: counts and coverage come from SQLite, never from a result set. ## Changes Adds `POST /v1/context` and `POST /v1/context/search` (`life context add` / `life context search`) on top of the #16 adapter and the #17 outbox — no second queue, no second table, no second worker. - `src/hindsight/recall.rs` — the typed recall request/response. `RecallScope` is part of the request type: an item-scoped query cannot be built without its tag and is fixed to `all_strict`, and a bank-wide read is a separate, explicit constructor. There is no expression that starts item-scoped and ends bank-wide. - `src/hindsight/provenance.rs` — the local supersession ledger, read from `life_events` (`correction_target` / `provenance_json.supersedes_context_id`), resolved to the end of the supersession chain and applied to each recalled memory as `is_stale` + `superseded_by`, plus the journal counts (`context_events`, `undelivered_context_events`). No migration and no new table: the ledger *is* the append-only journal. - `src/hindsight/why.rs` — live-calendar authority. Live state leads the answer; every recollection is labelled historical, and one that asserts a state contradicting the live `STATUS` (over a short, documented, deterministic vocabulary) is flagged and quoted as historical context. - `src/hindsight/context.rs` — the parts captures and context-add must not diverge on: the derived source-event identity, the retain body (item tag vs. stable context tag) and the single transaction that commits the fact together with its queued delivery. `src/api/captures.rs` now uses these, so the two routes share one implementation rather than two copies (net −37 lines there). - `src/api/context.rs` — the two routes. Add resolves an item-linked context against the live calendar to derive `life-item:<sha256>` (refusing rather than inventing a tag), records the fact and enqueues the delivery in one transaction, and reports the delivery state the durable row actually holds. Search resolves the item live for the tag *and* for the state it reports, sends one recall in one scope, labels the hits against the ledger, reads counts from the journal, and returns the live read never from `item_projection`. - `src/hindsight/client.rs` — `HindsightClient::recall`, one request, no follow-up. - `src/cli.rs` — `life context add --input - [--operation-id KEY]` (write, operation key required) and `life context search [--input - | --query TEXT [--item-id ID]]` (read, no key), with terminal rendering that puts the live state first. Degradation is explicit in both directions. Hindsight disabled or unreachable leaves the deterministic parts working under `context_unavailable`, recorded-but-undelivered context is `context_pending` from the journal's rows, superseded recollections carry `context_stale`, and no context failure can alter a calendar result. ### Note on the recall route `docs/technical/hindsight.md`'s endpoint table lists recall as `POST /v1/default/banks/{bank_id}/recall`. The reviewed OpenAPI 0.10.1 that same page pins compatibility to serves it at `/v1/default/banks/{bank_id}/memories/recall` (its retain and operations rows match the table exactly, so the table's recall row looks like the odd one out). This change uses the pinned deployment's route, `memories/recall`, so a real deployment answers rather than 404s. ## Verification `cargo test` — 26 targets, all green (480 passed, 0 failed, exit 0; up from 448). `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` clean. The rebase onto `origin/main` (which now carries #34's `src/api/mod.rs` changes) auto-merged cleanly; the merged router registers both sides and the suite was re-run on the rebased tree. `tests/context_tests.rs` (14 tests) drives the real router over TCP against a mock Hindsight that records every request and implements `all_strict` faithfully, and a mock Radicale that serves the live calendar, with the daemon's SQLite file read back. Per criterion: - **Strict item scoping** — the *captured request body* is asserted, not just the filtered outcome: `{"query":"Why was an outline chosen first?","tags":["life-item:c84962…"],"tags_match":"all_strict","budget":"mid","max_tokens":1024,"trace":false}`. With an item-tagged, a foreign-tagged and an untagged memory present, the item query returns only the item's own memory. - **No widening** — an item-scoped query against a bank holding only foreign/untagged memories returns `hits: []` and issues exactly **one** recall request, whose body carries the tag and `all_strict`. A separate explicit bank-wide query finds all three memories, and its captured body carries no `tags`/`tags_match` field at all. - **Live authority** — with a VTODO at `STATUS:COMPLETED` and a recollection saying the outline was postponed, the answer reports `live.authority = "live_calendar"`, `live.item.status = "COMPLETED"`, a statement beginning `Item … is COMPLETED (observed live in CalDAV at …)` followed by `Historical context from evt_123 indicated a deferral (…)`, and the hit flagged `contradicts_live_state: true`. The test also asserts one live `GET` against Radicale and **zero** `item_projection` rows, so the state came from the live read and not the cached projection. - **Supersession** — C1 then C2 (`supersedes_context_id` = C1's source event) are recorded through the route; the older recalled memory comes back `is_stale: true`, `superseded_by: <C2 event id>`, with a `context_stale` warning naming the superseding event, while C2 is not stale. - **Bounded retrieval & journal counts** — with `recall_max_tokens = 256` the captured body carries `max_tokens: 256` and the response reports `recall_max_tokens: 256`; with one memory recalled and two journal context events, the response reports `journal.context_events: 2` (not `hits.len()`). A supplied `temporal_window` is asserted on the request as a preference. - **Degradation** — against a closed port the search returns `partial` with `context_unavailable`, empty hits (never a silent "there is no context"), and the live `COMPLETED` state, while `GET /v1/items/{id}` beside it returns the unaltered calendar result with no warnings. A Hindsight-disabled deployment reports `context_unavailable` and dispatches nothing, and a context add still records the fact and the durable outbox row. - **Add records and queues** — the returned `source_event_id` matches the `life_events` row (`kind: context_captured`, `collection_id`/`uid` set, provenance containing `/v1/context`); the outbox row is `pending`, addressed to `life-event:<event_id>`, and its stored retain body carries `tags: ["life-item:<sha256>"]`, `metadata.source_event_id`, `update_mode: replace` and `observation_scopes: combined`. Replaying the operation records nothing new. The queued delivery is then dispatched by the **existing** `OutboxWorker` (one pass → `completed`, one retain body captured with the item tag) and the item-scoped search finds the delivered memory. - **CLI contract** — `life context add --input - --operation-id KEY --json` exits 6 with `context_delivery: "pending"`; an item-scoped `life context search --input - --json` exits 6 with the live state, the labelled hit and `context_pending`; a bank-wide search with nothing pending exits 0 with `status: "ok"`; an unrecordable payload exits 2 with `invalid_input`. ## Related Issues Closes #18

yuris added 1 commit 2026-09-23 00:01:48 +00:00

feat(hindsight): strict context search and supersession provenance 85d024f096

yuris commented 2026-09-23 00:03:23 +00:00

Author

Owner

Copy link

Coordinator review — APPROVED

Verified 85d024f on a tree based on current main (1c35748, so #34's lock and drain are present). Independent verification: cargo test 480 passed / 0 failed (14 new context integration tests plus unit tests); cargo clippy --all-targets -- -D warnings 0 errors; cargo fmt --check clean.

What I checked beyond the summary, and why each matters:

• "Never widens on empty" is structural, not a policy. TagsMatch has only the all_strict variant representable, RecallQuery's tags/tags_match are private and set only by the item_scoped/bank_wide constructors, and RecallScope is part of the query — so there is no code path that starts item-scoped and ends bank-wide. The test additionally asserts the captured body carries the tag plus all_strict and that exactly one recall request was issued.
• Live authority is proven by what was not read. the_contextual_why_prefers_live_state_over_a_contradicting_recollection asserts caldav.live_reads() == 1 and that the item_projection row count is 0 — that is the evidence that the answer came from CalDAV rather than a cached projection, which is what the criterion actually requires. The contradiction is resolved in favour of live state with the recalled claim labelled.
• Supersession is derived from the journal, not from Hindsight's claim. SupersessionLedger reads life_events.provenance_json.supersedes_context_id by SQL and walks the chain to its end, so a correction-of-a-correction shows the newest rationale.
• The chain walk is bounded, and I checked why that matters. A reciprocal supersession pair would otherwise spin forever in superseding(). It is capped by MAX_SUPERSESSION_HOPS with a comment naming the cycle case explicitly and a defined fallback (report the last event reached, and the memory is superseded either way). That is the right handling for input the journal cannot currently forbid.
• The "check did not run" distinction is carried through again: a None ledger means the journal could not be read, is disclosed separately, and a hit is never marked fresh on the strength of a check that did not happen — the same honesty rule I asked for in the item-get path.
• Scope was respected: enqueue goes through #17's outbox (commit_operation_result_with_delivery + NewContextDelivery::hindsight), captures.rs now shares those helpers (net −37 lines, no duplication), and no second queue, table or worker appears.

Cargo.toml untouched, so the deferred authors chore stays pending — it lands with #35, which mandates the subtle crate.

Merging. This completes Phase 3.

## Coordinator review — APPROVED Verified `85d024f` on a tree based on current `main` (`1c35748`, so #34's lock and drain are present). Independent verification: `cargo test` **480 passed / 0 failed** (14 new context integration tests plus unit tests); `cargo clippy --all-targets -- -D warnings` 0 errors; `cargo fmt --check` clean. What I checked beyond the summary, and why each matters: - **"Never widens on empty" is structural, not a policy.** `TagsMatch` has only the `all_strict` variant representable, `RecallQuery`'s `tags`/`tags_match` are private and set only by the `item_scoped`/`bank_wide` constructors, and `RecallScope` is part of the query — so there is no code path that starts item-scoped and ends bank-wide. The test additionally asserts the captured body carries the tag plus `all_strict` and that exactly one recall request was issued. - **Live authority is proven by what was *not* read.** `the_contextual_why_prefers_live_state_over_a_contradicting_recollection` asserts `caldav.live_reads() == 1` **and** that the `item_projection` row count is `0` — that is the evidence that the answer came from CalDAV rather than a cached projection, which is what the criterion actually requires. The contradiction is resolved in favour of live state with the recalled claim labelled. - **Supersession is derived from the journal, not from Hindsight's claim.** `SupersessionLedger` reads `life_events.provenance_json.supersedes_context_id` by SQL and walks the chain to its end, so a correction-of-a-correction shows the newest rationale. - **The chain walk is bounded, and I checked why that matters.** A reciprocal supersession pair would otherwise spin forever in `superseding()`. It is capped by `MAX_SUPERSESSION_HOPS` with a comment naming the cycle case explicitly and a defined fallback (report the last event reached, and the memory is superseded either way). That is the right handling for input the journal cannot currently forbid. - **The "check did not run" distinction is carried through again:** a `None` ledger means the journal could not be read, is disclosed separately, and a hit is never marked fresh on the strength of a check that did not happen — the same honesty rule I asked for in the item-get path. - Scope was respected: enqueue goes through #17's outbox (`commit_operation_result_with_delivery` + `NewContextDelivery::hindsight`), `captures.rs` now shares those helpers (net −37 lines, no duplication), and no second queue, table or worker appears. `Cargo.toml` untouched, so the deferred authors chore stays pending — it lands with #35, which mandates the `subtle` crate. Merging. This completes Phase 3.

yuris merged commit 9a5e9094cb into main 2026-09-23 00:03:24 +00:00

yuris referenced this pull request from a commit 2026-09-23 00:03:26 +00:00

feat(hindsight): strict context search and supersession provenance (#61)

yuris changed title from feat(hindsight): strict context search and supersession provenance (#18) to feat(hindsight): strict context search and supersession provenance 2026-09-23 05:12:03 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service!61

No description provided.