yuris/life-service

Fork 0

Strict Context Search & Supersession Provenance #18

New issue

Closed

opened 2026-09-22 17:58:56 +00:00 by yuris · 0 comments

yuris commented

2026-09-22 17:58:56 +00:00

Owner

Summary

Implement contextual search (POST /v1/context/search / life context search) and contextual rationale retrieval with strict all_strict tag scoping, supersession tracking, and stale provenance labeling.

Background & Technical Rationale

Semantic memory retrieval via vector embeddings introduces significant operational hazards if left unconstrained: semantic drift, cross-item leakage, outdated context hallucination, and contradiction of confirmed factual state. Life establishes rigorous governance rules for contextual search:

Calendar Authority vs Semantic Recall Invariant (Scenario W11):
Recalled memory text is NEVER authoritative for current calendar or task state!
Consider an operational scenario where a user asks: "Why is the portfolio outline task postponed?" A semantic memory recall might return: "User chose to postpone the portfolio outline until next week." However, live Radicale state shows that the user completed the task yesterday (STATUS: COMPLETED).
Promoting the recalled memory to current authority would hallucinate a false postponement. Under Life's architecture, Radicale live reads are unconditionally authoritative for current state. The system must report the live calendar fact first (COMPLETED), citing the recalled memory strictly as historical background rationale.
Strict Item Scoping & No-Widening Invariant (Scenario I02):
When querying Hindsight for context regarding a specific item (life context search --item-id <id> or internal "Why" resolution):
- The query MUST set tags: ["life-item:<hash>"] and tags_match: "all_strict".
- In Hindsight OpenAPI 0.10.1, all_strict requires that returned memories contain all specified tags and strictly excludes untagged memories.
- No-Widening Rule: If an item-scoped recall returns zero hits, the system MUST NEVER silently widen the search to a bank-wide query. An empty result is valid and simply indicates no context has been recorded for that item. Widening to bank-wide search leaks unrelated decisions and private user thoughts across tasks.
Supersession & Stale Context Tracking (Scenario I05):
When a user updates a rationale, plan, or decision, Life commits a new context event with a supersedes_context_id link in life_events provenance.
Because Hindsight embeddings and consolidated observations may still rank the older memory highly, Life maintains a local supersession ledger in SQLite. When memories are recalled:
- Life extracts the source_event_id metadata from each recalled memory.
- It checks whether that event ID has been superseded in the local ledger.
- If superseded, Life tags the memory with a context_stale warning, discloses the superseding event ID, and presents the newer rationale as authoritative.
Bounded Temporal Semantics:
temporal_window in Hindsight influences temporal retrieval ranking, but is not an exhaustive filter (memories outside the window may still appear in search results). All authoritative counts, ranges, and coverage metrics come from SQLite WAL, not memory recall.

Exact Technical References & Specifications

Normative Recall Request Payload

From docs/technical/hindsight.md (lines 43–45):

{"query":"Why was an outline chosen first?","tags":["life-item:example-derived-tag"],"tags_match":"all_strict","budget":"mid","max_tokens":1024,"trace":false}

Normative recall rules (hindsight.md, lines 47–51):

"all_strict requires all supplied tags and excludes untagged memories. It is not exact set equality (exact is a different schema literal). Metadata linkage alone is not evidence of arbitrary server-side metadata filtering. Validate returned scope/provenance; suppress or label unverifiable occurrence-specific attribution. Generic context search is explicitly bank-wide; item-specific why queries must never accidentally widen to it on an empty result.

temporal_window influences temporal retrieval ranking, not exhaustive filtering; out-of-window memories may return. Recall is semantic and bounded, not a complete event enumeration. Counts and coverage always come from SQLite. Current tasks must be reread from Radicale even if recalled text looks recent.

Corrections/supersessions create new source events and links; mark supersedes_context_id in Life provenance and deliver an explicit supersession statement. Do not assume all derived observations instantly forget old facts. On recall, compare source identities with the local supersession ledger, label stale/unverified provenance, prefer current verified context and warn context_stale when consolidation is unresolved. If an upstream result lacks traceable sources, present it as an interpretation, not an authoritative current preference. Destructive upstream deletion is an exceptional operator privacy workflow, not ordinary append-only correction."

Verification Matrix Scenarios

From docs/technical/verification.md (lines 25, 116, 119):

ID	Requirement / scenario	Input / action	Observable acceptance
W11	Why	Item question with old recalled status but different live status	Live status wins; rationale sources/freshness shown; no recalled state promoted to authority
I02	Strict item recall	Unrelated and untagged memories present	all_strict excludes untagged/unrelated results; metadata alone is not a server filter
I05	Superseded context	Superseded context and temporal_window outside-range hit	Stale provenance labeled; no exhaustive time-filter/count assertion

CLI & API Routes and Payloads

From docs/technical/cli-api.md:

Route table (lines 46–47):
- life context add --input -: POST /v1/context (scope: write)
- life context search --input -: POST /v1/context/search (scope: read)
Command payloads and results (lines 93–94):
- Context add: text, optional item_id, reason, supersedes_context_id \rightarrow operation_id, source event ID, delivery state.
- Context search: query, optional item_id and bounded temporal preference \rightarrow context hits with source/document IDs, freshness and superseded markers.
Warning identifiers (line 127): context_stale, context_unavailable.

Section Citations

docs/technical/hindsight.md: Section Recall and stale context (lines 39–53), Section Compatibility requirements and sources (lines 53–57).
docs/technical/verification.md: Section Workflow scenarios (line 25, Scenario W11), Section Integration scenarios (lines 116, 119, Scenarios I02, I05).
docs/technical/cli-api.md: Section Command-to-route table (lines 46–47), Section Command payloads and results (lines 93–94), Section Error identifiers (line 127).

What to Build

1. Recall Query Builder (`src/hindsight/recall.rs`)

Define typed recall structures:

#[derive(Debug, Serialize, Deserialize)]
pub struct RecallQuery {
    pub query: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub tags: Option<Vec<String>>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub tags_match: Option<String>, // "all_strict"
    pub budget: String,             // "low" | "mid" | "high"
    pub max_tokens: usize,
    pub trace: bool,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct ContextHit {
    pub content: String,
    pub context: Option<String>,
    pub document_id: String,
    pub source_event_id: Option<String>,
    pub timestamp: Option<String>,
    pub tags: Vec<String>,
    pub is_stale: bool,
    pub superseded_by: Option<String>,
}

Implement search_context:
- When searching for a specific item (item_id provided):
  - Resolve canonical item tags (life-item:<sha256>).
  - Build RecallQuery with tags = vec![item_tag] and tags_match = "all_strict".
  - Dispatches to Hindsight POST /v1/default/banks/{bank_id}/recall.
  - If response returns zero hits, return empty results immediately. NEVER fall back to bank-wide search.
- When searching generally (item_id omitted):
  - Build RecallQuery without tags; search across entire bank.

2. Supersession Ledger & Provenance Filter (`src/hindsight/provenance.rs`)

Query local SQLite life_events table to build the supersession map:

SELECT event_id, correction_target, json_extract(provenance_json, '$.supersedes_context_id') as supersedes_id
FROM life_events
WHERE correction_target IS NOT NULL OR supersedes_id IS NOT NULL;

For each memory hit returned by Hindsight:
- Extract source_event_id from metadata.
- Traverse supersession chain in SQLite: if source_event_id appears as supersedes_id or correction_target of a newer event, mark the hit as is_stale = true and superseded_by = Some(newer_event_id).
- Attach a structured warning to the response envelope:
```
{
  "code": "context_stale",
  "message": "Recalled context from event evt_old was superseded by evt_new"
}
```

3. Calendar Authority Arbitrator (`src/hindsight/why.rs`)

Implement the "Why" explanation resolver:
- When asked why an item has a specific state or deadline:
  1. Read live item state directly from Radicale (authoritative CalDAV fetch). Do not substitute cached item_projection without live confirmation; if Radicale is unavailable, report calendar unavailability rather than promoting cached state to live authority (docs/technical/hindsight.md:49).
  2. Query Hindsight for item-scoped context.
  3. If the recalled text makes a factual claim about item state (e.g. "task deferred") that contradicts live status (e.g. STATUS: COMPLETED):
    - Live calendar state takes unconditional precedence.
    - The response formats: "Item itm_demo is COMPLETED (observed live in CalDAV). Historical context from evt_123 indicated a deferral reason: '...'."
    - Under no circumstances is the recalled state presented as current truth.

4. Axum Routes & CLI Handlers

Implement routes in src/api.rs:
- POST /v1/context:
  - Validates idempotency key.
  - Commits context entry to life_events with optional supersedes_context_id.
  - Queues delivery into outbox.
  - Returns operation_id, source event_id, and context_delivery: "pending".
- POST /v1/context/search:
  - Accepts { "query": String, "item_id": Option<String> }.
  - Returns { "hits": Vec<ContextHit> } with warnings.
Implement CLI in src/cli.rs:
- life context add --input - [--operation-id KEY] [--json]
- life context search --input - [--json]

5. Integration Test Suite (`tests/hindsight_search_tests.rs`)

Test Scenario I02 (Strict item scoping):
- Mock Hindsight with untagged memories, memories tagged life-item:other, and memories tagged life-item:target.
- Query with item_id of target.
- Assert that request wire payload sent tags_match: "all_strict".
- Assert that mock excludes untagged and differently tagged memories.
- Assert that an empty search for an item does not trigger a secondary bank-wide request.
Test Scenario W11 (Live calendar authority):
- Target task in Radicale has STATUS: COMPLETED.
- Hindsight memory states "Task postponed until next week".
- Execute "Why" query.
- Assert output reports COMPLETED as authoritative, labeling memory as historical rationale.
Test Scenario I05 (Supersession tracking):
- Seed context C_1 (evt_01), then record context C_2 (evt_02) with supersedes_context_id: "evt_01".
- Trigger search that returns C_1.
- Assert response includes C_1 with is_stale: true, superseded_by: "evt_02", and context_stale warning.

Acceptance Criteria

Item-scoped context queries strictly include tags: ["life-item:<hash>"] and tags_match: "all_strict", excluding untagged or foreign-tagged memories (Scenario I02).
An empty item-scoped recall result returns an empty list and NEVER widens to bank-wide search (Scenario I02).
Live Radicale calendar/task state is unconditionally authoritative over recalled memory claims; contradictions are resolved in favor of live calendar state, and live reads directly query Radicale rather than relying on cached projections (Scenario W11).
Superseded memories returned by Hindsight are flagged with is_stale: true, cite the superseding event ID, and emit context_stale warnings (Scenario I05).
Context add records source events and outbox queue entries with optional supersedes_context_id.
CLI and API commands match life context add and life context search contracts.

Notes

Enqueue reuses #17's outbox and capture helpers (no second queue/table/worker); captures.rs was refactored onto the same helpers.
all_strict and the item tag are structurally enforced (private fields, constructors only), and the supersession chain walk is bounded against a reciprocal-pair cycle.
No dependency added, so the deferred Cargo.toml authors chore lands with #35.

Blocked by

#017: Durable Context Outbox Queue & Background Delivery Worker

## Summary Implement contextual search (`POST /v1/context/search` / `life context search`) and contextual rationale retrieval with strict `all_strict` tag scoping, supersession tracking, and stale provenance labeling. ## Background & Technical Rationale Semantic memory retrieval via vector embeddings introduces significant operational hazards if left unconstrained: semantic drift, cross-item leakage, outdated context hallucination, and contradiction of confirmed factual state. Life establishes rigorous governance rules for contextual search: 1. **Calendar Authority vs Semantic Recall Invariant (Scenario W11):** Recalled memory text is NEVER authoritative for current calendar or task state! Consider an operational scenario where a user asks: *"Why is the portfolio outline task postponed?"* A semantic memory recall might return: *"User chose to postpone the portfolio outline until next week."* However, live Radicale state shows that the user completed the task yesterday (`STATUS: COMPLETED`). Promoting the recalled memory to current authority would hallucinate a false postponement. Under Life's architecture, Radicale live reads are unconditionally authoritative for current state. The system must report the live calendar fact first (`COMPLETED`), citing the recalled memory strictly as historical background rationale. 2. **Strict Item Scoping & No-Widening Invariant (Scenario I02):** When querying Hindsight for context regarding a specific item (`life context search --item-id <id>` or internal "Why" resolution): - The query MUST set `tags: ["life-item:<hash>"]` and `tags_match: "all_strict"`. - In Hindsight OpenAPI 0.10.1, `all_strict` requires that returned memories contain *all* specified tags and strictly *excludes* untagged memories. - **No-Widening Rule:** If an item-scoped recall returns zero hits, the system MUST NEVER silently widen the search to a bank-wide query. An empty result is valid and simply indicates no context has been recorded for that item. Widening to bank-wide search leaks unrelated decisions and private user thoughts across tasks. 3. **Supersession & Stale Context Tracking (Scenario I05):** When a user updates a rationale, plan, or decision, Life commits a new context event with a `supersedes_context_id` link in `life_events` provenance. Because Hindsight embeddings and consolidated observations may still rank the older memory highly, Life maintains a local supersession ledger in SQLite. When memories are recalled: - Life extracts the `source_event_id` metadata from each recalled memory. - It checks whether that event ID has been superseded in the local ledger. - If superseded, Life tags the memory with a `context_stale` warning, discloses the superseding event ID, and presents the newer rationale as authoritative. 4. **Bounded Temporal Semantics:** `temporal_window` in Hindsight influences temporal retrieval ranking, but is not an exhaustive filter (memories outside the window may still appear in search results). All authoritative counts, ranges, and coverage metrics come from SQLite WAL, not memory recall. ## Exact Technical References & Specifications ### Normative Recall Request Payload From `docs/technical/hindsight.md` (lines 43–45): ```json {"query":"Why was an outline chosen first?","tags":["life-item:example-derived-tag"],"tags_match":"all_strict","budget":"mid","max_tokens":1024,"trace":false} ``` Normative recall rules (`hindsight.md`, lines 47–51): > "`all_strict` requires all supplied tags and excludes untagged memories. It is not exact set equality (`exact` is a different schema literal). Metadata linkage alone is not evidence of arbitrary server-side metadata filtering. Validate returned scope/provenance; suppress or label unverifiable occurrence-specific attribution. Generic context search is explicitly bank-wide; item-specific why queries must never accidentally widen to it on an empty result. > > `temporal_window` influences temporal retrieval ranking, not exhaustive filtering; out-of-window memories may return. Recall is semantic and bounded, not a complete event enumeration. Counts and coverage always come from SQLite. Current tasks must be reread from Radicale even if recalled text looks recent. > > Corrections/supersessions create new source events and links; mark supersedes_context_id in Life provenance and deliver an explicit supersession statement. Do not assume all derived observations instantly forget old facts. On recall, compare source identities with the local supersession ledger, label stale/unverified provenance, prefer current verified context and warn context_stale when consolidation is unresolved. If an upstream result lacks traceable sources, present it as an interpretation, not an authoritative current preference. Destructive upstream deletion is an exceptional operator privacy workflow, not ordinary append-only correction." ### Verification Matrix Scenarios From `docs/technical/verification.md` (lines 25, 116, 119): | ID | Requirement / scenario | Input / action | Observable acceptance | |---|---|---|---| | W11 | [Why](hindsight.md#recall-and-stale-context) | Item question with old recalled status but different live status | Live status wins; rationale sources/freshness shown; no recalled state promoted to authority | | I02 | Strict item recall | Unrelated and untagged memories present | all_strict excludes untagged/unrelated results; metadata alone is not a server filter | | I05 | Superseded context | Superseded context and temporal_window outside-range hit | Stale provenance labeled; no exhaustive time-filter/count assertion | ### CLI & API Routes and Payloads From `docs/technical/cli-api.md`: - Route table (lines 46–47): - `life context add --input -`: `POST /v1/context` (scope: `write`) - `life context search --input -`: `POST /v1/context/search` (scope: `read`) - Command payloads and results (lines 93–94): - `Context add`: `text`, optional `item_id`, `reason`, `supersedes_context_id` $\rightarrow$ `operation_id`, source event ID, delivery state. - `Context search`: `query`, optional `item_id` and bounded temporal preference $\rightarrow$ context hits with source/document IDs, freshness and superseded markers. - Warning identifiers (line 127): `context_stale`, `context_unavailable`. ### Section Citations - `docs/technical/hindsight.md`: Section *Recall and stale context* (lines 39–53), Section *Compatibility requirements and sources* (lines 53–57). - `docs/technical/verification.md`: Section *Workflow scenarios* (line 25, Scenario W11), Section *Integration scenarios* (lines 116, 119, Scenarios I02, I05). - `docs/technical/cli-api.md`: Section *Command-to-route table* (lines 46–47), Section *Command payloads and results* (lines 93–94), Section *Error identifiers* (line 127). ## What to Build ### 1. Recall Query Builder (`src/hindsight/recall.rs`) - Define typed recall structures: ```rust #[derive(Debug, Serialize, Deserialize)] pub struct RecallQuery { pub query: String, #[serde(skip_serializing_if = "Option::is_none")] pub tags: Option<Vec<String>>, #[serde(skip_serializing_if = "Option::is_none")] pub tags_match: Option<String>, // "all_strict" pub budget: String, // "low" | "mid" | "high" pub max_tokens: usize, pub trace: bool, } #[derive(Debug, Serialize, Deserialize)] pub struct ContextHit { pub content: String, pub context: Option<String>, pub document_id: String, pub source_event_id: Option<String>, pub timestamp: Option<String>, pub tags: Vec<String>, pub is_stale: bool, pub superseded_by: Option<String>, } ``` - Implement `search_context`: - When searching for a specific item (`item_id` provided): - Resolve canonical item tags (`life-item:<sha256>`). - Build `RecallQuery` with `tags = vec![item_tag]` and `tags_match = "all_strict"`. - Dispatches to Hindsight `POST /v1/default/banks/{bank_id}/recall`. - If response returns zero hits, return empty results immediately. NEVER fall back to bank-wide search. - When searching generally (`item_id` omitted): - Build `RecallQuery` without tags; search across entire bank. ### 2. Supersession Ledger & Provenance Filter (`src/hindsight/provenance.rs`) - Query local SQLite `life_events` table to build the supersession map: ```sql SELECT event_id, correction_target, json_extract(provenance_json, '$.supersedes_context_id') as supersedes_id FROM life_events WHERE correction_target IS NOT NULL OR supersedes_id IS NOT NULL; ``` - For each memory hit returned by Hindsight: - Extract `source_event_id` from metadata. - Traverse supersession chain in SQLite: if `source_event_id` appears as `supersedes_id` or `correction_target` of a newer event, mark the hit as `is_stale = true` and `superseded_by = Some(newer_event_id)`. - Attach a structured warning to the response envelope: ```json { "code": "context_stale", "message": "Recalled context from event evt_old was superseded by evt_new" } ``` ### 3. Calendar Authority Arbitrator (`src/hindsight/why.rs`) - Implement the "Why" explanation resolver: - When asked why an item has a specific state or deadline: 1. Read live item state directly from Radicale (authoritative CalDAV fetch). Do not substitute cached `item_projection` without live confirmation; if Radicale is unavailable, report calendar unavailability rather than promoting cached state to live authority (`docs/technical/hindsight.md:49`). 2. Query Hindsight for item-scoped context. 3. If the recalled text makes a factual claim about item state (e.g. "task deferred") that contradicts live status (e.g. `STATUS: COMPLETED`): - Live calendar state takes unconditional precedence. - The response formats: *"Item itm_demo is COMPLETED (observed live in CalDAV). Historical context from evt_123 indicated a deferral reason: '...'."* - Under no circumstances is the recalled state presented as current truth. ### 4. Axum Routes & CLI Handlers - Implement routes in `src/api.rs`: - `POST /v1/context`: - Validates idempotency key. - Commits context entry to `life_events` with optional `supersedes_context_id`. - Queues delivery into `outbox`. - Returns `operation_id`, source `event_id`, and `context_delivery: "pending"`. - `POST /v1/context/search`: - Accepts `{ "query": String, "item_id": Option<String> }`. - Returns `{ "hits": Vec<ContextHit> }` with warnings. - Implement CLI in `src/cli.rs`: - `life context add --input - [--operation-id KEY] [--json]` - `life context search --input - [--json]` ### 5. Integration Test Suite (`tests/hindsight_search_tests.rs`) - Test Scenario I02 (Strict item scoping): - Mock Hindsight with untagged memories, memories tagged `life-item:other`, and memories tagged `life-item:target`. - Query with `item_id` of target. - Assert that request wire payload sent `tags_match: "all_strict"`. - Assert that mock excludes untagged and differently tagged memories. - Assert that an empty search for an item does not trigger a secondary bank-wide request. - Test Scenario W11 (Live calendar authority): - Target task in Radicale has `STATUS: COMPLETED`. - Hindsight memory states "Task postponed until next week". - Execute "Why" query. - Assert output reports `COMPLETED` as authoritative, labeling memory as historical rationale. - Test Scenario I05 (Supersession tracking): - Seed context $C_1$ (`evt_01`), then record context $C_2$ (`evt_02`) with `supersedes_context_id: "evt_01"`. - Trigger search that returns $C_1$. - Assert response includes $C_1$ with `is_stale: true`, `superseded_by: "evt_02"`, and `context_stale` warning. ## Acceptance Criteria - [x] Item-scoped context queries strictly include `tags: ["life-item:<hash>"]` and `tags_match: "all_strict"`, excluding untagged or foreign-tagged memories (Scenario I02). - [x] An empty item-scoped recall result returns an empty list and NEVER widens to bank-wide search (Scenario I02). - [x] Live Radicale calendar/task state is unconditionally authoritative over recalled memory claims; contradictions are resolved in favor of live calendar state, and live reads directly query Radicale rather than relying on cached projections (Scenario W11). - [x] Superseded memories returned by Hindsight are flagged with `is_stale: true`, cite the superseding event ID, and emit `context_stale` warnings (Scenario I05). - [x] Context add records source events and outbox queue entries with optional `supersedes_context_id`. - [x] CLI and API commands match `life context add` and `life context search` contracts. ## Notes - Enqueue reuses #17's outbox and capture helpers (no second queue/table/worker); `captures.rs` was refactored onto the same helpers. - `all_strict` and the item tag are structurally enforced (private fields, constructors only), and the supersession chain walk is bounded against a reciprocal-pair cycle. - No dependency added, so the deferred `Cargo.toml` authors chore lands with #35. ## Blocked by - #017: Durable Context Outbox Queue & Background Delivery Worker

yuris referenced this issue

2026-09-22 17:58:57 +00:00

Next-Task Policy Filter & Candidate Window Slicing #22

yuris changed title from ~~Issue #018: Strict Context Search & Supersession Provenance~~ to Strict Context Search & Supersession Provenance

2026-09-22 20:02:37 +00:00

yuris referenced this issue

2026-09-22 23:37:13 +00:00

feat(hindsight): durable context outbox and background delivery worker #59

yuris referenced this issue

2026-09-22 23:44:19 +00:00

Durable Context Outbox Queue & Background Delivery Worker #17

yuris referenced this issue from a pull request that will close it,