yuris/life-service
1
0
Fork 0
Code Issues Pull requests Projects Releases 6 Packages 1 Wiki Activity Actions

Hindsight Reqwest Client & Deterministic Document Identity #16

New issue
Closed
opened 2026-09-22 17:58:55 +00:00 by yuris · 0 comments
yuris commented 2026-09-22 17:58:55 +00:00
Owner
Copy link

Summary

Implement the direct Rust reqwest adapter for Hindsight, enforcing deterministic document identity (life-event:<event_id>), canonical SHA-256 item tags (life-item:<hash>), typed retain requests, and strict replace semantics.

Background & Technical Rationale

Life incorporates Hindsight as a semantic memory and context backend. To maintain architectural purity and rock-solid reliability, the system defines unambiguous authority and identity boundaries:

  1. Semantic Authority vs Current State Authority:
    Hindsight owns semantic context—user preferences, explanations, interpersonal relationships, decisions, and meaningful rationales. It is NEVER authoritative for current calendar or task state. Current schedule, task completion, and deadlines are owned strictly by CalDAV / Radicale. If Hindsight is completely unavailable, calendar operations and agenda queries proceed without interruption.
  2. Direct Rust Adapter Architecture:
    The integration uses a direct, pooled reqwest::Client with rustls communicating over HTTP/HTTPS with the configured Hindsight bank. No external Python, Node.js, or TypeScript sidecar processes or generic agent SDK frameworks are used.
  3. Deterministic Document Identity Invariant (Scenario I01):
    When Life retains a memory derived from a Life event, it must derive a stable, deterministic document ID: 
    \text{document\_id} = \text{"life-event:"} + \text{event\_id}
    Every retain request MUST set update_mode: "replace". Naive retain implementations that append or generate random document UUIDs cause memory duplication, fragmented knowledge bases, and hallucinated conflicting contexts. Re-retaining the same event replaces the existing document rather than accumulating duplicate memories.
  4. Deterministic Item Tag Invariant:
    To scope memories to specific tasks or events, Life derives an item tag: 
    \text{item\_tag} = \text{"life-item:"} + \text{lowercase\_hex}(\text{SHA-256}(\text{canonical\_utf8\_json}([\text{collection\_id}, \text{uid}])
    The canonical JSON array MUST be serialized without superfluous whitespace (e.g. ["personal","fictional-demo-uid"]). This tag deliberately covers the entire recurring series. Occurrence-specific details are recorded as string metadata and verified locally by Life.
  5. Context-Only Records:
    When the user captures pure context or decision rationales not tied to a calendar item, Life uses the stable tag life-context and the source event ID, without inventing a fake calendar UID.
  6. Observation Scope:
    Retain requests configure observation_scopes: "combined" over the stable item tag, allowing Hindsight to consolidate observations across time for that item without tag fragmentation.

Exact Technical References & Specifications

Upstream Endpoint & Responsibility Matrix

From docs/technical/hindsight.md (lines 9–14):

Operation Upstream endpoint Life responsibility
Retain POST /v1/default/banks/{bank_id}/memories Durable outbox, stable document identity, typed validation
Recall POST /v1/default/banks/{bank_id}/recall Bounded semantic retrieval and strict item scoping
Poll GET /v1/default/banks/{bank_id}/operations/{operation_id} Persist known operation status and returned IDs

Normative Retain Request Payload

From docs/technical/hindsight.md (lines 25–27):

{"items":[{"content":"Fictional user chose an outline before implementation to clarify scope.","timestamp":"unset","context":"Explicit fictional decision rationale","document_id":"life-event:evt_demo","update_mode":"replace","metadata":{"collection_id":"personal","uid":"fictional-demo-uid","source_event_id":"evt_demo"},"tags":["life-item:example-derived-tag"],"observation_scopes":"combined"}]}

Normative identity rules (hindsight.md, lines 17–22):

"Derive document_id as life-event:<event_id>. Retry the same event with update_mode: replace, never append or generate a fresh random document. Metadata values are strings: collection_id, uid, source_event_id and optional typed recurrence identity. Store Hindsight-returned memory/operation IDs separately; neither is the Life item ID.

Derive item tag as life-item: plus lowercase SHA-256 of the UTF-8 canonical JSON array [collection_id,uid] (no extra whitespace). The tag deliberately covers the series; occurrence-specific provenance remains metadata and is checked by Life when answering an occurrence question. Context-only records use a stable life-context tag and source event identity, without a fabricated UID. Use combined observation scope over the one stable item tag, not volatile per-event tags that fragment consolidation."

Verification Matrix Scenario I01

From docs/technical/verification.md (lines 113–115):

ID Integration and input Observable acceptance
I01 Hindsight retain same event document twice Stable document_id and replace mode; returned IDs persisted; no exactly-once consolidation claim

Section Citations

  • docs/technical/hindsight.md: Section Authority and adapter (lines 5–16), Section Identity and retain (lines 17–30), Section Compatibility requirements and sources (lines 53–57).
  • docs/technical/verification.md: Section Integration scenarios (lines 113–115, Scenario I01).
  • docs/technical/timeline-consistency.md: Section Context delivery and correction (lines 160–165).

What to Build

1. Deterministic Identity Derivation (src/hindsight/identity.rs)

  • Implement derive_document_id(event_id: &str) -> String:
    • Formats as format!("life-event:{}", event_id).
  • Implement derive_item_tag(collection_id: &str, uid: &str) -> String:
    • Serializes canonical JSON array [&collection_id, &uid] via serde_json::to_string.
    • Computes SHA-256 hash using sha2::Sha256.
    • Encodes hash as lowercase hex string.
    • Formats as format!("life-item:{}", hex).
  • Implement derive_context_tag() -> &'static str:
    • Returns "life-context".
  • Unit tests validating hash stability against known test vectors:
    • Input ["personal", "fictional-demo-uid"] must yield predictable, whitespace-independent SHA-256 hex digest.

2. Upstream OpenAPI 0.10.1 Data Models (src/hindsight/models.rs)

  • Implement typed request and response schemas: 
    #[derive(Debug, Serialize, Deserialize)]
pub struct RetainRequest {
    pub items: Vec<RetainItem>,
    #[serde(rename = "async", skip_serializing_if = "Option::is_none")]
    pub is_async: Option<bool>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct RetainItem {
    pub content: String,
    pub timestamp: String, // ISO8601 UTC timestamp or literal "unset"
    pub context: String,
    pub document_id: String,
    pub update_mode: String, // Must always serialize as "replace"
    pub metadata: HashMap<String, String>,
    pub tags: Vec<String>,
    pub observation_scopes: String, // Must serialize as "combined"
}

#[derive(Debug, Serialize, Deserialize)]
pub struct RetainResponse {
    pub memory_ids: Option<Vec<String>>,
    pub operation_id: Option<String>,
    pub status: Option<String>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct OperationStatusResponse {
    pub operation_id: String,
    pub status: String, // "pending", "in_progress", "completed", "failed"
    pub memory_ids: Option<Vec<String>>,
    pub error: Option<String>,
}

3. Authenticated Hindsight Client (src/hindsight/client.rs)

  • Struct HindsightClient: 
    pub struct HindsightClient {
    client: reqwest::Client,
    base_url: reqwest::Url,
    bank_id: String,
    auth_token: secrecy::SecretString,
}
  • Constructor reading authorization credentials from the configured authorization_file (or token file).
  • Capability gate verifying the deployed Hindsight authentication scheme and compatibility during startup before traffic is dispatched (docs/technical/hindsight.md:7).
  • Method retain(&self, request: RetainRequest) -> Result<RetainResponse, HindsightError>:
    • Dispatches POST /v1/default/banks/{bank_id}/memories.
    • Sets verified Authorization header, Content-Type: application/json.
    • Handles HTTP 200 OK (synchronous retain) and HTTP 202 Accepted (asynchronous retain).
  • Method poll_operation(&self, operation_id: &str) -> Result<OperationStatusResponse, HindsightError>:
    • Dispatches GET /v1/default/banks/{bank_id}/operations/{operation_id}.
  • Error handling: Maps HTTP 401/403 to HindsightError::Unauthorized, 404 to HindsightError::NotFound, 429/503 to HindsightError::Unavailable(retry_after).

4. Conformance Test Suite (tests/hindsight_client_tests.rs)

  • Test deterministic hashing:
    • Verify that varying whitespace or field order in caller code does not affect canonical tag derivation.
  • Test retain wire serialization:
    • Serialize RetainRequest and assert exact JSON match with hindsight.md normative example.
  • Mock server integration test (Scenario I01):
    • Send retain request for life-event:evt_demo.
    • Send identical retain request for life-event:evt_demo a second time with modified content.
    • Verify that both requests sent update_mode: "replace" and identical document_id.
    • Assert that mock server receives update without generating duplicate memory entries.

Acceptance Criteria

  • document_id is strictly and deterministically formatted as life-event:<event_id>.
  • Retain requests serialize update_mode: "replace", observation_scopes: "combined", and string metadata values matching OpenAPI 0.10.1 (Scenario I01).
  • Item tag is deterministically derived as life-item: plus lowercase SHA-256 of the UTF-8 canonical JSON array [collection_id, uid] without whitespace sensitivity.
  • Context-only records without a calendar item use the stable life-context tag and source event ID without fabricating a fake UID.
  • Re-retaining the same event reuses the stable document_id and replace semantics without creating duplicate memories (Scenario I01).
  • Authenticated client securely loads credentials from secret files and handles HTTP 200, 202, 401, 403, and 503 responses appropriately.
  • Deployed authentication scheme is verified against Hindsight before retain requests are enabled, confirming instance compatibility.

Notes

  • Adapter + identity only: the durable outbox table, background polling worker and retry scheduling belong to #17; wiring verify_compatibility into startup belongs with #17/#18.
  • No dependency added (ring/zeroize already present), so the deferred Cargo.toml authors chore stays pending.

Blocked by

  • #005: Two-Phase Commit Journaling & Idempotency Fingerprinting
  • #010: Conditional Task & Event Creation with Prepared Intent
## Summary Implement the direct Rust `reqwest` adapter for Hindsight, enforcing deterministic document identity (`life-event:<event_id>`), canonical SHA-256 item tags (`life-item:<hash>`), typed retain requests, and strict replace semantics. ## Background & Technical Rationale Life incorporates Hindsight as a semantic memory and context backend. To maintain architectural purity and rock-solid reliability, the system defines unambiguous authority and identity boundaries: 1. **Semantic Authority vs Current State Authority:** Hindsight owns *semantic context*—user preferences, explanations, interpersonal relationships, decisions, and meaningful rationales. It is NEVER authoritative for current calendar or task state. Current schedule, task completion, and deadlines are owned strictly by CalDAV / Radicale. If Hindsight is completely unavailable, calendar operations and agenda queries proceed without interruption. 2. **Direct Rust Adapter Architecture:** The integration uses a direct, pooled `reqwest::Client` with `rustls` communicating over HTTP/HTTPS with the configured Hindsight bank. No external Python, Node.js, or TypeScript sidecar processes or generic agent SDK frameworks are used. 3. **Deterministic Document Identity Invariant (Scenario I01):** When Life retains a memory derived from a Life event, it must derive a stable, deterministic document ID: $$\text{document\_id} = \text{"life-event:"} + \text{event\_id}$$ Every retain request MUST set `update_mode: "replace"`. Naive retain implementations that append or generate random document UUIDs cause memory duplication, fragmented knowledge bases, and hallucinated conflicting contexts. Re-retaining the same event replaces the existing document rather than accumulating duplicate memories. 4. **Deterministic Item Tag Invariant:** To scope memories to specific tasks or events, Life derives an item tag: $$\text{item\_tag} = \text{"life-item:"} + \text{lowercase\_hex}(\text{SHA-256}(\text{canonical\_utf8\_json}([\text{collection\_id}, \text{uid}])$$ The canonical JSON array MUST be serialized without superfluous whitespace (e.g. `["personal","fictional-demo-uid"]`). This tag deliberately covers the entire recurring series. Occurrence-specific details are recorded as string metadata and verified locally by Life. 5. **Context-Only Records:** When the user captures pure context or decision rationales not tied to a calendar item, Life uses the stable tag `life-context` and the source event ID, without inventing a fake calendar UID. 6. **Observation Scope:** Retain requests configure `observation_scopes: "combined"` over the stable item tag, allowing Hindsight to consolidate observations across time for that item without tag fragmentation. ## Exact Technical References & Specifications ### Upstream Endpoint & Responsibility Matrix From `docs/technical/hindsight.md` (lines 9–14): | Operation | Upstream endpoint | Life responsibility | |---|---|---| | Retain | POST /v1/default/banks/{bank_id}/memories | Durable outbox, stable document identity, typed validation | | Recall | POST /v1/default/banks/{bank_id}/recall | Bounded semantic retrieval and strict item scoping | | Poll | GET /v1/default/banks/{bank_id}/operations/{operation_id} | Persist known operation status and returned IDs | ### Normative Retain Request Payload From `docs/technical/hindsight.md` (lines 25–27): ```json {"items":[{"content":"Fictional user chose an outline before implementation to clarify scope.","timestamp":"unset","context":"Explicit fictional decision rationale","document_id":"life-event:evt_demo","update_mode":"replace","metadata":{"collection_id":"personal","uid":"fictional-demo-uid","source_event_id":"evt_demo"},"tags":["life-item:example-derived-tag"],"observation_scopes":"combined"}]} ``` Normative identity rules (`hindsight.md`, lines 17–22): > "Derive document_id as `life-event:<event_id>`. Retry the same event with `update_mode: replace`, never append or generate a fresh random document. Metadata values are strings: collection_id, uid, source_event_id and optional typed recurrence identity. Store Hindsight-returned memory/operation IDs separately; neither is the Life item ID. > > Derive item tag as `life-item:` plus lowercase SHA-256 of the UTF-8 canonical JSON array `[collection_id,uid]` (no extra whitespace). The tag deliberately covers the series; occurrence-specific provenance remains metadata and is checked by Life when answering an occurrence question. Context-only records use a stable `life-context` tag and source event identity, without a fabricated UID. Use combined observation scope over the one stable item tag, not volatile per-event tags that fragment consolidation." ### Verification Matrix Scenario I01 From `docs/technical/verification.md` (lines 113–115): | ID | Integration and input | Observable acceptance | |---|---|---| | I01 | Hindsight retain same event document twice | Stable document_id and replace mode; returned IDs persisted; no exactly-once consolidation claim | ### Section Citations - `docs/technical/hindsight.md`: Section *Authority and adapter* (lines 5–16), Section *Identity and retain* (lines 17–30), Section *Compatibility requirements and sources* (lines 53–57). - `docs/technical/verification.md`: Section *Integration scenarios* (lines 113–115, Scenario I01). - `docs/technical/timeline-consistency.md`: Section *Context delivery and correction* (lines 160–165). ## What to Build ### 1. Deterministic Identity Derivation (`src/hindsight/identity.rs`) - Implement `derive_document_id(event_id: &str) -> String`: - Formats as `format!("life-event:{}", event_id)`. - Implement `derive_item_tag(collection_id: &str, uid: &str) -> String`: - Serializes canonical JSON array `[&collection_id, &uid]` via `serde_json::to_string`. - Computes SHA-256 hash using `sha2::Sha256`. - Encodes hash as lowercase hex string. - Formats as `format!("life-item:{}", hex)`. - Implement `derive_context_tag() -> &'static str`: - Returns `"life-context"`. - Unit tests validating hash stability against known test vectors: - Input `["personal", "fictional-demo-uid"]` must yield predictable, whitespace-independent SHA-256 hex digest. ### 2. Upstream OpenAPI 0.10.1 Data Models (`src/hindsight/models.rs`) - Implement typed request and response schemas: ```rust #[derive(Debug, Serialize, Deserialize)] pub struct RetainRequest { pub items: Vec<RetainItem>, #[serde(rename = "async", skip_serializing_if = "Option::is_none")] pub is_async: Option<bool>, } #[derive(Debug, Serialize, Deserialize)] pub struct RetainItem { pub content: String, pub timestamp: String, // ISO8601 UTC timestamp or literal "unset" pub context: String, pub document_id: String, pub update_mode: String, // Must always serialize as "replace" pub metadata: HashMap<String, String>, pub tags: Vec<String>, pub observation_scopes: String, // Must serialize as "combined" } #[derive(Debug, Serialize, Deserialize)] pub struct RetainResponse { pub memory_ids: Option<Vec<String>>, pub operation_id: Option<String>, pub status: Option<String>, } #[derive(Debug, Serialize, Deserialize)] pub struct OperationStatusResponse { pub operation_id: String, pub status: String, // "pending", "in_progress", "completed", "failed" pub memory_ids: Option<Vec<String>>, pub error: Option<String>, } ``` ### 3. Authenticated Hindsight Client (`src/hindsight/client.rs`) - Struct `HindsightClient`: ```rust pub struct HindsightClient { client: reqwest::Client, base_url: reqwest::Url, bank_id: String, auth_token: secrecy::SecretString, } ``` - Constructor reading authorization credentials from the configured `authorization_file` (or token file). - Capability gate verifying the deployed Hindsight authentication scheme and compatibility during startup before traffic is dispatched (`docs/technical/hindsight.md:7`). - Method `retain(&self, request: RetainRequest) -> Result<RetainResponse, HindsightError>`: - Dispatches `POST /v1/default/banks/{bank_id}/memories`. - Sets verified Authorization header, `Content-Type: application/json`. - Handles HTTP 200 OK (synchronous retain) and HTTP 202 Accepted (asynchronous retain). - Method `poll_operation(&self, operation_id: &str) -> Result<OperationStatusResponse, HindsightError>`: - Dispatches `GET /v1/default/banks/{bank_id}/operations/{operation_id}`. - Error handling: Maps HTTP 401/403 to `HindsightError::Unauthorized`, 404 to `HindsightError::NotFound`, 429/503 to `HindsightError::Unavailable(retry_after)`. ### 4. Conformance Test Suite (`tests/hindsight_client_tests.rs`) - Test deterministic hashing: - Verify that varying whitespace or field order in caller code does not affect canonical tag derivation. - Test retain wire serialization: - Serialize `RetainRequest` and assert exact JSON match with `hindsight.md` normative example. - Mock server integration test (Scenario I01): - Send retain request for `life-event:evt_demo`. - Send identical retain request for `life-event:evt_demo` a second time with modified content. - Verify that both requests sent `update_mode: "replace"` and identical `document_id`. - Assert that mock server receives update without generating duplicate memory entries. ## Acceptance Criteria - [x] `document_id` is strictly and deterministically formatted as `life-event:<event_id>`. - [x] Retain requests serialize `update_mode: "replace"`, `observation_scopes: "combined"`, and string metadata values matching OpenAPI 0.10.1 (Scenario I01). - [x] Item tag is deterministically derived as `life-item:` plus lowercase SHA-256 of the UTF-8 canonical JSON array `[collection_id, uid]` without whitespace sensitivity. - [x] Context-only records without a calendar item use the stable `life-context` tag and source event ID without fabricating a fake UID. - [x] Re-retaining the same event reuses the stable `document_id` and replace semantics without creating duplicate memories (Scenario I01). - [x] Authenticated client securely loads credentials from secret files and handles HTTP 200, 202, 401, 403, and 503 responses appropriately. - [x] Deployed authentication scheme is verified against Hindsight before retain requests are enabled, confirming instance compatibility. ## Notes - Adapter + identity only: the durable outbox table, background polling worker and retry scheduling belong to #17; wiring `verify_compatibility` into startup belongs with #17/#18. - No dependency added (ring/zeroize already present), so the deferred `Cargo.toml` authors chore stays pending. ## Blocked by - #005: Two-Phase Commit Journaling & Idempotency Fingerprinting - #010: Conditional Task & Event Creation with Prepared Intent
yuris changed title from Issue #016: Hindsight Reqwest Client & Deterministic Document Identity to Hindsight Reqwest Client & Deterministic Document Identity 2026-09-22 20:02:37 +00:00
yuris closed this issue 2026-09-22 22:58:11 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/life-service#16
No description provided.