Pagination: shared materialized read result #45

New issue

Closed

opened 2026-09-22 20:38:06 +00:00 by yuris · 0 comments

yuris commented

2026-09-22 20:38:06 +00:00

Owner

Context

Follow-up from #9 (PR #44). docs/technical/cli-api.md states: "Pages share a bounded materialized read result; expiry yields cursor_expired and requires restarting."

The #9 implementation deliberately deviates: the cursor carries only (offset, filter-hash, expiry), and each page re-queries the backend live. Ordering is deterministic so paging is stable for an unchanged backend, but a concurrent external edit can cause a skipped or repeated item across pages.

Why deferred from #9

The acceptance checklist for #9 only requires default limit 50 / clamp 200 / opaque cursor / cursor_expired. A real materialized page store means daemon-side bounded session state (keyed by cursor), LRU eviction, and restart semantics — infrastructure that belongs with the sync/reconcile work, not the read surface.

What to build

Daemon-side bounded page-snapshot store: key = cursor session id, value = materialized item list + filter fingerprint + capture instant, with a bounded entry count and TTL.
Page requests within a live session read the snapshot instead of re-querying; the snapshot's observation time is disclosed as as_of.
Expiry or eviction yields cursor_expired (HTTP 422 / exit 2), requiring the client to restart.
Restart invalidates sessions structurally (cursors must not be silently resumed against a fresh snapshot).
Test: page 1 then mutate the backend, page 2 continues from the captured snapshot; after expiry the cursor is rejected.

Notes

Reviewed by an independent reviewer agent; verdict APPROVE with no blockers and no major defects.
Store lifecycle: constructed once in api::serve() and shared via an Axum extension — not per request. Bounds: 16 sessions per route with LRU eviction, TTL 15 minutes (= CURSOR_TTL, so a cursor can never outlive its snapshot). Worst-case residency is 32 sessions × 10,000 items ≈ 96–160 MB in the extreme theoretical case; finite and enforceable for a single-user daemon.
The headline claim is proven by request counts, not by eyeballing items: page 1 → mutate the backend → page 2 asserts the CalDAV report count does not grow, with the backend data rewritten underneath. Same for the timeline against concurrent journal writes.
as_of reports the snapshot's capture instant on every page rather than the page-time wall clock, so staleness is disclosed rather than hidden.
Expiry and eviction are byte-for-byte indistinguishable (cursor_expired, 422, exit 2) — an evicted session cannot be mistaken for a different error or silently restarted at offset 0.
Restart invalidation is structural: a random 64-bit boot epoch inside the HMAC-signed cursor, tested by two stores sharing a key where the second still rejects the first's cursor. A fresh boot key adds a second independent layer.
Existing paging contract preserved: default limit 50, clamp 200, opaque cursor, constant-time filter fingerprint comparison (a cursor replayed under a different filter is still refused), deterministic ordering.
Newly introduced bound, disclosed: /v1/timeline now refuses >10,000 instances with limit_exceeded instead of paging an unbounded result; sanctioned by the spec's bounded-materialized-result requirement and stated in the PR body. Known coverage gap: that refusal path has no test (generating 10k facts is impractical in the suite); the code mirrors the tested tasks/agenda check.
The reviewer audited all three modified existing test files hunk by hunk — signature-only updates, no weakened assertions, no relocated fixtures.
Test count: 798 passed / 0 failed.

Blocked by

None — independent of the read surface, but should land with or after the reconcile/sync machinery (#6 / #36) rather than before it.

## Context Follow-up from #9 (PR #44). `docs/technical/cli-api.md` states: *"Pages share a bounded materialized read result; expiry yields `cursor_expired` and requires restarting."* The #9 implementation deliberately deviates: the cursor carries only `(offset, filter-hash, expiry)`, and each page re-queries the backend live. Ordering is deterministic so paging is stable for an unchanged backend, but a concurrent external edit can cause a skipped or repeated item across pages. ## Why deferred from #9 The acceptance checklist for #9 only requires default limit 50 / clamp 200 / opaque cursor / `cursor_expired`. A real materialized page store means daemon-side bounded session state (keyed by cursor), LRU eviction, and restart semantics — infrastructure that belongs with the sync/reconcile work, not the read surface. ## What to build - Daemon-side bounded page-snapshot store: key = cursor session id, value = materialized item list + filter fingerprint + capture instant, with a bounded entry count and TTL. - Page requests within a live session read the snapshot instead of re-querying; the snapshot's observation time is disclosed as `as_of`. - Expiry or eviction yields `cursor_expired` (HTTP 422 / exit 2), requiring the client to restart. - Restart invalidates sessions structurally (cursors must not be silently resumed against a fresh snapshot). - Test: page 1 then mutate the backend, page 2 continues from the captured snapshot; after expiry the cursor is rejected. ## Notes - Reviewed by an independent reviewer agent; verdict APPROVE with no blockers and no major defects. - **Store lifecycle:** constructed once in `api::serve()` and shared via an Axum extension — not per request. Bounds: 16 sessions per route with LRU eviction, TTL 15 minutes (`= CURSOR_TTL`, so a cursor can never outlive its snapshot). Worst-case residency is 32 sessions × 10,000 items ≈ 96–160 MB in the extreme theoretical case; finite and enforceable for a single-user daemon. - **The headline claim is proven by request counts, not by eyeballing items:** page 1 → mutate the backend → page 2 asserts the CalDAV report count does not grow, with the backend data rewritten underneath. Same for the timeline against concurrent journal writes. - **`as_of`** reports the snapshot's capture instant on every page rather than the page-time wall clock, so staleness is disclosed rather than hidden. - **Expiry and eviction are byte-for-byte indistinguishable** (`cursor_expired`, 422, exit 2) — an evicted session cannot be mistaken for a different error or silently restarted at offset 0. - **Restart invalidation is structural:** a random 64-bit boot epoch inside the HMAC-signed cursor, tested by two stores sharing a key where the second still rejects the first's cursor. A fresh boot key adds a second independent layer. - **Existing paging contract preserved:** default limit 50, clamp 200, opaque cursor, constant-time filter fingerprint comparison (a cursor replayed under a different filter is still refused), deterministic ordering. - **Newly introduced bound, disclosed:** `/v1/timeline` now refuses >10,000 instances with `limit_exceeded` instead of paging an unbounded result; sanctioned by the spec's bounded-materialized-result requirement and stated in the PR body. **Known coverage gap:** that refusal path has no test (generating 10k facts is impractical in the suite); the code mirrors the tested tasks/agenda check. - The reviewer audited all three modified existing test files hunk by hunk — signature-only updates, no weakened assertions, no relocated fixtures. - Test count: 798 passed / 0 failed. ## Blocked by None — independent of the read surface, but should land with or after the reconcile/sync machinery (#6 / #36) rather than before it.

yuris referenced this issue

2026-09-22 20:45:43 +00:00

Agenda & Tasks List CLI/API with Completeness Metrics #9

yuris referenced this issue

2026-09-22 22:37:51 +00:00

Classify custom non-IANA TZID as calendar_unsupported, not dependency_unavailable #53

yuris changed title from ~~Pagination: shared materialized read result (deferred from #9)~~ to Pagination: shared materialized read result

2026-09-23 05:14:44 +00:00

yuris referenced this issue from a pull request that will close it,

2026-09-23 05:47:52 +00:00

feat(api): bounded materialized page snapshots for paged reads #83

yuris referenced this issue