Implement POST /v1/sync: incremental and rebaseline collection sync #88

New issue

Closed

opened 2026-09-23 06:23:14 +00:00 by yuris · 0 comments

yuris commented

2026-09-23 06:23:14 +00:00

Owner

Context

POST /v1/sync was documented in docs/technical/cli-api.md with an admin scope but never implemented — the router had no handler, so it answered 501, and the CLI verb exited 5. Issue #85 closed that gap by descoping: the route row, the orphaned Scope::Admin entry, the CLI verb and the skill's not-implemented row were all removed so the contract and the binary agree.

This issue is the deferred half: the actual feature. It exists so the descope does not become a silent deletion of planned capability — the specification text for sync is still in docs/technical/, and the semantics were never trivial.

What the specification already says

docs/technical/calendar.md:39 — initial import records observed baselines, not historical creation events. An advertised sync-collection may supply incremental changes; otherwise enumerate full href/ETag sets. A sync REPORT followed by GET is not atomic: compare ETags, repeat bounded reconciliation on mismatch, and surface concurrent_change uncertainty after the retry bound.
docs/technical/calendar.md:51 — the conformance gate to satisfy: ETag races, sync cursor invalidation, per-resource errors, concurrent REPORT/GET changes.
docs/technical/timeline-consistency.md:89 — a sync_state table is already specified in the schema; use it rather than inventing a parallel cursor store.
docs/technical/timeline-consistency.md:119 — external observation dedupe keys are collection + sync generation + observation identity; a later identical transition must not be suppressed because an earlier one occurred.
docs/technical/timeline-consistency.md:154 — commit observation events, projection changes and cursor advancement together; for paged sync, commit a cursor only when it represents all durably processed changes; interrupted pages remain resumable or repeat idempotently.
docs/technical/cli-api.md (response table) — the response shape: operation_id, observed counts, per-collection coverage/cursor state, no invented historical events; and sync is a state-changing route, so it requires Idempotency-Key.
docs/technical/calendar.md:57 — RFC 6578 (WebDAV sync) is the protocol reference for sync-collection.

What to build

Re-add POST /v1/sync (admin scope) and the life sync --input - verb, with mode = incremental | rebaseline validated strictly (unknown values refused, not defaulted), and Idempotency-Key required like every other state-changing route.
Incremental: use sync-collection only when the backend advertises it and the conformance gate passes; otherwise fall back to full href/ETag enumeration with bounded GETs. Persist the cursor per collection in sync_state.
Rebaseline: re-enumerate and reconcile against the stored projection, surfacing concurrent_change uncertainty after the retry bound rather than silently overwriting.
Durability: cursor advancement commits in the same transaction as the observation events and projection changes. A sync interrupted between pages is either resumable from the committed cursor or idempotently repeatable — prove it by killing the run between pages.
Coverage: report per-collection coverage and cursor state; a collection that failed is named in the response and in a warning, never silently omitted from the counts.
No invention: pre-existing resources produce observed baselines only. A test must prove that a resource that existed before the first sync produces no fabricated creation event.

Acceptance criteria

life sync --input - --json transmits, returns one envelope, and is admin-scoped (a read or write token is refused).
mode is validated; an unknown mode is refused with invalid_input rather than defaulting.
Incremental sync uses sync-collection only when advertised; a backend that does not advertise it falls back to enumeration, proven by a test with both fixtures.
Cursor advancement is committed with the observation events and projection changes in one transaction; an interruption between pages is resumable or idempotently repeatable, proven by a test.
A per-collection failure is named in coverage/warnings rather than dropped.
No historical creation event is invented for a pre-existing resource; only observed baselines are recorded.
The conformance gate in calendar.md:51 is satisfied, or the unsupported path is refused with calendar_unsupported rather than pretending to have synced.
Spec, CLI and skill agree again — the route row, the scope entry, the verb and the skill's command reference are restored in one commit.

Blocked by

None technically. Related: #85 (the descope this supersedes), #45 (paged reads share the same snapshot machinery), #6 (recovery scanner), #36 (backup/DR).

## Context POST /v1/sync was documented in docs/technical/cli-api.md with an admin scope but never implemented — the router had no handler, so it answered 501, and the CLI verb exited 5. Issue #85 closed that gap by **descoping**: the route row, the orphaned Scope::Admin entry, the CLI verb and the skill's not-implemented row were all removed so the contract and the binary agree. This issue is the deferred half: the actual feature. It exists so the descope does not become a silent deletion of planned capability — the specification text for sync is still in docs/technical/, and the semantics were never trivial. ## What the specification already says - docs/technical/calendar.md:39 — initial import records **observed baselines**, not historical creation events. An advertised sync-collection may supply incremental changes; otherwise enumerate full href/ETag sets. A sync REPORT followed by GET is **not atomic**: compare ETags, repeat bounded reconciliation on mismatch, and surface concurrent_change uncertainty after the retry bound. - docs/technical/calendar.md:51 — the conformance gate to satisfy: ETag races, sync cursor invalidation, per-resource errors, concurrent REPORT/GET changes. - docs/technical/timeline-consistency.md:89 — a sync_state table is already specified in the schema; use it rather than inventing a parallel cursor store. - docs/technical/timeline-consistency.md:119 — external observation dedupe keys are collection + sync generation + observation identity; a later identical transition must not be suppressed because an earlier one occurred. - docs/technical/timeline-consistency.md:154 — commit observation events, projection changes and cursor advancement **together**; for paged sync, commit a cursor only when it represents all durably processed changes; interrupted pages remain resumable or repeat idempotently. - docs/technical/cli-api.md (response table) — the response shape: operation_id, observed counts, per-collection coverage/cursor state, **no invented historical events**; and sync is a state-changing route, so it requires Idempotency-Key. - docs/technical/calendar.md:57 — RFC 6578 (WebDAV sync) is the protocol reference for sync-collection. ## What to build - Re-add POST /v1/sync (admin scope) and the life sync --input - verb, with mode = incremental | rebaseline validated strictly (unknown values refused, not defaulted), and Idempotency-Key required like every other state-changing route. - **Incremental:** use sync-collection only when the backend advertises it and the conformance gate passes; otherwise fall back to full href/ETag enumeration with bounded GETs. Persist the cursor per collection in sync_state. - **Rebaseline:** re-enumerate and reconcile against the stored projection, surfacing concurrent_change uncertainty after the retry bound rather than silently overwriting. - **Durability:** cursor advancement commits in the same transaction as the observation events and projection changes. A sync interrupted between pages is either resumable from the committed cursor or idempotently repeatable — prove it by killing the run between pages. - **Coverage:** report per-collection coverage and cursor state; a collection that failed is named in the response and in a warning, never silently omitted from the counts. - **No invention:** pre-existing resources produce observed baselines only. A test must prove that a resource that existed before the first sync produces no fabricated creation event. ## Acceptance criteria - [ ] life sync --input - --json transmits, returns one envelope, and is admin-scoped (a read or write token is refused). - [ ] mode is validated; an unknown mode is refused with invalid_input rather than defaulting. - [ ] Incremental sync uses sync-collection only when advertised; a backend that does not advertise it falls back to enumeration, proven by a test with both fixtures. - [ ] Cursor advancement is committed with the observation events and projection changes in one transaction; an interruption between pages is resumable or idempotently repeatable, proven by a test. - [ ] A per-collection failure is named in coverage/warnings rather than dropped. - [ ] No historical creation event is invented for a pre-existing resource; only observed baselines are recorded. - [ ] The conformance gate in calendar.md:51 is satisfied, or the unsupported path is refused with calendar_unsupported rather than pretending to have synced. - [ ] Spec, CLI and skill agree again — the route row, the scope entry, the verb and the skill's command reference are restored in one commit. ## Blocked by None technically. Related: #85 (the descope this supersedes), #45 (paged reads share the same snapshot machinery), #6 (recovery scanner), #36 (backup/DR).

yuris changed title from ~~Implement POST /v1/sync: incremental and rebaseline collection sync (deferred by #85)~~ to Implement POST /v1/sync: incremental and rebaseline collection sync

2026-09-23 06:24:12 +00:00

yuris changed title from ~~Implement POST /v1/sync: incremental and rebaseline collection sync~~ to Implement POST /v1/sync: incremental and rebaseline collection sync

2026-09-23 06:25:41 +00:00

yuris referenced this issue

2026-09-23 06:32:09 +00:00

life sync / POST /v1/sync is documented with an admin scope but has no handler #85

yuris referenced this issue from a pull request that will close it,

2026-09-23 11:17:30 +00:00

feat(api): implement sync collection reconciliation #93

yuris closed this issue

2026-09-23 11:47:19 +00:00