yuris/spacetimedb-defold
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 9 Packages Wiki Activity Actions

Time-slice the frame that applies a first subscription #67

New issue
Closed
opened 2026-08-20 14:41:54 +00:00 by yuris · 1 comment
yuris commented 2026-08-20 14:41:54 +00:00
Owner
Copy link

Implementation follow-up to #30, whose measurement (#66) established that this is needed. Filed separately so the design is reviewed against the data rather than bundled with it.

What the measurement says

Applying a first subscription is linear in rows, ~2.7us/row native, all in one frame:

  • 60fps budget (16.7ms) is exceeded at ~6,400 rows native. Measured: 6,000 -> 15.9/16.2ms, 6,400 -> 16.8/17.8ms.
  • 100ms, a hitch a player notices, at ~32,000 rows native and ~9,400 rows on HTML5.
  • On HTML5 an idle wasm frame already costs 18ms to requestAnimationFrame, so 16.7ms of actual work arrives at ~2,600 rows - roughly 2.5x worse than native.
  • The typed path (protocol.decode_typed), which is the one the README recommends, costs 12-17% MORE, so real thresholds are ~12% lower again.

6,400 rows is a chat backlog, a leaderboard, or an inventory across a player base. This is a cliff real games hit.

Note what will NOT fix it: json.decode is only 40-49% of the frame natively and 20-25% in wasm, so replacing it with a native BSATN decoder has an Amdahl ceiling of 1.8x / 1.3x - it moves the native 100ms threshold from ~32k to ~58k rows and does not remove the hitch. Recorded on #9.

The shape #30 specifies

  • A per-frame deadline shared across every callback OnUpdate delivers in that frame, NOT per callback - OnUpdate drains the whole queue in one pass, so a per-callback budget would let one frame spend N budgets.
  • A coroutine per in-flight batch.
  • Later frames queued behind an in-progress batch, so ordering holds absolutely.
  • Per-table callbacks firing only on batch completion, so no consumer ever sees a half-populated table.

Acceptance

  • A 10k-row subscription no longer produces a single frame over budget. PHASE6 of the conformance harness (added by #66) is the instrument.
  • A TransactionUpdate arriving mid-batch is applied AFTER it, never interleaved. This is the ordering property that makes the feature safe rather than merely smooth.
  • The existing PHASE1-PHASE6 keep passing, and stdb/ stays frozen in shape - this is internal to the apply path.
Implementation follow-up to #30, whose measurement (#66) established that this is needed. Filed separately so the design is reviewed against the data rather than bundled with it. ## What the measurement says Applying a first subscription is linear in rows, ~2.7us/row native, all in one frame: - 60fps budget (16.7ms) is exceeded at **~6,400 rows** native. Measured: 6,000 -> 15.9/16.2ms, 6,400 -> 16.8/17.8ms. - 100ms, a hitch a player notices, at **~32,000 rows** native and **~9,400 rows** on HTML5. - On HTML5 an idle wasm frame already costs 18ms to requestAnimationFrame, so 16.7ms of actual work arrives at **~2,600 rows** - roughly 2.5x worse than native. - The typed path (protocol.decode_typed), which is the one the README recommends, costs 12-17% MORE, so real thresholds are ~12% lower again. 6,400 rows is a chat backlog, a leaderboard, or an inventory across a player base. This is a cliff real games hit. Note what will NOT fix it: json.decode is only 40-49% of the frame natively and 20-25% in wasm, so replacing it with a native BSATN decoder has an Amdahl ceiling of 1.8x / 1.3x - it moves the native 100ms threshold from ~32k to ~58k rows and does not remove the hitch. Recorded on #9. ## The shape #30 specifies - A per-**frame** deadline shared across every callback OnUpdate delivers in that frame, NOT per callback - OnUpdate drains the whole queue in one pass, so a per-callback budget would let one frame spend N budgets. - A coroutine per in-flight batch. - Later frames queued **behind** an in-progress batch, so ordering holds absolutely. - Per-table callbacks firing only on batch completion, so no consumer ever sees a half-populated table. ## Acceptance - A 10k-row subscription no longer produces a single frame over budget. PHASE6 of the conformance harness (added by #66) is the instrument. - A TransactionUpdate arriving mid-batch is applied AFTER it, never interleaved. This is the ordering property that makes the feature safe rather than merely smooth. - The existing PHASE1-PHASE6 keep passing, and stdb/ stays frozen in shape - this is internal to the apply path.
yuris closed this issue 2026-08-20 18:45:24 +00:00
yuris commented 2026-08-20 18:48:50 +00:00
Author
Owner
Copy link

Closed by #76.

frame_budget (seconds, opt-in, nil by default) spreads the row decoding of an apply across engine frames. Measured on main @ 52dd72d, 25000 rows, 8ms budget, medians over six samples:

unsliced  BULK SUMMARY  rows=25000 runs=3 frame med=54.1ms / 55.0ms
sliced    SLICED SWEEP  rows=25000 runs=3 budget=8.0ms worst decode med=8.5ms store med=10.6ms span med=51.0ms frames med=4

The worst row-decoding frame goes from ~55ms to 8.5ms, over 4 frames, with the same wall-clock total — redistribution, not overhead.

What this does not do, stated plainly

Two parts of an apply are O(rows) and cannot yield, and after this they are the worst frames:

  1. decode_server's single json.decode of the whole frame text, before any row loop exists to yield from — 12.8–23.3ms at 25000 rows.
  2. The store mutation, atomic by design.

So the honest headline is ~55ms → ~20ms at 25000 rows, not "down to the budget". The acceptance sketched in this issue ("no frame over budget + idle + 5ms") is not achievable without a streaming parser; PHASE6 asserts the ceiling on the row-decoding frames and reports first frame= and store frame= beside them rather than widening slack until the numbers fit.

On wasm the mechanism works but the tier dominates: a ~17ms rAF floor means an 8ms budget is below the frame quantum, and socket delivery sits in the first frame. At 10000 rows: decode frames 17ms (the floor), first frame ~75ms.

Two bugs worth recording, both found by measuring rather than reading

The invariant this design rests on was false, and the tests passed anyway. apply_database_update interleaved decode and store per updates entry, so a yield in the second table's decode suspended with the first table's rows already written and unannounced. A two-table subscription of 600 rows each sat at alpha=600, beta=0, events=0 for two whole engine frames — iter and find answering with a frame half applied — and a socket drop there stranded 600 rows in the store for the whole outage. Every test fixture had been single-table, which is exactly the case where the invariant does hold. apply_database_update is now two passes (decode everything, then store everything), which also strengthens the pre-existing malformed-row guarantee from per-updates-entry to per-frame. Verified equivalent to the old one-pass version by differential oracle: 1800 randomised frames over a four-table schema, comparing counts and the full store snapshot, zero divergences.

The hook closure was cached across pumps. decode_rows hoisted local hook = M.slice_hook, so after the first yield it kept calling the first pump's closure with a deadline that had expired frames ago — yielding at every 256-row slice point. 25000 rows took 97 frames at 1.5ms each instead of 4 at 8.5ms; 97 ≈ 25000/256 is the signature. M.slice_hook is now re-read per slice point.

Also fixed: close() from inside a per-table callback raised coroutine.status(nil) out of the timer callback; PHASE6 could report PASS having sliced nothing when the budget exceeded the whole decode; and html5-conformance.mjs never forwarded STDB_BULK_BUDGET, so wasm runs measured the unsliced path while claiming otherwise.

Verification on main @ 52dd72d

8 passed, 0 failed; stylua --check clean; shellcheck exit 0. Native conformance passes unsliced (all six phases) and sliced at 10000/25000; HTML5 passes both ways. dev/test/test_slicing.lua is mutation-tested — FIFO→LIFO, dropping the per-invocation deadline, dropping the hook clearing, hoisting the hook, and reverting the two-pass split each fail a distinct named assertion.

Closed by #76. `frame_budget` (seconds, opt-in, nil by default) spreads the row decoding of an apply across engine frames. Measured on `main` @ 52dd72d, 25000 rows, 8ms budget, medians over six samples: ``` unsliced BULK SUMMARY rows=25000 runs=3 frame med=54.1ms / 55.0ms sliced SLICED SWEEP rows=25000 runs=3 budget=8.0ms worst decode med=8.5ms store med=10.6ms span med=51.0ms frames med=4 ``` **The worst row-decoding frame goes from ~55ms to 8.5ms**, over 4 frames, with the same wall-clock total — redistribution, not overhead. ## What this does not do, stated plainly Two parts of an apply are O(rows) and cannot yield, and after this they are the worst frames: 1. `decode_server`'s single `json.decode` of the whole frame text, before any row loop exists to yield from — 12.8–23.3ms at 25000 rows. 2. The store mutation, atomic by design. So the honest headline is **~55ms → ~20ms at 25000 rows**, not "down to the budget". The acceptance sketched in this issue ("no frame over budget + idle + 5ms") is not achievable without a streaming parser; PHASE6 asserts the ceiling on the row-decoding frames and reports `first frame=` and `store frame=` beside them rather than widening slack until the numbers fit. On wasm the mechanism works but the tier dominates: a ~17ms rAF floor means an 8ms budget is below the frame quantum, and socket delivery sits in the first frame. At 10000 rows: decode frames 17ms (the floor), first frame ~75ms. ## Two bugs worth recording, both found by measuring rather than reading **The invariant this design rests on was false, and the tests passed anyway.** `apply_database_update` interleaved decode and store per `updates` entry, so a yield in the second table's decode suspended with the first table's rows already written and unannounced. A two-table subscription of 600 rows each sat at `alpha=600, beta=0, events=0` for two whole engine frames — `iter` and `find` answering with a frame half applied — and a socket drop there stranded 600 rows in the store for the whole outage. Every test fixture had been single-table, which is exactly the case where the invariant does hold. `apply_database_update` is now two passes (decode everything, then store everything), which also strengthens the pre-existing malformed-row guarantee from per-`updates`-entry to per-frame. Verified equivalent to the old one-pass version by differential oracle: 1800 randomised frames over a four-table schema, comparing counts and the full store snapshot, zero divergences. **The hook closure was cached across pumps.** `decode_rows` hoisted `local hook = M.slice_hook`, so after the first yield it kept calling the first pump's closure with a deadline that had expired frames ago — yielding at every 256-row slice point. 25000 rows took 97 frames at 1.5ms each instead of 4 at 8.5ms; `97 ≈ 25000/256` is the signature. `M.slice_hook` is now re-read per slice point. Also fixed: `close()` from inside a per-table callback raised `coroutine.status(nil)` out of the timer callback; PHASE6 could report PASS having sliced nothing when the budget exceeded the whole decode; and `html5-conformance.mjs` never forwarded `STDB_BULK_BUDGET`, so wasm runs measured the unsliced path while claiming otherwise. ## Verification on `main` @ 52dd72d `8 passed, 0 failed`; `stylua --check` clean; `shellcheck` exit 0. Native conformance passes unsliced (all six phases) and sliced at 10000/25000; HTML5 passes both ways. `dev/test/test_slicing.lua` is mutation-tested — FIFO→LIFO, dropping the per-invocation deadline, dropping the hook clearing, hoisting the hook, and reverting the two-pass split each fail a distinct named assertion.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
release/v4.0.1
fix/hold-pre-open-frames
v4.0.2
v4.0.1
v4.0.0
v3.1.0
v3.0.0
v2.0.0
v1.1.0
v1.0.0
v0.1.0
Labels
Clear labels   
area:defold
Defold engine integration

  
area:docs
Documentation and examples

  
area:infra
Flake, CI, secretspec

  
area:protocol
Wire protocol conformance

  
area:testing
Test harness and fixtures

  
kind:bug
Something broken

  
kind:chore
Maintenance

  
phase:breadth
UnsubscribeMulti, procedures, BSATN

  
phase:core
Core SDK layers (protocol/cache/connection)

  
phase:hardening
Reconnect, token persistence, errors

  
phase:slice
Vertical slice: first playable end-to-end

No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/spacetimedb-defold#67
No description provided.