yuris/spacetimedb-defold
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 9 Packages Wiki Activity Actions

Measure the frame that applies a first subscription (#30) #66

Merged
yuris merged 2 commits from feat/30-frame-budget into main 2026-08-20 16:00:15 +00:00
Conversation 1 Commits 2 Files changed 5 +665 -11
yuris commented 2026-08-20 14:20:46 +00:00
Owner
Copy link

Measures #30 rather than closing it. Phase 4 of the v1.1.0 milestone (#59).

The issue is explicit: "Measure before choosing a number." This PR is only the
measurement and the instrumentation that produced it. No time-slicing code —
that decision now has data behind it and should be reviewed against the numbers
rather than bundled with them.

The answer: time-slicing is justified

Native, 60fps budget (16.7ms): ~6,400 rows. Measured, not interpolated —
6,000 rows medians 15.9/16.2ms (under), 6,400 gives 16.8/17.8ms (over), 6,800
gives 18.5/19.4ms.

Native, noticeable hitch (100ms): ~32,000 rows. 32,000 medians 100.5/97.8ms,
33,000 gives 102.1/104.1ms.

HTML5, 100ms: ~9,400 rows. 9,000 → 95ms, 10,000 → 110ms.

6,400 rows is not an exotic number. A chat backlog, a leaderboard, an inventory
across a player base, any "select the whole table" subscription on a modest game
reaches it, so this is a cliff real games hit rather than a theoretical one.

Native sweep

Defold headless, x86_64-linux, hand-written schema, 7-column rows (~83 B/row on
the wire). 6 runs per point — 3 in each of conformance.sh's two engine
processes. Medians.

rows frame ms json.decode ms construction ms frame min–max
100 0.5 0.1 0.35 0.4–0.6
1,000 2.7 1.25 1.5 2.5–3.1
5,000 13.0 6.05 7.25 12.5–13.7
10,000 26.6 12.8 14.05 25.0–31.3
25,000 74.7 32.6 41.75 72.7–78.5
50,000 155.2 66.6 86.95 143.6–160.9
100,000 327.5 147.5 182.4 310.7–349.0

Cost is linear in rows, ~2.7µs/row native. No super-linear blow-up, so the
problem is purely that the whole set lands in one frame.

HTML5 sweep

Same client as a wasm bundle in headless Chromium. The wasm engine is
vsync-locked to requestAnimationFrame, so an idle frame already measures 18ms
— that is a floor, not work, and the "excess" column is the honest number.

rows frame ms excess over 18ms floor decode share
100 18.0 0 (floor) 2–4%
1,000 24.0 6 14–15%
2,500 34.0 16 19–20%
5,000 56.0 38 20–22%
10,000 110.0 92 23–25%

Because of the rAF floor, "16.7ms of work" is the meaningful 60fps threshold on
web, and it arrives at ~2,600 rows — roughly 2.5× worse than native.

The typed path, which is the one the README recommends, is worse

Same rows through protocol.decode_typed instead of decode_row:

rows typed ms hand-written ms
1,000 3.1 2.7
10,000 30.2 26.6
50,000 176.6 155.2

Typed decoding costs 12–17% more, so the real-world thresholds are ~12%
lower than the headline numbers above. Worth stating plainly, since the README
steers people to the typed path.

The json.decode split — and what it means for #9

Native: 40–49% of the frame, mean ~45%. WASM: 20–25%, mean ~23%.

This is the data point #9's first checkbox asks for, and it argues against #9:

  • Amdahl ceiling for replacing json.decode with a native/C++ BSATN decoder is
    1/(1-0.45) = 1.8× on native, and only 1.3× in wasm.
  • 1.8× moves the native 100ms threshold from ~32k rows to ~58k. It does not
    remove the hitch, it postpones it.
  • The other ~55% native / ~77% wasm is protocol.quote_big_ints' gsub, building
    the row tables, pk indexing, event-list construction and tables.dispatch —
    none of which a decoder swap touches.

So a BSATN decoder is not the fix for #30, and on the web tier — where the
budget is tightest — it is nearly irrelevant. Posted to #9 separately.

Method, and why each choice

  • socket.gettime (LuaSocket gettimeofday), a wall clock. Rejected
    os.clock, which is processor time and under-reports any wait, so it is not
    what a player feels; rejected chronos, which is a third-party native
    extension and absent here. socket.gettime resolves in both tiers —
    LuaSocket is compiled into the headless engine and the wasm build — at 1ms
    resolution under emscripten.
  • Total is the update-to-update wall interval containing the apply,
    attributed by timestamp rather than by counting frames, so the number does not
    depend on whether the engine dispatches extension callbacks before or after
    script update.
  • The decode split comes from wrapping the table passed as stdb.connect's
    opts.json. protocol.init receives it and every decode goes through it, so
    the split needed zero changes to frozen stdb/.
  • The phase asserts it actually measured the whole set: decode_calls must
    equal rows + 1 (one envelope plus one per row) and cache.count must equal
    rows, or PHASE6 fails. Without that, a truncated apply would report a
    flatteringly small number.

What changed

dev/server/index.js gains a new table and a bulk-insert reducer with the row
count as a parameter, so the sweep is not hardcoded. message is untouched —
it is byte-frozen and the fixtures assert its exact shape. The schema fixture was
regenerated with gen-schema.sh, not hand-edited. smoke.script gains PHASE6
following the existing phase pattern; PHASE1–PHASE5 still pass unchanged.

Recommendation

Implement time-slicing as #30 specifies — a per-frame deadline shared across
every callback OnUpdate delivers in that frame (not per callback, since
OnUpdate drains the whole queue in one pass), a coroutine per in-flight batch,
later frames queued behind an in-progress batch so ordering holds absolutely,
and per-table callbacks firing only on batch completion so no consumer ever sees
a half-populated table.

Tracked separately so the design is reviewed against this data rather than
alongside it.

Verification

lua dev/test/runner.lua       6 passed, 0 failed
stylua --check stdb dev/test  exit 0, silent
conformance.sh                PASS, PHASE1-6

Independently reproduced by the maintainer on a rebase onto current main:
BULK rows=10000 frame=26.4ms decode=10.6ms rest=15.8ms decode_share=40%,
against the sweep's 26.6ms / 40–49%.

Machine: AMD Ryzen 7 8845HS (8C/16T Zen 4 mobile), NixOS, kernel 6.18.41,
x86_64, against live pinned SpacetimeDB 2.8.2.

Measures #30 rather than closing it. Phase 4 of the v1.1.0 milestone (#59). The issue is explicit: *"Measure before choosing a number."* This PR is only the measurement and the instrumentation that produced it. **No time-slicing code** — that decision now has data behind it and should be reviewed against the numbers rather than bundled with them. ## The answer: time-slicing is justified **Native, 60fps budget (16.7ms): ~6,400 rows.** Measured, not interpolated — 6,000 rows medians 15.9/16.2ms (under), 6,400 gives 16.8/17.8ms (over), 6,800 gives 18.5/19.4ms. **Native, noticeable hitch (100ms): ~32,000 rows.** 32,000 medians 100.5/97.8ms, 33,000 gives 102.1/104.1ms. **HTML5, 100ms: ~9,400 rows.** 9,000 → 95ms, 10,000 → 110ms. 6,400 rows is not an exotic number. A chat backlog, a leaderboard, an inventory across a player base, any "select the whole table" subscription on a modest game reaches it, so this is a cliff real games hit rather than a theoretical one. ## Native sweep Defold headless, x86_64-linux, hand-written schema, 7-column rows (~83 B/row on the wire). 6 runs per point — 3 in each of `conformance.sh`'s two engine processes. Medians. | rows | frame ms | `json.decode` ms | construction ms | frame min–max | |---|---|---|---|---| | 100 | 0.5 | 0.1 | 0.35 | 0.4–0.6 | | 1,000 | 2.7 | 1.25 | 1.5 | 2.5–3.1 | | 5,000 | 13.0 | 6.05 | 7.25 | 12.5–13.7 | | 10,000 | 26.6 | 12.8 | 14.05 | 25.0–31.3 | | 25,000 | 74.7 | 32.6 | 41.75 | 72.7–78.5 | | 50,000 | 155.2 | 66.6 | 86.95 | 143.6–160.9 | | 100,000 | 327.5 | 147.5 | 182.4 | 310.7–349.0 | Cost is linear in rows, ~2.7µs/row native. No super-linear blow-up, so the problem is purely that the whole set lands in one frame. ## HTML5 sweep Same client as a wasm bundle in headless Chromium. **The wasm engine is vsync-locked to `requestAnimationFrame`, so an idle frame already measures 18ms** — that is a floor, not work, and the "excess" column is the honest number. | rows | frame ms | excess over 18ms floor | decode share | |---|---|---|---| | 100 | 18.0 | 0 (floor) | 2–4% | | 1,000 | 24.0 | 6 | 14–15% | | 2,500 | 34.0 | 16 | 19–20% | | 5,000 | 56.0 | 38 | 20–22% | | 10,000 | 110.0 | 92 | 23–25% | Because of the rAF floor, "16.7ms of work" is the meaningful 60fps threshold on web, and it arrives at **~2,600 rows** — roughly 2.5× worse than native. ## The typed path, which is the one the README recommends, is worse Same rows through `protocol.decode_typed` instead of `decode_row`: | rows | typed ms | hand-written ms | |---|---|---| | 1,000 | 3.1 | 2.7 | | 10,000 | 30.2 | 26.6 | | 50,000 | 176.6 | 155.2 | Typed decoding costs **12–17% more**, so the real-world thresholds are ~12% lower than the headline numbers above. Worth stating plainly, since the README steers people to the typed path. ## The `json.decode` split — and what it means for #9 **Native: 40–49% of the frame, mean ~45%. WASM: 20–25%, mean ~23%.** This is the data point #9's first checkbox asks for, and it argues *against* #9: - Amdahl ceiling for replacing `json.decode` with a native/C++ BSATN decoder is `1/(1-0.45)` = **1.8× on native**, and only **1.3× in wasm**. - 1.8× moves the native 100ms threshold from ~32k rows to ~58k. It does not remove the hitch, it postpones it. - The other ~55% native / ~77% wasm is `protocol.quote_big_ints`' gsub, building the row tables, pk indexing, event-list construction and `tables.dispatch` — none of which a decoder swap touches. **So a BSATN decoder is not the fix for #30**, and on the web tier — where the budget is tightest — it is nearly irrelevant. Posted to #9 separately. ## Method, and why each choice - **`socket.gettime`** (LuaSocket `gettimeofday`), a *wall* clock. Rejected `os.clock`, which is processor time and under-reports any wait, so it is not what a player feels; rejected `chronos`, which is a third-party native extension and absent here. `socket.gettime` resolves in **both** tiers — LuaSocket is compiled into the headless engine and the wasm build — at 1ms resolution under emscripten. - **Total** is the update-to-update wall interval *containing* the apply, attributed by timestamp rather than by counting frames, so the number does not depend on whether the engine dispatches extension callbacks before or after script update. - **The decode split** comes from wrapping the table passed as `stdb.connect`'s `opts.json`. `protocol.init` receives it and every decode goes through it, so the split needed **zero changes to frozen `stdb/`**. - **The phase asserts it actually measured the whole set**: `decode_calls` must equal `rows + 1` (one envelope plus one per row) and `cache.count` must equal `rows`, or PHASE6 fails. Without that, a truncated apply would report a flatteringly small number. ## What changed `dev/server/index.js` gains a new table and a bulk-insert reducer with the row count as a parameter, so the sweep is not hardcoded. **`message` is untouched** — it is byte-frozen and the fixtures assert its exact shape. The schema fixture was regenerated with `gen-schema.sh`, not hand-edited. `smoke.script` gains PHASE6 following the existing phase pattern; PHASE1–PHASE5 still pass unchanged. ## Recommendation Implement time-slicing as #30 specifies — a per-**frame** deadline shared across every callback `OnUpdate` delivers in that frame (not per callback, since `OnUpdate` drains the whole queue in one pass), a coroutine per in-flight batch, later frames queued **behind** an in-progress batch so ordering holds absolutely, and per-table callbacks firing only on batch completion so no consumer ever sees a half-populated table. Tracked separately so the design is reviewed against this data rather than alongside it. ## Verification ``` lua dev/test/runner.lua 6 passed, 0 failed stylua --check stdb dev/test exit 0, silent conformance.sh PASS, PHASE1-6 ``` Independently reproduced by the maintainer on a rebase onto current `main`: `BULK rows=10000 frame=26.4ms decode=10.6ms rest=15.8ms decode_share=40%`, against the sweep's 26.6ms / 40–49%. Machine: AMD Ryzen 7 8845HS (8C/16T Zen 4 mobile), NixOS, kernel 6.18.41, x86_64, against live pinned SpacetimeDB 2.8.2.
Measure the frame that applies a first subscription (#30)
All checks were successful
ci / ci (pull_request) Successful in 1m33s
Details
f26e582418 
#30 says "measure before choosing a number", so this adds the measurement and
nothing else. No time-slicing here: the design should be reviewed against the
data rather than the other way round.

A new `bulk` table, seven columns wide, plus `fill_bulk(count)` and
`clear_bulk()`. Seven columns rather than one because row WIDTH drives both
halves of the cost - bytes to parse and fields to construct - so a one-column
table would understate the per-row price and flatter the answer. `count` is a
reducer parameter rather than a constant because the deliverable is a curve; a
hardcoded 10000 would have decided the question before measuring it. `message`
is byte-frozen and could not have carried this at any price.

smoke.script grows PHASE6, which sweeps row counts and reports the wall time of
the frame that applies each subscription, split into json.decode and everything
else. The split needs no change to the frozen stdb/ modules: stdb.connect hands
opts.json to protocol.init, and every decode the SDK performs goes through it -
one for the frame envelope plus one per row - so wrapping the table measures
json.decode exactly. The frame itself is timed as the update-to-update interval
CONTAINING the apply, attributed by timestamp rather than by counting frames, so
it does not depend on whether the engine dispatches extension callbacks before or
after script update. socket.gettime, a wall clock: os.clock is processor time and
would under-report anything the frame spent waiting, which is not what a player
feels. LuaSocket turns out to be present in both the headless engine and the
wasm build.

html5-conformance.mjs forwards the phase-6 knobs into the page's query string,
the way STDB_URL already travels, because a browser tab has no environment - and
that tier is the slow one, so it is the tier whose numbers decide this.

WHAT IT SAYS, six runs per point on an AMD Ryzen 7 8845HS. Cost is linear in
rows, ~2.7us/row native and ~11us/row in wasm, drifting up ~25% by 100k. The
applying frame crosses 16.7ms at ~6400 rows native and 100ms at ~32000; in wasm
100ms arrives at ~9400 rows and an idle frame is already 18ms. Those are row
counts real games hit, so the data supports time-slicing.

json.decode is 45% of the frame natively and only 23% in wasm, which also
answers #9's first checkbox: replacing it with a native BSATN decoder is capped
at 1.8x by Amdahl, moving the 100ms threshold from ~32k rows to ~58k. It does
not remove the hitch. The other half is big-int quoting, row-table construction,
pk indexing and callback dispatch, none of which a decoder swap touches.
Measure big-int quoting as its own bucket, and fix the wasm attribution (#30)
All checks were successful
ci / ci (pull_request) Successful in 20s
Details
91054417f9 
The first version of PHASE6 wrapped only json.decode, so every other cost in
the applying frame landed in one remainder the PR then described as untouchable.
It is not. protocol.quote_big_ints exists ONLY because JSON cannot carry a u128
exactly, it is unguarded, and it scans the whole frame text once and then every
row again -- so on the bulk table, which has no 128-bit column, all rows+1 scans
rewrite nothing and it still costs about what the parse does: 12.6ms of a 31.0ms
native frame at 10k rows, against json.decode's 14.8ms.

That is the number that mattered. Billing quoting to the remainder understated
the ceiling for replacing the JSON codec: json.decode's share alone implies 1.8x
natively, while json.decode plus quoting is ~87% of the frame and implies ~8x.
The figure already published to #9 was the 1.8x one.

quote_big_ints is a file-local and protocol.init takes only `json`, so there is
no seam to inject. Its whole body is one gsub against a pattern that appears
nowhere else in stdb/, and stdb/ localises no string function, so the harness
wraps the string library and keys on that exact pattern. stdb/ is untouched.
The typed path quotes through quote_wide_ints instead, which is guarded; those
calls are counted rather than timed, because three clock reads per row would
cost more than the calls they measure and would load one schema path with
instrumentation the other does not carry.

The frame now splits five ways and the parts sum to it exactly, using both ends
of the update-to-update interval rather than only its width. That is what fixes
the wasm story: at 10k rows in wasm, 45ms of a 102ms frame is outside the apply
altogether -- a 17ms requestAnimationFrame floor plus ~28ms taking delivery of
the websocket frame -- and no decoder touches either. So construction is 14% of
a wasm frame, not the ~77% the README's remainder sentence implied, and the wasm
cost is a floor plus a rate (~17ms + 8.5us/row), never the flat ~11us/row the
README quoted: that figure embeds the constant floor and misleads by 3.5x at 500
rows. `build` being the leftover is also load-bearing now -- it caught this
commit double-booking the frame-level scan, and a negative bucket fails the
phase.

Also here, all from the review:
  - config() normalises a set-but-empty value to nil. "" is truthy in Lua, so
    STDB_BULK_SWEEP= yielded "", matched no digits, and crashed PHASE6 on
    format("%d", nil). The HTML5 branch already normalised; the environment
    branch did not. Fixed in config() rather than at a callsite, because all six
    readers carried it.
  - The clock comment claimed socket.gettime is absent under HTML5. It is not;
    both tiers resolve it. The measured tick is printed instead of asserted --
    0.001ms native, 1.000ms wasm.
  - A bucket whose median is under four clock ticks is marked
    AT-CLOCK-RESOLUTION, which is what the wasm 100-row point is: 17 of its 18ms
    is the rAF floor.
  - RUNS still defaults to 1 so an ordinary conformance run stays quick, but a
    single sample now says ONE-SAMPLE. It earns that: one wasm sample read 136ms
    where the six-sample median is 102ms.
  - The apparatus reports its own cost, since it is 5% of a native frame and is
    the only asymmetry between the two schema measurements.

Measured thresholds, swept across each crossing rather than extrapolated:
16.7ms at ~6100 rows native (~5700 typed), 100ms at ~31000, 100ms at ~9800 in
wasm. The typed penalty is +13% to +20%. Per-row cost drifts +33% by 100k, not
25%, and the spread makes one number meaningless past ~25000 rows.
yuris force-pushed feat/30-frame-budget from 91054417f9
All checks were successful
ci / ci (pull_request) Successful in 20s
Details
to fd0cca21ad
All checks were successful
ci / ci (pull_request) Successful in 20s
Details
2026-08-20 15:57:51 +00:00 Compare
yuris commented 2026-08-20 16:00:04 +00:00
Author
Owner
Copy link

Review actioned. All eight findings, and the two blocking ones changed the
conclusions rather than just the prose.

F1 (BLOCKING) — quote_big_ints is now its own bucket, and it changed everything

You were right that the split stopped one function short, and the consequence was
larger than "a bucket is mislabelled".

quote_big_ints is not reachable through any injected table — it is a file-local
and protocol.init takes only json. Rather than edit frozen stdb/, the harness
wraps string.gsub and keys on the exact pattern literal, which is safe because
that pattern appears nowhere else in stdb/ and no string function is localised
there, so text:gsub resolves through the string table at call time. Verified in
bare Lua against the real module before spending an engine build, including that
gsub's two return values are preserved.

PHASE6 now reports five buckets that sum to the frame exactly — pre,
json.decode, quote, build, post — obtained with no extra clock reads by
keeping both ends of the interval instead of only its width.

Your numbers are confirmed. Native medians, 44 samples at 10k:

frame 31.0 = pre 0.6 + decode 14.8 + quote 12.6 + build 3.6 + post 0.1

At 50k, quoting (77.5ms) costs more than the JSON parser (76.6ms).

Corrected ceiling. Native untouchable is 11.3–11.7% net of apparatus → 8.5×,
which is above your 2.5–6× estimate. WASM is where I have to correct you back:
your ~13% is construction only, but the rAF floor plus websocket delivery are
another 45% of the wasm frame and no decoder touches those either, so the real wasm
frame ceiling is 1.7×, not 2.5–6×. On the SDK's apply alone it is 4.1×. The
published 1.8×/1.3× was Amdahl on json.decode's share alone; #9 now carries the
correction and a retraction.

And the finding under the finding, now #69: quote_big_ints has no guard,
while quote_wide_ints has a necessary-condition guard and short-circuits. On a
table with no 128-bit column the hand-written path scans the whole frame and every
row to rewrite nothing. That asymmetry is also why the typed path's quote bucket
halves (12.2 → 6.4ms at 10k) while build rises — neither of us had spotted the
cause. It is the cheapest real optimisation in the whole exercise and it takes much
of BSATN's ground for one plain find.

Your finding also indicted this PR's own apparatus. The old bracket= figure
was apply_end - decode_first, and decode_server quotes the whole frame before
it decodes anything — so the single most expensive scan of the batch was
structurally outside the measured window. It could not have seen this cost even if
it had wrapped the function. An early corrected run surfaced this as build = -2.8ms; the anchor is now "whichever instrumented call fires first", with a
permanent guard that fails the phase if build goes more negative than
max(4 ticks, 5% of frame).

F2 (BLOCKING) — wasm attribution fixed

Confirmed within a few ms on every bucket: 17 / 28 / 23 / 20 / 14 / 1. The rAF
floor is now measured rather than asserted — the 100-row wasm point is an 18.0ms
frame of which pre is 17.0ms, 94% — and the README now states the floor at all,
where before it appeared only in the PR body.

~11us/row is gone. Measured: 180 us/row at 100 rows, 23 at 1k, 11.0 at 5k, 10.2
at 10k. The README now gives wasm as a floor plus a rate (~17ms + 8.5us/row at 10k) and says the per-row part climbs, because socket delivery is inside it. The
remainder sentence is scoped to native.

F3 — re-measured, and both of us were off

Per-row drift to 100k is +33% on medians (yours +49%, the README's +25%), +45%
on worst-case samples. Now quoted as a range, +30% to +45%. Your stronger point
stands: the PR's 327.5ms sat below all eight of my samples (median 407.9,
range 364.5–445.3, spread 20%). The README now also says where one number stops
describing the behaviour — ~25,000 rows, past which per-row cost is +10% and
worst-case spread reaches 28–31%.

F4 — you were slightly high, and the growth claim does not reproduce

The naive comparison is unusable: the untyped baseline itself drifted 15% between
two passes 30s apart, the same magnitude as the effect. Six alternating passes
(U,T,U,T,U,T), 18 samples per point per schema, with json.decode's median as a
control (it must match on both paths — same bytes, same C parser — and it does).

Corrected for apparatus (the hand-written path carries ~2× the instrumentation,
which biases against the typed penalty): +13% at 1k, +20% at 10k, +16% at
50k — band 13–20%. So the PR's 12–17% was essentially right, your 18–25% slightly
high, and the penalty does not grow with row count: 50k is below 10k. Typed
16.7ms threshold measured directly at ~5,700 rows (yours ~5,300).

F5, F6, F7, F8 — all fixed

  • Fixed inside config() itself, which repaired the same set-but-empty trap for
    all six readers, not just STDB_BULK_SWEEP. STDB_BULK_SWEEP= and
    STDB_BULK_SWEEP=none both pass now.
  • The clock comment is corrected, and the tick is measured at load (spin until
    it moves) and printed rather than asserted: 0.001ms native, 1.000ms wasm.
  • Sub-resolution buckets are now auto-marked AT-CLOCK-RESOLUTION with a
    quant=±sqrt(bucket*tick) uncertainty on every line. It fires on exactly the
    wasm 100- and 1000-row points. I kept the 100-row point rather than dropping it,
    because it is the direct measurement of the rAF floor.
  • STDB_BULK_RUNS stays at 1 with a ONE-SAMPLE marker, and the justification
    turned out empirical rather than a guess: the single-sample acceptance run read
    136ms where the six-sample median is 102ms, i.e. 31% high. Raising the default
    triples PHASE6 in every ordinary conformance run for a number that run does not
    publish; the marker cannot be missed.

Thresholds — all three hold, all swept across the crossing

README before now
native 16.7ms ~6,400 ~6,100
native 100ms ~32,000 ~31,000
wasm 100ms ~9,400 ~9,800

Each swept across the crossing rather than extrapolated, which is why they are
quotable at all. Your ~6,200 for the first was closer than mine.

State

Rebased onto current main (d6ce8a2) — it was still on cf8b430, which you
would have seen as phantom deletions. Diff is 5 files, stdb/ untouched.

lua dev/test/runner.lua       7 passed, 0 failed
stylua --check stdb dev/test  exit 0
conformance.sh                PASS, PHASE1-6
html5-conformance.sh          PASS

Independently reproduced after the rebase:
dec=39% quo=42% bld=15%, 28.0ms at 10k rows.

One environment note, not a code failure: one engine process in an alternating
pass could not reach the server at all (182 reconnect attempts over 900s) while
every other pass passed and the server pinged 200 afterwards. Re-run clean. Worth
recording in case it recurs.

Review actioned. All eight findings, and the two blocking ones changed the conclusions rather than just the prose. ## F1 (BLOCKING) — `quote_big_ints` is now its own bucket, and it changed everything You were right that the split stopped one function short, and the consequence was larger than "a bucket is mislabelled". `quote_big_ints` is not reachable through any injected table — it is a file-local and `protocol.init` takes only `json`. Rather than edit frozen `stdb/`, the harness wraps `string.gsub` and keys on the exact pattern literal, which is safe because that pattern appears nowhere else in `stdb/` and no string function is localised there, so `text:gsub` resolves through the string table at call time. Verified in bare Lua against the real module before spending an engine build, including that `gsub`'s two return values are preserved. PHASE6 now reports **five** buckets that sum to the frame exactly — `pre`, `json.decode`, `quote`, `build`, `post` — obtained with no extra clock reads by keeping both ends of the interval instead of only its width. Your numbers are confirmed. Native medians, 44 samples at 10k: ``` frame 31.0 = pre 0.6 + decode 14.8 + quote 12.6 + build 3.6 + post 0.1 ``` At 50k, quoting (77.5ms) costs **more than the JSON parser** (76.6ms). **Corrected ceiling.** Native untouchable is 11.3–11.7% net of apparatus → **8.5×**, which is *above* your 2.5–6× estimate. WASM is where I have to correct you back: your ~13% is construction only, but the rAF floor plus websocket delivery are another 45% of the wasm frame and no decoder touches those either, so the real wasm *frame* ceiling is **1.7×**, not 2.5–6×. On the SDK's apply alone it is 4.1×. The published 1.8×/1.3× was Amdahl on `json.decode`'s share alone; #9 now carries the correction and a retraction. **And the finding under the finding, now #69:** `quote_big_ints` has **no guard**, while `quote_wide_ints` has a necessary-condition guard and short-circuits. On a table with no 128-bit column the hand-written path scans the whole frame and every row to rewrite nothing. That asymmetry is also why the typed path's quote bucket *halves* (12.2 → 6.4ms at 10k) while `build` rises — neither of us had spotted the cause. It is the cheapest real optimisation in the whole exercise and it takes much of BSATN's ground for one plain `find`. **Your finding also indicted this PR's own apparatus.** The old `bracket=` figure was `apply_end - decode_first`, and `decode_server` quotes the whole frame *before* it decodes anything — so the single most expensive scan of the batch was structurally outside the measured window. It could not have seen this cost even if it had wrapped the function. An early corrected run surfaced this as `build = -2.8ms`; the anchor is now "whichever instrumented call fires first", with a permanent guard that fails the phase if `build` goes more negative than `max(4 ticks, 5% of frame)`. ## F2 (BLOCKING) — wasm attribution fixed Confirmed within a few ms on every bucket: 17 / 28 / 23 / 20 / 14 / 1. The rAF floor is now measured rather than asserted — the 100-row wasm point is an 18.0ms frame of which `pre` is 17.0ms, 94% — and the README now states the floor at all, where before it appeared only in the PR body. `~11us/row` is gone. Measured: 180 us/row at 100 rows, 23 at 1k, 11.0 at 5k, 10.2 at 10k. The README now gives wasm as a floor plus a rate (`~17ms + 8.5us/row at 10k`) and says the per-row part *climbs*, because socket delivery is inside it. The remainder sentence is scoped to native. ## F3 — re-measured, and both of us were off Per-row drift to 100k is **+33%** on medians (yours +49%, the README's +25%), +45% on worst-case samples. Now quoted as a range, +30% to +45%. Your stronger point stands: the PR's 327.5ms sat below **all eight** of my samples (median 407.9, range 364.5–445.3, spread 20%). The README now also says where one number stops describing the behaviour — ~25,000 rows, past which per-row cost is +10% and worst-case spread reaches 28–31%. ## F4 — you were slightly high, and the growth claim does not reproduce The naive comparison is unusable: the untyped baseline itself drifted 15% between two passes 30s apart, the same magnitude as the effect. Six alternating passes (U,T,U,T,U,T), 18 samples per point per schema, with `json.decode`'s median as a control (it must match on both paths — same bytes, same C parser — and it does). Corrected for apparatus (the hand-written path carries ~2× the instrumentation, which biases *against* the typed penalty): **+13% at 1k, +20% at 10k, +16% at 50k — band 13–20%.** So the PR's 12–17% was essentially right, your 18–25% slightly high, and **the penalty does not grow with row count**: 50k is below 10k. Typed 16.7ms threshold measured directly at **~5,700 rows** (yours ~5,300). ## F5, F6, F7, F8 — all fixed - Fixed inside `config()` itself, which repaired the same set-but-empty trap for all six readers, not just `STDB_BULK_SWEEP`. `STDB_BULK_SWEEP=` and `STDB_BULK_SWEEP=none` both pass now. - The clock comment is corrected, and the tick is **measured** at load (spin until it moves) and printed rather than asserted: 0.001ms native, 1.000ms wasm. - Sub-resolution buckets are now auto-marked `AT-CLOCK-RESOLUTION` with a `quant=±sqrt(bucket*tick)` uncertainty on every line. It fires on exactly the wasm 100- and 1000-row points. I kept the 100-row point rather than dropping it, because it is the direct measurement of the rAF floor. - `STDB_BULK_RUNS` stays at 1 with a `ONE-SAMPLE` marker, and the justification turned out empirical rather than a guess: the single-sample acceptance run read 136ms where the six-sample median is 102ms, i.e. 31% high. Raising the default triples PHASE6 in every ordinary conformance run for a number that run does not publish; the marker cannot be missed. ## Thresholds — all three hold, all swept across the crossing | | README before | now | |---|---|---| | native 16.7ms | ~6,400 | **~6,100** | | native 100ms | ~32,000 | **~31,000** | | wasm 100ms | ~9,400 | **~9,800** | Each swept across the crossing rather than extrapolated, which is why they are quotable at all. Your ~6,200 for the first was closer than mine. ## State Rebased onto current `main` (`d6ce8a2`) — it was still on `cf8b430`, which you would have seen as phantom deletions. Diff is 5 files, `stdb/` untouched. ``` lua dev/test/runner.lua 7 passed, 0 failed stylua --check stdb dev/test exit 0 conformance.sh PASS, PHASE1-6 html5-conformance.sh PASS ``` Independently reproduced after the rebase: `dec=39% quo=42% bld=15%`, 28.0ms at 10k rows. One environment note, not a code failure: one engine process in an alternating pass could not reach the server at all (182 reconnect attempts over 900s) while every other pass passed and the server pinged 200 afterwards. Re-run clean. Worth recording in case it recurs.
yuris merged commit 8d8bc2c6a5 into main 2026-08-20 16:00:15 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area:defold
Defold engine integration

  
area:docs
Documentation and examples

  
area:infra
Flake, CI, secretspec

  
area:protocol
Wire protocol conformance

  
area:testing
Test harness and fixtures

  
kind:bug
Something broken

  
kind:chore
Maintenance

  
phase:breadth
UnsubscribeMulti, procedures, BSATN

  
phase:core
Core SDK layers (protocol/cache/connection)

  
phase:hardening
Reconnect, token persistence, errors

  
phase:slice
Vertical slice: first playable end-to-end

No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/spacetimedb-defold!66
No description provided.