yuris/spacetimedb-defold
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 9 Packages Wiki Activity Actions

BSATN codec — the only route to the v2 message schema #9

New issue
Closed
opened 2026-08-19 06:49:31 +00:00 by yuris · 8 comments
yuris commented 2026-08-19 06:49:31 +00:00
Owner
Copy link

Rescoped after the 2.8.2 protocol capture (see #3).

This was filed as a bandwidth optimisation. It is more than that: the v1.json.spacetimedb subprotocol carries the v1 message schema, and the v2 schema (InitialConnection, QuerySetUpdate, ReducerResult) is reachable only over v2.bsatn.spacetimedb / v3 framing. The server's routes/subscribe.rs maps TEXT_PROTOCOL to WsVersion::V1 — there is no JSON transport for v2.

So the real decision: stay on v1-over-JSON (works against 2.x servers, zero binary code, shipping today) or write BSATN to unlock v2 semantics.

Reasons that might force BSATN later:

  • v2 subscriptions are query-set scoped (query_set_id), giving cleaner unsubscribe and per-set error handling
  • v2 separates ReducerResult from TransactionUpdate, making correlation less awkward
  • bandwidth: JSON rows are double-encoded strings, roughly 2x the bytes

Decision made 2026-08-21: build it now; benchmark demoted from gate to deliverable.
The v2 semantics — event tables above all — are the reason, not the bytes. Waiting on a benchmark
was gating an unlock behind a measurement of the wrong axis, so the numbers now ship with the
codec instead of deciding whether it exists. v2 lands opt-in (connect({ protocol = "v2" }));
v1-over-JSON stays the default and its v1.0 value surface stays frozen. Flipping the default is a
separate later decision, after v2 conformance runs green for a while.

Implementation tracker — five phases, one PR each against main. Wire facts read from the v2.8.2
tag; each phase is gated on lua dev/test/runner.lua, with the conformance suites as the Phase 4
gate.

  • Phase 1 — stdb/bsatn.lua: primitive codec. Reader over a Lua string with an offset
    cursor, writer accumulating fragments into table.concat. All little-endian: u8..u64 / i8..i64
    (64-bit as a double, exact ≤ 2^53), u128/i128 → exact decimal digit strings, u256/i256 →
    minimal-width 0x…/-0x… hex, f32/f64 via IEEE-754 + math.ldexp, string/bytes as u32
    length + bytes. Pure arithmetic — no require("bit"), because the unit-test host's Lua 5.1
    lacks the module. Tests: byte-exact vectors per primitive, both directions.
  • Phase 2 — stdb/protocol_v2.lua: message schema + typed row decode. decode_server for
    the 8 ServerMessage variants (compression byte first; BsatnRowList slices to an array of
    still-encoded row strings, both FixedSize and RowOffsets), encode_client for the 5
    ClientMessage variants, and decode_typed_bsatn — the binary twin of protocol.decode_typed
    over the same compositional tag grammar. Tests: hand-built frames per variant, plus a
    differential test proving a BSATN all_types row decodes value-for-value equal to the JSON
    one.
  • Phase 3 — integration. stdb/init.lua: SUBPROTOCOL_V2, the protocol opt, the
    compression=None param, binary frames. stdb/connection.lua: a handlers_v2 table mapping
    the v2 messages onto the existing event and cache surface, so on_event consumers do not
    fork — plus the new event type for event-table rows, which are emitted per row and never
    cached. stdb/cache.lua: one decoder seam in decode_rows, and a table without generated
    types reported undecodable (nothing names columns on the binary wire).
  • Phase 4 — capture, fixtures, conformance. dev/tools/capture-v2.mjs dumps hex frames to
    session_v2.jsonl; pin the open questions against it (ReducerResult timestamp unit — micros
    or nanos, client frames accepted without a compression tag, Option/Result tag ordinals,
    identity byte order) and record each answer at the decode site. Then the full six-phase smoke
    over v2, native and HTML5 — the latter also proves binary frames survive emscripten.
  • Phase 5 — benchmark and docs. PHASE6 bulk sweep over both protocols and a wire-byte
    comparison from the two captures, published here and on the wiki. Then Protocol-Notes gains a
    "v2 over BSATN" section, Usage-Guide documents the opt-in and the event-table event type,
    Roadmap moves to done-pending-default-flip, and #77 gets its tick.

Deliberately out of scope: v3 (same message schema, multiple messages coalesced per frame — v2
sends exactly one, which keeps handle_frame unchanged) and frame decompression (we always ask
for compression=None; a pure-Lua inflate stays unbuilt). CI keeps running v1 per-PR: a v2
dispatch input lands in Phase 4, but adding it to the path filter doubles run time and waits until
v2 conformance is green and stable locally.

Results

All five phases shipped. Every number below was measured on one workstation — a Ryzen 7 8845HS
running headless Defold against a local spacetime:v2.8.2 — so read them as local figures. The
~117ms/10k baseline this issue once quoted came off a different machine and is not comparable to
them.

Wire bytes — v2 is 46% of v1

Identical traffic over both wires, back to back against one server, from a purpose-built probe. The
two committed capture files drive different sequences and are not byte-comparable, so they
were not the source. The figure counted is the subscription readback frame.

rows v1 bytes v2 bytes v2 as % of v1
100 9,818 4,621 47.1%
1,000 100,406 46,921 46.7%
10,000 1,042,187 478,921 46.0%
50,000 5,360,120 2,438,921 45.5%

Whole session at 10k, server → client: 2,085,999 B against 958,329 B (45.9%). The identical four
client messages: 309 B against 96 B (31.1%). "Roughly half the bytes" — the estimate this issue was
justified with — is confirmed, and turns out to have been slightly conservative.

Apply latency — v2 is faster at every row count; the absolutes are not pinned down

This section is a correction. It previously published 20.0ms against 10.5ms at 10k rows, a ratio
of 0.53, and a per-row 2.00µs against 1.05µs, with no statement of the conditions they were measured
under. Those figures were a median of 3 on a Ryzen 7 8845HS with the machine's load unrecorded. A
re-run of the same commit measured every native absolute 2-3x higher — v1 median 61.7ms at 10k
against 20.0ms, v2 44.6ms against 10.5ms, over 5 runs — with /proc/loadavg at 11.06 on 16 cores
from concurrent work, and the harness itself flagging up to 47% spread on two of the sweeps.

The ratio survived in direction but not in value: 0.59-0.65 re-measured, against the 0.53
published. That is a material difference, not a rounding one, so it is stated rather than smoothed
into a range that flatters the change. The honest claim is: v2 is consistently faster at every row
count on both tiers, and the size of the win is not pinned to two digits.

PHASE6 times the single frame that applies a first subscription. The original table is kept below as
one machine's snapshot under unknown load — not a regression baseline:

rows v1 frame ms v2 frame ms
100 0.4, 0.4 0.4, 0.4
1,000 2.1, 2.3 1.3, 1.3
5,000 9.9, 10.4 6.3, 5.4
10,000 20.0, 20.1 10.5, 13.6
25,000 53.6, 54.3 35.5, 31.1
50,000 105.7, 113.7 86.3, 81.9

Both sweeps of each run are given because conformance.sh runs the smoke twice. Before reading a
slower number here as a regression, check loadavg and the harness spread warning — that exact
ambiguity is what forced this correction. A clean re-measure on a quiesced box is an open
follow-up below. The intended command, for whoever gets an idle machine:

STDB_BULK_SWEEP=100,1000,5000,10000,25000,50000 STDB_BULK_RUNS=5 STDB_TIMEOUT=900 \
  nix develop .#conformance --command bash dev/tools/conformance.sh      # and again with STDB_PROTOCOL=v2

The wire-byte tables above are unaffected. They are byte counts, not timings, and they stand
exactly as published: v2 is ~46% of v1 on row payloads and 31% on client frames.

Where the time goes, at 10,000 rows, native — same caveat on the absolutes, but the shape is the
load-independent part:

bucket v1 v2
pre 0.5 0.3
decode 11.4 (json.decode, 57%) 8.4 (BSATN, 80%)
big-integer quoting 0.0 structurally absent
row construction 6.8 (34%) 1.9 (18%)
post 0.1 0.1
frame 20.0 ms 10.5 ms

Two claims in that split are structural — they follow from the encodings, not from the clock, and
both reproduce regardless of load.

The big-integer quoting bucket cannot exist on v2. That pass is v1 rewriting wide integer
literals as strings before json.decode can round them; it exists at all only because JSON cannot
carry a u128 exactly. A binary wire has no text to rescue. This is now enforced, not merely
observed to be zero: the v2 path asserts zero quoting ATTEMPTS — a claim about
guard_calls/wide_calls that fails if a per-row string.find scan ever returns on the binary wire
with its time billed nowhere. The three hit-based assertions beside it are 0 == 0 on v2 and would
have passed unchanged, which is why the ATTEMPTS assertion had to be added.

Row construction falls with decoding, 6.8ms to 1.9ms: a BSATN row builds fewer intermediate Lua
values on its way to the same table, so the win is not confined to the decode bucket.

HTML5 (the wasm bundle in Chromium), 10,000 rows, one sample each: v1 is 112.0ms total — 68.0 pre,
21.0 json.decode, 23.0 build — against v2's 70.0ms — 17.0 pre, 45.0 BSATN decode, 8.0 build. The
structural fact is the collapse of pre, which is frame delivery: on v1 it is the single largest
bucket at 61% of the frame, on v2 it is 24% and no longer dominant. That is the larger v1 payload
showing up as delivery cost, and it dominates the browser tier in a way it never does natively. The
collapse is load-independent; the millisecond totals carry the same caveat as the native table.

Per-sweep tables, the B/row columns and the methodology note are on the wiki:
Bytes and latency, v1 against v2.

The four open questions, pinned at capture

  1. ReducerResult.timestamp is MICROSECONDS. Upstream v2.rs:378 documents the field as
    nanoseconds and is wrong — the captured value is three orders of magnitude short of nanos, and
    is exactly the unit v1's envelope uses. Row-level Timestamp columns are micros on both
    protocols, so nothing divides differently per protocol.
  2. Identity hex text is the byte-REVERSE of the 32 wire bytes. An Identity is a little-endian
    32-byte number in BSATN and the server renders hex from the big-endian array, so the 64 digits a
    v1 frame carries — and the JWT's own hex_identity — hold the wire bytes reversed. Confirmed
    against the same identity over both protocols. Decode without the reversal and you get a plausible
    hex string that matches nothing.
  3. Option is 0 Some / 1 None; Result is 0 Ok / 1 Err. The same ordinals v1's
    positional sums use, so [0, v] / [1, {}] on JSON and tag 0 / 1 on BSATN say the same thing.
  4. A client frame with no compression byte is accepted. The envelope asymmetry is the protocol,
    not something to work around: the server reads the sum tag first on the frames it receives, and
    writes a compression byte on the frames it sends.

Each answer is recorded as a comment at the affected decode site.

Conformance and CI

All four suites pass end to end — the full six-phase smoke over v1 native, v2 native, v2 HTML5 and
v1 HTML5. The HTML5 pair is what proves binary frames survive emscripten; keeping the v1 pair is
what shows a second protocol cost the first nothing. Unit suite: 11 passed, 0 failed.

CI runs v2 by dispatch only — a workflow_dispatch protocol input that defaults to v1. The
automatic push and pull-request path filter stays v1-only, because a second tier doubles run time for
a protocol nothing defaults to yet.

Divergences from v1, worth having written down

  • v2 has no OutOfEnergy outcome; it arrives as InternalError text.
  • The legacy subscribe raises on v2 — use subscribe_query.
  • light is v1-only; the v2 schema has no light mode.
  • A table needs generated types to be decodable at all: nothing on the binary wire names a column.
  • A reducer's ret_value stays raw BSATN (see the follow-up below).

Follow-ups, not part of this milestone

The three below were open when this milestone closed and remain open — none was picked up during
the review round. Two more were added by it.

  • Flip the default to v2. A separate decision, to be taken once v2 conformance has been green
    for a while. Until then v1 stays what connect negotiates when you say nothing.
  • Wire v2 into the automatic CI triggers, with that flip and not before — the per-PR path
    filter is v1-only today for run time.
  • Emit reducer returns from the schema generator, so a ReducerResult's ret_value can be
    decoded to a typed value instead of handed back as raw BSATN. Procedures already carry
    returns; reducers do not, which is the only reason that value is untyped.
  • Emit the event = true marker for event tables from the schema generator. Verified against
    dev/test/fixtures/chat_schema.lua as of this comment: dev/server/index.js:183 declares
    ping with { name: 'ping', public: true, event: true }, and the generated fixture entry for
    ping (lines 113-116) carries only columns and types — the marker is dropped. Views are
    not: the generator does emit view = true for message_flip and message_latest, so this is
    an omission in one code path rather than a missing capability. The SDK works without it because
    an event table's transience is carried by the wire (TableUpdateRows::EventTable) rather than
    by the schema, so nothing is broken today — but a consumer reading the generated schema cannot
    tell an event table from a pk-less persistent one.
  • Re-measure apply latency on a quiesced machine and replace the caveated absolutes in the
    Results section above with real ones. Both wires, STDB_BULK_RUNS=5, the six-bucket sweep, with
    /proc/loadavg recorded alongside the figures so the next reader can tell a regression from a
    busy box. The attempt during the review round was abandoned rather than published: the machine
    was at loadavg 8.7-10.0 on 16 cores throughout (a Wine game and Steam, ~4 cores of it), which is
    the same class of noise that produced the numbers this correction fixes. Publishing a second set
    of load-contaminated figures would have re-created the problem, not solved it.
**Rescoped after the 2.8.2 protocol capture (see #3).** This was filed as a bandwidth optimisation. It is more than that: the `v1.json.spacetimedb` subprotocol carries the **v1** message schema, and the v2 schema (`InitialConnection`, `QuerySetUpdate`, `ReducerResult`) is reachable **only** over `v2.bsatn.spacetimedb` / v3 framing. The server's `routes/subscribe.rs` maps TEXT_PROTOCOL to `WsVersion::V1` — there is no JSON transport for v2. So the real decision: stay on v1-over-JSON (works against 2.x servers, zero binary code, shipping today) or write BSATN to unlock v2 semantics. Reasons that might force BSATN later: - v2 subscriptions are query-set scoped (`query_set_id`), giving cleaner unsubscribe and per-set error handling - v2 separates `ReducerResult` from `TransactionUpdate`, making correlation less awkward - bandwidth: JSON rows are double-encoded strings, roughly 2x the bytes --- **Decision made 2026-08-21: build it now; benchmark demoted from gate to deliverable.** The v2 semantics — event tables above all — are the reason, not the bytes. Waiting on a benchmark was gating an unlock behind a measurement of the wrong axis, so the numbers now ship *with* the codec instead of deciding whether it exists. v2 lands **opt-in** (`connect({ protocol = "v2" })`); v1-over-JSON stays the default and its v1.0 value surface stays frozen. Flipping the default is a separate later decision, after v2 conformance runs green for a while. Implementation tracker — five phases, one PR each against `main`. Wire facts read from the `v2.8.2` tag; each phase is gated on `lua dev/test/runner.lua`, with the conformance suites as the Phase 4 gate. - [x] **Phase 1 — `stdb/bsatn.lua`: primitive codec.** Reader over a Lua string with an offset cursor, writer accumulating fragments into `table.concat`. All little-endian: u8..u64 / i8..i64 (64-bit as a double, exact ≤ 2^53), u128/i128 → exact decimal digit strings, u256/i256 → minimal-width `0x…`/`-0x…` hex, f32/f64 via IEEE-754 + `math.ldexp`, `string`/`bytes` as u32 length + bytes. Pure arithmetic — no `require("bit")`, because the unit-test host's Lua 5.1 lacks the module. Tests: byte-exact vectors per primitive, both directions. - [x] **Phase 2 — `stdb/protocol_v2.lua`: message schema + typed row decode.** `decode_server` for the 8 `ServerMessage` variants (compression byte first; `BsatnRowList` slices to an array of still-encoded row strings, both `FixedSize` and `RowOffsets`), `encode_client` for the 5 `ClientMessage` variants, and `decode_typed_bsatn` — the binary twin of `protocol.decode_typed` over the same compositional tag grammar. Tests: hand-built frames per variant, plus a differential test proving a BSATN `all_types` row decodes value-for-value equal to the JSON one. - [x] **Phase 3 — integration.** `stdb/init.lua`: `SUBPROTOCOL_V2`, the `protocol` opt, the `compression=None` param, binary frames. `stdb/connection.lua`: a `handlers_v2` table mapping the v2 messages onto the *existing* event and cache surface, so `on_event` consumers do not fork — plus the new `event` type for event-table rows, which are emitted per row and never cached. `stdb/cache.lua`: one decoder seam in `decode_rows`, and a table without generated `types` reported undecodable (nothing names columns on the binary wire). - [x] **Phase 4 — capture, fixtures, conformance.** `dev/tools/capture-v2.mjs` dumps hex frames to `session_v2.jsonl`; pin the open questions against it (`ReducerResult` timestamp unit — micros or nanos, client frames accepted without a compression tag, `Option`/`Result` tag ordinals, identity byte order) and record each answer at the decode site. Then the full six-phase smoke over v2, native and HTML5 — the latter also proves binary frames survive emscripten. - [x] **Phase 5 — benchmark and docs.** PHASE6 bulk sweep over both protocols and a wire-byte comparison from the two captures, published here and on the wiki. Then Protocol-Notes gains a "v2 over BSATN" section, Usage-Guide documents the opt-in and the event-table event type, Roadmap moves to done-pending-default-flip, and #77 gets its tick. Deliberately out of scope: **v3** (same message schema, multiple messages coalesced per frame — v2 sends exactly one, which keeps `handle_frame` unchanged) and **frame decompression** (we always ask for `compression=None`; a pure-Lua inflate stays unbuilt). CI keeps running v1 per-PR: a v2 dispatch input lands in Phase 4, but adding it to the path filter doubles run time and waits until v2 conformance is green and stable locally. --- ## Results All five phases shipped. Every number below was measured on one workstation — a Ryzen 7 8845HS running headless Defold against a local `spacetime:v2.8.2` — so read them as **local** figures. The ~117ms/10k baseline this issue once quoted came off a different machine and is **not** comparable to them. ### Wire bytes — v2 is 46% of v1 Identical traffic over both wires, back to back against one server, from a purpose-built probe. The two committed capture files drive **different** sequences and are **not** byte-comparable, so they were not the source. The figure counted is the subscription readback frame. | rows | v1 bytes | v2 bytes | v2 as % of v1 | |---|---|---|---| | 100 | 9,818 | 4,621 | 47.1% | | 1,000 | 100,406 | 46,921 | 46.7% | | 10,000 | 1,042,187 | 478,921 | 46.0% | | 50,000 | 5,360,120 | 2,438,921 | 45.5% | Whole session at 10k, server → client: 2,085,999 B against 958,329 B (45.9%). The identical four client messages: 309 B against 96 B (31.1%). "Roughly half the bytes" — the estimate this issue was justified with — is confirmed, and turns out to have been slightly conservative. ### Apply latency — v2 is faster at every row count; the absolutes are not pinned down **This section is a correction.** It previously published 20.0ms against 10.5ms at 10k rows, a ratio of 0.53, and a per-row 2.00µs against 1.05µs, with no statement of the conditions they were measured under. Those figures were a median of 3 on a Ryzen 7 8845HS with the machine's load unrecorded. A re-run of the same commit measured every native absolute **2-3x higher** — v1 median 61.7ms at 10k against 20.0ms, v2 44.6ms against 10.5ms, over 5 runs — with `/proc/loadavg` at **11.06 on 16 cores** from concurrent work, and the harness itself flagging up to **47% spread** on two of the sweeps. The ratio survived in direction but not in value: **0.59-0.65 re-measured, against the 0.53 published.** That is a material difference, not a rounding one, so it is stated rather than smoothed into a range that flatters the change. The honest claim is: v2 is consistently faster at every row count on both tiers, and the size of the win is not pinned to two digits. PHASE6 times the single frame that applies a first subscription. The original table is kept below as one machine's snapshot under unknown load — **not** a regression baseline: | rows | v1 frame ms | v2 frame ms | |---|---|---| | 100 | 0.4, 0.4 | 0.4, 0.4 | | 1,000 | 2.1, 2.3 | 1.3, 1.3 | | 5,000 | 9.9, 10.4 | 6.3, 5.4 | | 10,000 | 20.0, 20.1 | 10.5, 13.6 | | 25,000 | 53.6, 54.3 | 35.5, 31.1 | | 50,000 | 105.7, 113.7 | 86.3, 81.9 | Both sweeps of each run are given because `conformance.sh` runs the smoke twice. Before reading a slower number here as a regression, check `loadavg` and the harness spread warning — that exact ambiguity is what forced this correction. **A clean re-measure on a quiesced box is an open follow-up below.** The intended command, for whoever gets an idle machine: ``` STDB_BULK_SWEEP=100,1000,5000,10000,25000,50000 STDB_BULK_RUNS=5 STDB_TIMEOUT=900 \ nix develop .#conformance --command bash dev/tools/conformance.sh # and again with STDB_PROTOCOL=v2 ``` **The wire-byte tables above are unaffected.** They are byte counts, not timings, and they stand exactly as published: v2 is ~46% of v1 on row payloads and 31% on client frames. Where the time goes, at 10,000 rows, native — same caveat on the absolutes, but the *shape* is the load-independent part: | bucket | v1 | v2 | |---|---|---| | pre | 0.5 | 0.3 | | decode | 11.4 (`json.decode`, 57%) | 8.4 (BSATN, 80%) | | big-integer quoting | 0.0 | structurally absent | | row construction | 6.8 (34%) | 1.9 (18%) | | post | 0.1 | 0.1 | | **frame** | **20.0 ms** | **10.5 ms** | Two claims in that split are structural — they follow from the encodings, not from the clock, and both reproduce regardless of load. **The big-integer quoting bucket cannot exist on v2.** That pass is v1 rewriting wide integer literals as strings before `json.decode` can round them; it exists at all only because JSON cannot carry a `u128` exactly. A binary wire has no text to rescue. This is now **enforced, not merely observed to be zero**: the v2 path asserts zero quoting ATTEMPTS — a claim about `guard_calls`/`wide_calls` that fails if a per-row `string.find` scan ever returns on the binary wire with its time billed nowhere. The three hit-based assertions beside it are `0 == 0` on v2 and would have passed unchanged, which is why the ATTEMPTS assertion had to be added. **Row construction falls with decoding**, 6.8ms to 1.9ms: a BSATN row builds fewer intermediate Lua values on its way to the same table, so the win is not confined to the decode bucket. HTML5 (the wasm bundle in Chromium), 10,000 rows, one sample each: v1 is 112.0ms total — 68.0 pre, 21.0 `json.decode`, 23.0 build — against v2's 70.0ms — 17.0 pre, 45.0 BSATN decode, 8.0 build. The structural fact is the **collapse of `pre`**, which is frame delivery: on v1 it is the single largest bucket at 61% of the frame, on v2 it is 24% and no longer dominant. That is the larger v1 payload showing up as delivery cost, and it dominates the browser tier in a way it never does natively. The collapse is load-independent; the millisecond totals carry the same caveat as the native table. Per-sweep tables, the B/row columns and the methodology note are on the wiki: [Bytes and latency, v1 against v2](https://git.nokopia.com/yuris/spacetimedb-defold/wiki/Protocol-Notes#bytes-and-latency-v1-against-v2). ### The four open questions, pinned at capture 1. **`ReducerResult.timestamp` is MICROSECONDS.** Upstream `v2.rs:378` documents the field as nanoseconds and is **wrong** — the captured value is three orders of magnitude short of nanos, and is exactly the unit v1's envelope uses. Row-level `Timestamp` columns are micros on both protocols, so nothing divides differently per protocol. 2. **Identity hex text is the byte-REVERSE of the 32 wire bytes.** An `Identity` is a little-endian 32-byte number in BSATN and the server renders hex from the big-endian array, so the 64 digits a v1 frame carries — and the JWT's own `hex_identity` — hold the wire bytes reversed. Confirmed against the same identity over both protocols. Decode without the reversal and you get a plausible hex string that matches nothing. 3. **`Option` is 0 `Some` / 1 `None`; `Result` is 0 `Ok` / 1 `Err`.** The same ordinals v1's positional sums use, so `[0, v]` / `[1, {}]` on JSON and tag `0` / `1` on BSATN say the same thing. 4. **A client frame with no compression byte is accepted.** The envelope asymmetry is the protocol, not something to work around: the server reads the sum tag first on the frames it receives, and writes a compression byte on the frames it sends. Each answer is recorded as a comment at the affected decode site. ### Conformance and CI All four suites pass end to end — the full six-phase smoke over **v1 native, v2 native, v2 HTML5 and v1 HTML5**. The HTML5 pair is what proves binary frames survive emscripten; keeping the v1 pair is what shows a second protocol cost the first nothing. Unit suite: 11 passed, 0 failed. CI runs v2 **by dispatch only** — a `workflow_dispatch` protocol input that defaults to v1. The automatic push and pull-request path filter stays v1-only, because a second tier doubles run time for a protocol nothing defaults to yet. ### Divergences from v1, worth having written down - v2 has no `OutOfEnergy` outcome; it arrives as `InternalError` text. - The legacy `subscribe` raises on v2 — use `subscribe_query`. - `light` is v1-only; the v2 schema has no light mode. - A table needs generated `types` to be decodable at all: nothing on the binary wire names a column. - A reducer's `ret_value` stays raw BSATN (see the follow-up below). ### Follow-ups, not part of this milestone The three below were open when this milestone closed and **remain open** — none was picked up during the review round. Two more were added by it. - [ ] **Flip the default to v2.** A separate decision, to be taken once v2 conformance has been green for a while. Until then v1 stays what `connect` negotiates when you say nothing. - [ ] **Wire v2 into the automatic CI triggers**, with that flip and not before — the per-PR path filter is v1-only today for run time. - [ ] **Emit reducer `returns` from the schema generator**, so a `ReducerResult`'s `ret_value` can be decoded to a typed value instead of handed back as raw BSATN. Procedures already carry `returns`; reducers do not, which is the only reason that value is untyped. - [ ] **Emit the `event = true` marker for event tables from the schema generator.** Verified against `dev/test/fixtures/chat_schema.lua` as of this comment: `dev/server/index.js:183` declares `ping` with `{ name: 'ping', public: true, event: true }`, and the generated fixture entry for `ping` (lines 113-116) carries only `columns` and `types` — the marker is dropped. Views are not: the generator does emit `view = true` for `message_flip` and `message_latest`, so this is an omission in one code path rather than a missing capability. The SDK works without it because an event table's transience is carried by the wire (`TableUpdateRows::EventTable`) rather than by the schema, so nothing is broken today — but a consumer reading the generated schema cannot tell an event table from a pk-less persistent one. - [ ] **Re-measure apply latency on a quiesced machine** and replace the caveated absolutes in the Results section above with real ones. Both wires, `STDB_BULK_RUNS=5`, the six-bucket sweep, with `/proc/loadavg` recorded alongside the figures so the next reader can tell a regression from a busy box. The attempt during the review round was abandoned rather than published: the machine was at loadavg 8.7-10.0 on 16 cores throughout (a Wine game and Steam, ~4 cores of it), which is the same class of noise that produced the numbers this correction fixes. Publishing a second set of load-contaminated figures would have re-created the problem, not solved it.
yuris changed title from BSATN binary protocol support to BSATN codec — the only route to the v2 message schema 2026-08-19 07:37:34 +00:00
yuris commented 2026-08-19 14:59:06 +00:00
Author
Owner
Copy link

Adding a source-verified constraint for the BSATN branch of this decision, and closing off one suspicion about the current JSON transport.

Investigated clockworklabs/SpacetimeDB at tag v2.8.2.

JSON is never compressed — the SDK needs no decoder today

The compression query parameter defaults to Brotli when omitted (crates/client-api-messages/src/websocket/common.rs:36-45, #[default] on the Brotli variant; parsed with #[serde(default)] at routes/subscribe.rs:86-91). That looked like a latent hazard for a pure-Lua client that cannot inflate anything, and the SDK does not send ?compression=None.

It is inert. The deciding function is serialize in crates/core/src/client/messages.rs:160-194, which branches on format:

ws_v1::FormatSwitch::Json(msg) => {
    serde_json::to_writer(out, &SerializeWrapper::new(msg))...;
    let (in_use, out) = buffer.uncompressed();          // never compressed
    ...
}
ws_v1::FormatSwitch::Bsatn(msg) => {
    ...
    let (in_use, msg_bytes) = finalize_binary_serialize_buffer(buffer, srv_msg_len, config.compression);
}

The JSON arm never reaches a compress call, and SerializeBuffer::new allocates zero compressed capacity for text protocol (messages.rs:52-57: ... || config.protocol == Protocol::Text { 0 }). The per-QueryUpdate hook agrees — JsonFormat::into_query_update takes the compression argument as _ and returns the update untouched (subscription/websocket_building.rs:52-57), while BsatnFormat's impl compresses.

So: omitting compression is safe, sending ?compression=None is unnecessary, and no Brotli/Gzip work is needed to stay on v1-over-JSON. One less reason to move.

If BSATN happens, the framing gains a leading tag byte

This belongs on the checklist, because it is easy to miss and produces garbage rather than an error:

  • Binary server messages are [1-byte tag][payload], tag 0 = none, 1 = brotli, 2 = gzip (common.rs:47-54, written by write_with_tag / compress_with_tag at messages.rs:75-99).
  • Compression only kicks in above 1024 bytes — decide_compression returns None at or below COMPRESS_THRESHOLD (websocket_building.rs:200-210).

That threshold is the trap. A BSATN implementation tested on small fixtures sees tag 0 every time and looks finished; it then breaks in production on the first row batch that crosses 1 KiB. So a BSATN codec must ship both decoders, or explicitly send ?compression=None and rely on that — worth deciding rather than defaulting into.

Two amendments to the checkbox list, if this ever goes ahead:

  • Handle the leading compression tag byte, with fixtures above 1024 bytes so the brotli path is actually exercised
  • Decide explicitly between shipping brotli+gzip decoders and pinning ?compression=None

Also worth noting for the benchmark checkbox: the "JSON rows are roughly 2x the bytes" estimate compares uncompressed JSON against compressed BSATN. Since JSON is never compressed and BSATN is compressed above 1 KiB, the real gap on large row batches is wider than 2x — and on small frames, where both are uncompressed, narrower. A benchmark should measure both sides of that threshold rather than reporting one ratio.

Adding a source-verified constraint for the BSATN branch of this decision, and closing off one suspicion about the current JSON transport. Investigated `clockworklabs/SpacetimeDB` at tag **v2.8.2**. ## JSON is never compressed — the SDK needs no decoder today The `compression` query parameter defaults to **Brotli** when omitted (`crates/client-api-messages/src/websocket/common.rs:36-45`, `#[default]` on the `Brotli` variant; parsed with `#[serde(default)]` at `routes/subscribe.rs:86-91`). That looked like a latent hazard for a pure-Lua client that cannot inflate anything, and the SDK does not send `?compression=None`. It is inert. The deciding function is `serialize` in `crates/core/src/client/messages.rs:160-194`, which branches on format: ```rust ws_v1::FormatSwitch::Json(msg) => { serde_json::to_writer(out, &SerializeWrapper::new(msg))...; let (in_use, out) = buffer.uncompressed(); // never compressed ... } ws_v1::FormatSwitch::Bsatn(msg) => { ... let (in_use, msg_bytes) = finalize_binary_serialize_buffer(buffer, srv_msg_len, config.compression); } ``` The JSON arm never reaches a compress call, and `SerializeBuffer::new` allocates **zero** compressed capacity for text protocol (`messages.rs:52-57`: `... || config.protocol == Protocol::Text { 0 }`). The per-`QueryUpdate` hook agrees — `JsonFormat::into_query_update` takes the compression argument as `_` and returns the update untouched (`subscription/websocket_building.rs:52-57`), while `BsatnFormat`'s impl compresses. So: omitting `compression` is safe, sending `?compression=None` is unnecessary, and no Brotli/Gzip work is needed to stay on v1-over-JSON. One less reason to move. ## If BSATN happens, the framing gains a leading tag byte This belongs on the checklist, because it is easy to miss and produces garbage rather than an error: - Binary server messages are **`[1-byte tag][payload]`**, tag `0` = none, `1` = brotli, `2` = gzip (`common.rs:47-54`, written by `write_with_tag` / `compress_with_tag` at `messages.rs:75-99`). - Compression only kicks in **above 1024 bytes** — `decide_compression` returns `None` at or below `COMPRESS_THRESHOLD` (`websocket_building.rs:200-210`). That threshold is the trap. A BSATN implementation tested on small fixtures sees tag `0` every time and looks finished; it then breaks in production on the first row batch that crosses 1 KiB. So a BSATN codec must ship **both decoders**, or explicitly send `?compression=None` and rely on that — worth deciding rather than defaulting into. Two amendments to the checkbox list, if this ever goes ahead: - [ ] Handle the leading compression tag byte, with fixtures **above** 1024 bytes so the brotli path is actually exercised - [ ] Decide explicitly between shipping brotli+gzip decoders and pinning `?compression=None` Also worth noting for the benchmark checkbox: the "JSON rows are roughly 2x the bytes" estimate compares *uncompressed* JSON against *compressed* BSATN. Since JSON is never compressed and BSATN is compressed above 1 KiB, the real gap on large row batches is wider than 2x — and on small frames, where both are uncompressed, narrower. A benchmark should measure both sides of that threshold rather than reporting one ratio.
yuris commented 2026-08-19 15:43:56 +00:00
Author
Owner
Copy link

Cost evidence for the BSATN decision, from the peer community SDK for Godot (flametime/Godot-SpacetimeDB-SDK). It made the opposite call — v3.bsatn.spacetimedb from the start — so it is the closest thing to a measurement of what this checkbox costs in a scripting language.

The codec is ~1.3k lines

  • core/bsatn_serializer.gd — 546 lines
  • core/bsatn_deserializer.gd — 795 lines
  • core/data_decompressor.gd — 35 lines
  • plus ~25 small wrapper classes for the message shapes

Built on Godot's native StreamPeerBuffer for little-endian byte primitives. So the honest estimate for a Lua port is "about that, plus whatever byte-packing costs without engine primitives" — which is the one real unknown below.

Unresolved prerequisite: string.pack

Lua 5.1 has no string.pack/string.unpack; those arrived in 5.3. Defold backports some later features selectively, and I have not verified whether it ships these. This matters a lot:

  • If present: byte primitives are close to Godot's StreamPeerBuffer and the estimate above roughly holds.
  • If absent: every integer width needs hand-rolled string.byte/string.char arithmetic in pure Lua, on the main thread, per field, per row. That is both more code and materially slower than the JSON path it is meant to beat — which would invert the entire rationale.

Verify this before any other BSATN work. It is a five-line probe in a Defold build and it decides whether this issue is viable.

Details the Godot implementation had to handle

Useful as a checklist, since each one is a place a naive port breaks:

  • Compression tag byte: raw_bytes[0] — 0=none, 1=brotli, 2=gzip. It errors out on brotli and supports gzip via native StreamPeerGZIP. Defold has an existing zlib extension for gzip; punting on brotli is evidently an acceptable position for a community SDK, not a shortcut. Recall the threshold from the earlier note: compression only engages above 1024 bytes, so small-fixture testing will show tag 0 every time and look finished.
  • BsatnRowList row-size hints: tag 0 = FixedSize (u16 row_size + u32 data_len + packed rows), tag 1 = RowOffsets (u32 count + u64 offsets + u32 data_len). Two distinct parse paths, both required.
  • Byte reversal: identity, connection_id, u128 and uuid arrive reversed and are flipped client-side.
  • Delimiter-less framing: messages are parsed back-to-back from the buffer. Their partial-message handling is shallow — a truncated message is silently dropped — which is a bug worth not copying.
  • Schema-driven decode tables: each generated row class carries a column → BSATN-type-string map, with opt_/vec_ prefixes for nesting, and the deserializer recurses on those prefixes. This is a ready-made design for the Lua equivalent, and it is exactly the output #25 would need to emit.
  • Per-row skip-on-error plus size guards (4 MiB string, 128 KiB vec, 16 MiB bytearray). Good hardening precedent.

Sequencing

#25 (typed codegen) is a hard prerequisite: a wire codec cannot exist without column types, and #25 is on the 1.0 path anyway for reasons that have nothing to do with BSATN. So the useful order is #25 → string.pack probe → benchmark → decide. The first checkbox on this issue (benchmark JSON vs BSATN) should not be attempted before the probe, since a negative probe result makes the benchmark moot.

Also worth weighing against the move, from the earlier compression finding: v1-over-JSON receives richer non-caller transaction metadata than v2/v3 do. Going binary trades that away.

Cost evidence for the BSATN decision, from the peer community SDK for Godot ([flametime/Godot-SpacetimeDB-SDK](https://github.com/flametime/Godot-SpacetimeDB-SDK)). It made the opposite call — `v3.bsatn.spacetimedb` from the start — so it is the closest thing to a measurement of what this checkbox costs in a scripting language. ## The codec is ~1.3k lines - `core/bsatn_serializer.gd` — 546 lines - `core/bsatn_deserializer.gd` — 795 lines - `core/data_decompressor.gd` — 35 lines - plus ~25 small wrapper classes for the message shapes Built on Godot's native `StreamPeerBuffer` for little-endian byte primitives. So the honest estimate for a Lua port is "about that, plus whatever byte-packing costs without engine primitives" — which is the one real unknown below. ## Unresolved prerequisite: `string.pack` Lua 5.1 has no `string.pack`/`string.unpack`; those arrived in 5.3. Defold backports some later features selectively, and **I have not verified whether it ships these**. This matters a lot: - If present: byte primitives are close to Godot's `StreamPeerBuffer` and the estimate above roughly holds. - If absent: every integer width needs hand-rolled `string.byte`/`string.char` arithmetic in pure Lua, on the main thread, per field, per row. That is both more code and materially slower than the JSON path it is meant to beat — which would invert the entire rationale. **Verify this before any other BSATN work.** It is a five-line probe in a Defold build and it decides whether this issue is viable. ## Details the Godot implementation had to handle Useful as a checklist, since each one is a place a naive port breaks: - **Compression tag byte**: `raw_bytes[0]` — 0=none, 1=brotli, 2=gzip. It **errors out on brotli** and supports gzip via native `StreamPeerGZIP`. Defold has an existing zlib extension for gzip; punting on brotli is evidently an acceptable position for a community SDK, not a shortcut. Recall the threshold from the earlier note: compression only engages above 1024 bytes, so small-fixture testing will show tag 0 every time and look finished. - **`BsatnRowList` row-size hints**: tag 0 = FixedSize (u16 row_size + u32 data_len + packed rows), tag 1 = RowOffsets (u32 count + u64 offsets + u32 data_len). Two distinct parse paths, both required. - **Byte reversal**: identity, connection_id, u128 and uuid arrive reversed and are flipped client-side. - **Delimiter-less framing**: messages are parsed back-to-back from the buffer. Their partial-message handling is shallow — a truncated message is silently dropped — which is a bug worth not copying. - **Schema-driven decode tables**: each generated row class carries a column → BSATN-type-string map, with `opt_`/`vec_` prefixes for nesting, and the deserializer recurses on those prefixes. This is a ready-made design for the Lua equivalent, and it is exactly the output #25 would need to emit. - **Per-row skip-on-error plus size guards** (4 MiB string, 128 KiB vec, 16 MiB bytearray). Good hardening precedent. ## Sequencing #25 (typed codegen) is a hard prerequisite: a wire codec cannot exist without column types, and #25 is on the 1.0 path anyway for reasons that have nothing to do with BSATN. So the useful order is #25 → `string.pack` probe → benchmark → decide. The first checkbox on this issue (benchmark JSON vs BSATN) should not be attempted before the probe, since a negative probe result makes the benchmark moot. Also worth weighing against the move, from the earlier compression finding: v1-over-JSON receives **richer** non-caller transaction metadata than v2/v3 do. Going binary trades that away.
yuris commented 2026-08-20 14:21:04 +00:00
Author
Owner
Copy link

The JSON-vs-BSATN benchmark this issue's first checkbox asks for

Collected while measuring #30 (see #66), so it cost nothing extra. It argues
against doing this work.

json.decode's share of the frame that applies a first subscription, measured
over a sweep from 100 to 100,000 rows, 6 runs per point, against live pinned
2.8.2:

  • Native: 40–49%, mean ~45%.
  • WASM/HTML5: 20–25%, mean ~23%.

Full sweep, native, medians (rows → frame ms / decode ms / construction ms):

   100 ->   0.5 /   0.1  /   0.35
  1000 ->   2.7 /   1.25 /   1.5
  5000 ->  13.0 /   6.05 /   7.25
 10000 ->  26.6 /  12.8  /  14.05
 25000 ->  74.7 /  32.6  /  41.75
 50000 -> 155.2 /  66.6  /  86.95
100000 -> 327.5 / 147.5  / 182.4

What that implies for a native decoder

Amdahl ceiling on replacing json.decode with a C++/native BSATN decoder,
assuming the decoder were infinitely fast:

  • native: 1/(1 - 0.45) = 1.8×
  • wasm: 1/(1 - 0.23) = 1.3×

In #30's terms, 1.8× moves the native 100ms threshold from ~32,000 rows to
~58,000. It postpones the hitch; it does not remove it. And on the web tier —
where the frame budget is tightest, since an idle wasm frame already costs 18ms
to requestAnimationFrame — a 1.3× ceiling is close to irrelevant.

The remaining ~55% native / ~77% wasm is protocol.quote_big_ints' gsub,
building the row tables, pk indexing, event-list construction and
tables.dispatch. None of that is touched by swapping the wire codec.

Conclusion

BSATN is not the fix for the frame-budget problem, which was the main practical
argument for it. #30's answer is time-slicing, and that works regardless of codec.

Staying closed for v1.1.0. Beyond the weak payoff, it re-opens protocol surface
one release after freezing the public API at v1.0.0, and the benchmark that was
this issue's own precondition now points the other way.

Worth noting the honest caveat: this measures decode of the JSON the server
already sends. It does not measure a hypothetical BSATN path end to end, so it
bounds the upper limit of the win rather than predicting the actual one. The
bound is what matters here, because the bound is already too small.

## The JSON-vs-BSATN benchmark this issue's first checkbox asks for Collected while measuring #30 (see #66), so it cost nothing extra. It argues **against** doing this work. `json.decode`'s share of the frame that applies a first subscription, measured over a sweep from 100 to 100,000 rows, 6 runs per point, against live pinned 2.8.2: - **Native: 40–49%, mean ~45%.** - **WASM/HTML5: 20–25%, mean ~23%.** Full sweep, native, medians (rows → frame ms / decode ms / construction ms): ``` 100 -> 0.5 / 0.1 / 0.35 1000 -> 2.7 / 1.25 / 1.5 5000 -> 13.0 / 6.05 / 7.25 10000 -> 26.6 / 12.8 / 14.05 25000 -> 74.7 / 32.6 / 41.75 50000 -> 155.2 / 66.6 / 86.95 100000 -> 327.5 / 147.5 / 182.4 ``` ## What that implies for a native decoder Amdahl ceiling on replacing `json.decode` with a C++/native BSATN decoder, assuming the decoder were **infinitely fast**: - native: `1/(1 - 0.45)` = **1.8×** - wasm: `1/(1 - 0.23)` = **1.3×** In #30's terms, 1.8× moves the native 100ms threshold from ~32,000 rows to ~58,000. It postpones the hitch; it does not remove it. And on the web tier — where the frame budget is tightest, since an idle wasm frame already costs 18ms to `requestAnimationFrame` — a 1.3× ceiling is close to irrelevant. The remaining ~55% native / ~77% wasm is `protocol.quote_big_ints`' gsub, building the row tables, pk indexing, event-list construction and `tables.dispatch`. None of that is touched by swapping the wire codec. ## Conclusion BSATN is not the fix for the frame-budget problem, which was the main practical argument for it. #30's answer is time-slicing, and that works regardless of codec. Staying closed for v1.1.0. Beyond the weak payoff, it re-opens protocol surface one release after freezing the public API at v1.0.0, and the benchmark that was this issue's own precondition now points the other way. Worth noting the honest caveat: this measures decode of the JSON the server already sends. It does not measure a hypothetical BSATN path end to end, so it bounds the *upper* limit of the win rather than predicting the actual one. The bound is what matters here, because the bound is already too small.
yuris commented 2026-08-20 15:10:47 +00:00
Author
Owner
Copy link

Partial retraction: the ceiling above is too low

Review of #66 found the measurement it rests on stops one function short, so the
1.8x native / 1.3x wasm figure understates the case for BSATN. Flagging it now
rather than leaving a wrong number standing; corrected figures follow when the
re-measurement lands.

The instrumentation timed json.decode only. protocol.quote_big_ints fell into
the bucket I then called untouchable — and it is not untouchable. Measured:
12.3ms of a 30.1ms frame at 10k rows, 75.3ms of 165.5ms at 50k — as much as
json.decode itself, 41–45% of the frame, because it runs twice over the same
bytes (once over the whole frame text, then once per row).

It exists only because JSON cannot carry u128 exactly.
stdb/protocol.lua:711-714 documents it as a raw-text rescue. BSATN carries u128
natively, so a decoder swap deletes that cost outright rather than reducing it.

Genuinely untouchable work — building row tables, pk indexing, event-list
construction, dispatch — is therefore around 11% native and 13% wasm, not the
55%/77% I claimed. That puts the real ceiling somewhere near 2.5–6x rather
than 1.8x/1.3x.

What that does and does not change:

  • The conclusion for #30 is unaffected. Time-slicing is still the fix there,
    because it works regardless of codec and because even a 6x speedup leaves a
    10k-row apply costing several frames' worth of work in one frame.
  • The argument for keeping #9 closed is materially weaker than I posted. I
    closed it partly on "the main practical argument for BSATN does not survive the
    benchmark". At 2.5–6x that argument does survive, and the honest position is
    that BSATN is a real performance win whose cost is re-opening protocol surface
    one release after freezing the API — a tradeoff, not a settled no.

Leaving #9 closed for v1.1.0 stands on the API-freeze timing alone, which is a
scheduling judgement rather than a claim about the numbers. I will post the
corrected three-bucket measurements here once #66's re-run is done, so the next
person deciding this has the real figures.

Apologies for the noise; better than leaving the wrong number as the record.

## Partial retraction: the ceiling above is too low Review of #66 found the measurement it rests on stops one function short, so the `1.8x native / 1.3x wasm` figure understates the case for BSATN. Flagging it now rather than leaving a wrong number standing; corrected figures follow when the re-measurement lands. The instrumentation timed `json.decode` only. `protocol.quote_big_ints` fell into the bucket I then called untouchable — and it is **not** untouchable. Measured: **12.3ms of a 30.1ms frame at 10k rows, 75.3ms of 165.5ms at 50k** — as much as `json.decode` itself, 41–45% of the frame, because it runs twice over the same bytes (once over the whole frame text, then once per row). It exists *only* because JSON cannot carry u128 exactly. `stdb/protocol.lua:711-714` documents it as a raw-text rescue. BSATN carries u128 natively, so a decoder swap deletes that cost outright rather than reducing it. Genuinely untouchable work — building row tables, pk indexing, event-list construction, dispatch — is therefore around 11% native and 13% wasm, not the 55%/77% I claimed. That puts the real ceiling somewhere near **2.5–6x** rather than 1.8x/1.3x. What that does and does not change: - **The conclusion for #30 is unaffected.** Time-slicing is still the fix there, because it works regardless of codec and because even a 6x speedup leaves a 10k-row apply costing several frames' worth of work in one frame. - **The argument for keeping #9 closed is materially weaker than I posted.** I closed it partly on "the main practical argument for BSATN does not survive the benchmark". At 2.5–6x that argument does survive, and the honest position is that BSATN is a real performance win whose cost is re-opening protocol surface one release after freezing the API — a tradeoff, not a settled no. Leaving #9 closed for v1.1.0 stands on the API-freeze timing alone, which is a scheduling judgement rather than a claim about the numbers. I will post the corrected three-bucket measurements here once #66's re-run is done, so the next person deciding this has the real figures. Apologies for the noise; better than leaving the wrong number as the record.
yuris commented 2026-08-20 15:59:24 +00:00
Author
Owner
Copy link

Corrected measurements, as promised

Re-measured with a five-bucket split that sums to the frame exactly
(pre / json.decode / quote_big_ints / build / post), medians over 44
samples at 10k rows, native, against live pinned 2.8.2. Full detail in #66.

rows    | frame  | pre | decode | quote | build | post |  dec/quo/bld
100     |   0.6  | 0.1 |   0.1  |  0.1  |  0.1  | 0.0  |  17/17/17 %
1 000   |   3.2  | 0.2 |   1.3  |  1.2  |  0.4  | 0.1  |  41/38/13 %
5 000   |  15.3  | 0.4 |   6.9  |  6.3  |  2.0  | 0.1  |  45/41/13 %
10 000  |  31.0  | 0.6 |  14.8  | 12.6  |  3.6  | 0.1  |  47/40/12 %
25 000  |  83.9  | 1.5 |  38.2  | 35.8  |  8.9  | 0.3  |  46/43/11 %
50 000  | 174.3  | 2.9 |  76.6  | 77.5  | 19.5  | 0.5  |  43/44/11 %
100 000 | 407.9  | 5.4 | 183.6  | 177.7 | 46.0  | 0.9  |  45/44/11 %

WASM at 10k rows, 102ms total: rAF floor ~17ms (17%), websocket receipt/engine
~28ms (27%), json.decode 23ms (23%), quoting 20ms (20%), construction 14ms
(14%), dispatch 1ms.

The corrected ceiling

What a codec swap deletes is decode plus quote — quote_big_ints exists
only because JSON cannot carry a u128 exactly, and BSATN carries one natively.
What it cannot touch is build, post, and (on web) the rAF floor and socket
delivery.

Native: untouchable is 11.3–11.7% net of measurement apparatus →
ceiling ≈8.5× (7.4× raw, per-sample range 5.1–9.5×).

WASM: untouchable is 58% of the frame, because the floor and socket receipt
are untouchable too → frame ceiling only ≈1.7×. Measured on the SDK's apply
alone, excluding engine overhead, it is 4.1×.

So my original 1.8×/1.3× understated native by roughly 4.5× and wasm by 1.3×.
That figure was simply Amdahl on json.decode's share alone, which is the error.
My interim estimate of 2.5–6× was also wrong: too low for native, too high for a
wasm frame.

These are ceilings if decode and quoting go to zero, not predictions. A BSATN
decoder still parses bytes. The number bounds what is available; it does not
promise it.

But the practical conclusion has moved somewhere else entirely

The re-measurement turned up something better than either answer, now filed as
#69: quote_big_ints has no guard, while its sibling quote_wide_ints
has one and short-circuits. On a table with no 128-bit column the hand-written
path pays a full gsub over the whole frame and over every row to rewrite
nothing — 40–44% of the frame, and at 50k rows more than the JSON parser itself.

The evidence is already in the typed-path comparison: going typed halves the
quote bucket (12.2 → 6.4ms at 10k) purely because the guarded function replaces
the unguarded one.

So most of what BSATN would be credited with on this axis is available from one
necessary-condition guard, with no protocol surface re-opened and no API change.
That is the work to do first, and it makes the honest case for #9 weaker again —
not because BSATN is slow, but because the cheap fix takes the same ground.

Where that leaves this issue

Still closed for v1.1.0, and now on firmer ground than when I closed it:

  1. #69 takes a large share of the win for a fraction of the cost.
  2. #30's answer is time-slicing, which is codec-independent — even an 8.5×
    decoder leaves a 10k-row apply spending several frames of work in one frame.
  3. Re-opening protocol surface one release after freezing the public API remains
    the real cost, and it is unchanged by any of these numbers.

If #69 lands and the remaining decode share still hurts at a row count real games
hit, that is the moment to reconsider — with the buckets already instrumented to
prove it.

Method note: us/row is not constant. Native is linear within 1% to 10,000 rows,
then drifts +30–45% by 100,000 with sample spread reaching 31%, so a single
per-row figure stops describing the behaviour past ~25,000 rows.

## Corrected measurements, as promised Re-measured with a five-bucket split that sums to the frame exactly (`pre` / `json.decode` / `quote_big_ints` / `build` / `post`), medians over 44 samples at 10k rows, native, against live pinned 2.8.2. Full detail in #66. ``` rows | frame | pre | decode | quote | build | post | dec/quo/bld 100 | 0.6 | 0.1 | 0.1 | 0.1 | 0.1 | 0.0 | 17/17/17 % 1 000 | 3.2 | 0.2 | 1.3 | 1.2 | 0.4 | 0.1 | 41/38/13 % 5 000 | 15.3 | 0.4 | 6.9 | 6.3 | 2.0 | 0.1 | 45/41/13 % 10 000 | 31.0 | 0.6 | 14.8 | 12.6 | 3.6 | 0.1 | 47/40/12 % 25 000 | 83.9 | 1.5 | 38.2 | 35.8 | 8.9 | 0.3 | 46/43/11 % 50 000 | 174.3 | 2.9 | 76.6 | 77.5 | 19.5 | 0.5 | 43/44/11 % 100 000 | 407.9 | 5.4 | 183.6 | 177.7 | 46.0 | 0.9 | 45/44/11 % ``` WASM at 10k rows, 102ms total: rAF floor ~17ms (17%), websocket receipt/engine ~28ms (27%), `json.decode` 23ms (23%), quoting 20ms (20%), construction 14ms (14%), dispatch 1ms. ## The corrected ceiling What a codec swap deletes is `decode` **plus** `quote` — `quote_big_ints` exists only because JSON cannot carry a u128 exactly, and BSATN carries one natively. What it cannot touch is `build`, `post`, and (on web) the rAF floor and socket delivery. **Native:** untouchable is 11.3–11.7% net of measurement apparatus → ceiling **≈8.5×** (7.4× raw, per-sample range 5.1–9.5×). **WASM:** untouchable is 58% of the frame, because the floor and socket receipt are untouchable too → frame ceiling only **≈1.7×**. Measured on the SDK's apply alone, excluding engine overhead, it is 4.1×. So my original 1.8×/1.3× understated native by roughly 4.5× and wasm by 1.3×. That figure was simply Amdahl on `json.decode`'s share alone, which is the error. My interim estimate of 2.5–6× was also wrong: too low for native, too high for a wasm frame. **These are ceilings if decode and quoting go to zero, not predictions.** A BSATN decoder still parses bytes. The number bounds what is available; it does not promise it. ## But the practical conclusion has moved somewhere else entirely The re-measurement turned up something better than either answer, now filed as **#69**: `quote_big_ints` has **no guard**, while its sibling `quote_wide_ints` has one and short-circuits. On a table with no 128-bit column the hand-written path pays a full `gsub` over the whole frame *and* over every row to rewrite nothing — 40–44% of the frame, and at 50k rows *more than the JSON parser itself*. The evidence is already in the typed-path comparison: going typed halves the quote bucket (12.2 → 6.4ms at 10k) purely because the guarded function replaces the unguarded one. So most of what BSATN would be credited with on this axis is available from **one necessary-condition guard**, with no protocol surface re-opened and no API change. That is the work to do first, and it makes the honest case for #9 weaker again — not because BSATN is slow, but because the cheap fix takes the same ground. ## Where that leaves this issue Still closed for v1.1.0, and now on firmer ground than when I closed it: 1. #69 takes a large share of the win for a fraction of the cost. 2. #30's answer is time-slicing, which is codec-independent — even an 8.5× decoder leaves a 10k-row apply spending several frames of work in one frame. 3. Re-opening protocol surface one release after freezing the public API remains the real cost, and it is unchanged by any of these numbers. If #69 lands and the remaining decode share still hurts at a row count real games hit, that is the moment to reconsider — with the buckets already instrumented to prove it. Method note: `us/row` is not constant. Native is linear within 1% to 10,000 rows, then drifts +30–45% by 100,000 with sample spread reaching 31%, so a single per-row figure stops describing the behaviour past ~25,000 rows.
yuris closed this issue 2026-08-21 05:32:11 +00:00
yuris commented 2026-08-21 08:00:30 +00:00
Author
Owner
Copy link

All five phases are merged to main as of 2026-08-21:

PR Squashed as What
#82 cccfd7a stdb/bsatn.lua — pure-arithmetic BSATN codec
#83 91e3de1 stdb/protocol_v2.lua — v2 message schema
#84 4dd965b connect({ protocol = "v2" }), opt-in
#85 5da10db v2 conformance, capture, event table
#86 9f19af4 README

Review round. Four reviews (two code, one harness, one security) ran against the stack before merge, then a second adversarial pass re-tested every finding with crafted bytes rather than accepting the fix commits. What they caught:

  • Blocker: unvalidated RowOffsets gave quadratic memory amplification — a measured 232 KB frame producing 388 MB of Lua heap (model frame²/64, ~15 GB at 1 MB). Overlapping offsets made every row a near-copy of the buffer. Closed by validating the offsets as a non-descending in-bounds partition while the array is read, which also keeps an offset past 2^53 away from string.sub's undefined double→ptrdiff_t cast — that one returned the whole buffer on x86-64 while arm64 and wasm32 saturate the other way, so identical bytes would have decoded differently per platform. Re-proven closed: original PoC now rejected with a 2 KB heap delta, and 4000 randomised offset lists showed a worst decoded/rows_data ratio of exactly 1.000.
  • A NaN in a pk or unique column raised inside the cache's write pass, after that batch's deletes were committed — cache corrupt, counts lost, callback never settling. Moved into the pure pass; the relation is now skipped and reported with the column named. v2 is the first path that can produce a NaN; JSON has no NaN literal.
  • An unbounded array count turned 4 wire bytes into 2,000,000 tables for a zero-width element type. Counts are now bounded by the bytes remaining.
  • Both conformance gates accepted a run that silently fell back to v1, so the earlier "v2 HTML5 passes" claim was unproven at the time it was made. The native gate now requires the protocol-specific verdict line and HTML5 asserts the negotiated Sec-WebSocket-Protocol at every connect including reconnects. Negative controls confirm both now reject a v1-masquerading-as-v2 run that the old gates passed.
  • The committed capture contained no EventTable frame despite a whole smoke phase existing for that decode path, and was read by no test at all. Both fixed; test_fixtures_v2.lua now replays every captured frame.
  • Two guards whose own comments called them load-bearing had no test — both mutations passed green. Now pinned.
  • The README had inverted what v2 separates.

State on main: 11 unit suites green, stylua --check stdb dev/test clean, v1 and v2 native conformance both PASS post-merge (re-run after the final rebase), stdb/protocol.lua byte-identical to its pre-stack blob so the frozen v1 default is provably untouched.

Follow-ups, each a separate decision rather than leftover work:

  • Flip the default to v2 (after v2 conformance stays green for a while).
  • Wire v2 into the automatic CI triggers — deliberately dispatch-only today, since running both wires doubles every PR for a feature with no default consumers. Belongs with the default flip.
  • Emit reducer returns from the generator, so ret_value can be decoded instead of handed back as raw BSATN.
  • Emit event = true for event tables from the generator. Verified missing: dev/server/index.js declares ping with event: true but the generated fixture carries only columns and types, while view = true is emitted for views — an omission in one path, not a missing capability. Nothing breaks today because event-table transience rides the wire (TableUpdateRows::EventTable), not the schema.
  • Re-measure latency on an idle machine. The originally published 20.0/10.5 ms pair was measured with load unrecorded and does not reproduce; see the correction already in this issue and in Protocol-Notes. Wire bytes are unaffected — v2 is ~46% of v1 on row payloads, 31% on client frames.

One thing worth recording for whoever picks up the default flip: Call:on_out_of_energy can never fire on v2. The v2 ReducerOutcome has no OutOfEnergy variant, so energy exhaustion arrives as InternalError text. That is an upstream protocol change, not something the SDK can paper over, and it is a behavioural difference a default flip would impose on every consumer.

All five phases are merged to `main` as of 2026-08-21: | PR | Squashed as | What | |---|---|---| | #82 | `cccfd7a` | `stdb/bsatn.lua` — pure-arithmetic BSATN codec | | #83 | `91e3de1` | `stdb/protocol_v2.lua` — v2 message schema | | #84 | `4dd965b` | `connect({ protocol = "v2" })`, opt-in | | #85 | `5da10db` | v2 conformance, capture, event table | | #86 | `9f19af4` | README | **Review round.** Four reviews (two code, one harness, one security) ran against the stack before merge, then a second adversarial pass re-tested every finding with crafted bytes rather than accepting the fix commits. What they caught: - **Blocker:** unvalidated `RowOffsets` gave quadratic memory amplification — a measured 232 KB frame producing 388 MB of Lua heap (model frame²/64, ~15 GB at 1 MB). Overlapping offsets made every row a near-copy of the buffer. Closed by validating the offsets as a non-descending in-bounds partition while the array is read, which also keeps an offset past 2^53 away from `string.sub`'s undefined double→ptrdiff_t cast — that one returned the whole buffer on x86-64 while arm64 and wasm32 saturate the other way, so identical bytes would have decoded differently per platform. Re-proven closed: original PoC now rejected with a 2 KB heap delta, and 4000 randomised offset lists showed a worst decoded/rows_data ratio of exactly 1.000. - A NaN in a pk or unique column raised inside the cache's *write* pass, after that batch's deletes were committed — cache corrupt, counts lost, callback never settling. Moved into the pure pass; the relation is now skipped and reported with the column named. v2 is the first path that can produce a NaN; JSON has no NaN literal. - An unbounded array count turned 4 wire bytes into 2,000,000 tables for a zero-width element type. Counts are now bounded by the bytes remaining. - **Both conformance gates accepted a run that silently fell back to v1**, so the earlier "v2 HTML5 passes" claim was unproven at the time it was made. The native gate now requires the protocol-specific verdict line and HTML5 asserts the negotiated `Sec-WebSocket-Protocol` at every connect including reconnects. Negative controls confirm both now reject a v1-masquerading-as-v2 run that the old gates passed. - The committed capture contained no EventTable frame despite a whole smoke phase existing for that decode path, and was read by no test at all. Both fixed; `test_fixtures_v2.lua` now replays every captured frame. - Two guards whose own comments called them load-bearing had no test — both mutations passed green. Now pinned. - The README had inverted what v2 separates. **State on `main`:** 11 unit suites green, `stylua --check stdb dev/test` clean, v1 and v2 native conformance both PASS post-merge (re-run after the final rebase), `stdb/protocol.lua` byte-identical to its pre-stack blob so the frozen v1 default is provably untouched. **Follow-ups**, each a separate decision rather than leftover work: - [ ] Flip the default to v2 (after v2 conformance stays green for a while). - [ ] Wire v2 into the automatic CI triggers — deliberately dispatch-only today, since running both wires doubles every PR for a feature with no default consumers. Belongs with the default flip. - [ ] Emit reducer `returns` from the generator, so `ret_value` can be decoded instead of handed back as raw BSATN. - [ ] Emit `event = true` for event tables from the generator. Verified missing: `dev/server/index.js` declares `ping` with `event: true` but the generated fixture carries only `columns` and `types`, while `view = true` *is* emitted for views — an omission in one path, not a missing capability. Nothing breaks today because event-table transience rides the wire (`TableUpdateRows::EventTable`), not the schema. - [ ] Re-measure latency on an idle machine. The originally published 20.0/10.5 ms pair was measured with load unrecorded and does not reproduce; see the correction already in this issue and in Protocol-Notes. Wire bytes are unaffected — v2 is ~46% of v1 on row payloads, 31% on client frames. One thing worth recording for whoever picks up the default flip: `Call:on_out_of_energy` can never fire on v2. The v2 `ReducerOutcome` has no `OutOfEnergy` variant, so energy exhaustion arrives as `InternalError` text. That is an upstream protocol change, not something the SDK can paper over, and it is a behavioural difference a default flip would impose on every consumer.
yuris commented 2026-08-21 09:08:06 +00:00
Author
Owner
Copy link

Default-flip decision: not yet, and here is what changes that

A read-only assessment of the flip (v1 → v2) ran against main b057960. Nothing in the code blocks it. Two things outside the code do, and the assessment found eight divergences the previous list missed — two of which I would not have wanted to learn from a bug report.

Corrected already, in #88

  • f32 decodes to a different double on the two wires. v1 sends decimal text (json.decode → 0.10000000000000001); v2 carries the stored single's four bytes (→ 0.10000000149011612). The repo claimed identical values in five places, including inside this issue's own flip-decision text, where it would have fed a wrong decision. Now pinned two ways and documented. v2 is the more faithful side.
  • confirmed_reads resolves differently per protocol. An absent parameter resolves false on v1 but true on v2 (subscribe.rs:103-111, asserted upstream at :2838-2844), so an opted-in v2 caller today already has updates held until the transaction offset is durable. Documented; no behaviour change.

What blocks the flip

One item, and it is a decision rather than a defect: confirmed_reads. The flip's whole premise is "no consumer code changes", yet it would silently change update latency for every consumer who never touched the option — in a game SDK. Two honest resolutions: pin confirmed=false when the caller said nothing (one line, preserves today's behaviour exactly), or accept the server default and make the latency change a headline changelog item. Either is fine; shipping without picking one is not.

And one prerequisite that is not optional: v2 must be in the automatic CI triggers before it becomes the default, not alongside. Today ci.yml runs stylua + the Lua suite + nix flake check on every PR, which covers v2's pure-Lua half well. The engine tier runs STDB_PROTOCOL: v1 and reaches v2 only by workflow_dispatch. That engine tier is the only coverage for stdb/init.lua — subprotocol negotiation, DATA_TYPE_BINARY frame selection, compression=None, and the emscripten binary path. Flip without wiring it in and the shipped default has zero automatic engine coverage while the opt-out path keeps all of it: exactly inverted.

Grades

  • Blocks: confirmed_reads (undecided, silent).
  • Needs pre-announcement or a shim: light = true raising (the docs demonstrate that exact configuration, and the SDK's own smoke uses it on v1); reducers/procedures becoming mandatory to call anything (loud but currently unannounced, and no shim is possible — BSATN cannot encode without types).
  • Acceptable on a major bump with a changelog line: legacy subscribe raising, the reducer/subscribe callback msg changing shape, hand-written schemas and undecodable column tags degrading to skip-and-report, one-off rows becoming byte strings, raw ret_value.

Two capability losses worth stating plainly

  • on_out_of_energy cannot fire on v2 and there is no substitute signal. The v2 outcome has four variants and no out-of-energy; energy_quanta_used has no v2 counterpart anywhere in the schema. A consumer showing "you're out of energy" cannot do that on v2 by any means the wire provides. Changelog line, not a heuristic on error text.
  • Broadcast transactions lose their attribution — no reducer name, no caller identity, anywhere in the v2 TransactionUpdate. Deserves its own changelog paragraph.

Sequence, each gate observable

  1. Decide confirmed_reads; assert the chosen URL in test_init.lua.
  2. Pin the f32 divergence — done in #88.
  3. Wire v2 into the automatic triggers. Gate: two green automatic v2 runs on unrelated PRs, native and browser, no dispatch.
  4. Make the HTML5 tier gate or record on #81 why a binary default ships without it (it is continue-on-error today).
  5. Bring the v2 smoke to parity — a one-off query, a frame-budget run so slicing over BSATN is proven on an engine, and a v2 stand-in for the light assertion.
  6. Ship 1.2.0 as pre-announcement of the loud breaks.
  7. Flip as 2.0.0, docs in the same commit.

Versioning: this repo versions in game.project (1.1.0), tags v*, and publishes release notes from the tag annotation — release.yml refuses a tag without one. Development.md states an additive-only posture within 1.x. A flip changes observable behaviour for unchanged consumer code, so it is 2.0.0 and the upgrade note lives in the tag annotation.

Follow-up 4 is dropped

event = true from the generator is dead weight, verified: nothing in stdb/ reads such a flag — the v2 event path keys off the wire variant (TableUpdateRows::EventTable), never the schema. The comparison to view = true does not hold, because that flag is read (stdb/tables.lua:146-149, stdb/schema.lua:32-45). The only thing it would buy is a better require_pk diagnostic. Not a generator work item, certainly not a flip prerequisite.

Remaining follow-ups

  • Decide confirmed_reads for v2 (flip gate 1)
  • Wire v2 into the automatic CI triggers (flip gate 3 — prerequisite, not companion)
  • HTML5 tier: gate it, or record the exemption on #81 (flip gate 4)
  • Bring the v2 smoke to parity with the v1 smoke (flip gate 5)
  • Flip the default as 2.0.0 (after gates 1-5)
  • Emit reducer returns from the generator — additive, not a gate. Note the premise was wrong: v1 has no reducer return value at all, so v2 already gives strictly more. Worth pinning whether the schema endpoint even exposes a reducer return type before promising it.
  • Re-measure latency on an idle machine — docs correction, not a gate. The case for v2 rests on the byte count, which is load-independent.
  • Emit event = true for event tables — dropped, see above.

State on main b057960: 11 unit suites green, format gate clean, v1 and v2 native conformance both PASS, stdb/protocol.lua still byte-identical to its pre-stack blob.

## Default-flip decision: not yet, and here is what changes that A read-only assessment of the flip (v1 → v2) ran against `main` `b057960`. **Nothing in the code blocks it.** Two things outside the code do, and the assessment found **eight divergences the previous list missed** — two of which I would not have wanted to learn from a bug report. ### Corrected already, in #88 - **`f32` decodes to a different double on the two wires.** v1 sends decimal text (`json.decode` → `0.10000000000000001`); v2 carries the stored single's four bytes (→ `0.10000000149011612`). The repo claimed identical values in **five** places, including inside this issue's own flip-decision text, where it would have fed a wrong decision. Now pinned two ways and documented. v2 is the more faithful side. - **`confirmed_reads` resolves differently per protocol.** An absent parameter resolves `false` on v1 but `true` on v2 (`subscribe.rs:103-111`, asserted upstream at `:2838-2844`), so an opted-in v2 caller today already has updates held until the transaction offset is durable. Documented; no behaviour change. ### What blocks the flip **One item, and it is a decision rather than a defect: `confirmed_reads`.** The flip's whole premise is "no consumer code changes", yet it would silently change update latency for every consumer who never touched the option — in a game SDK. Two honest resolutions: pin `confirmed=false` when the caller said nothing (one line, preserves today's behaviour exactly), or accept the server default and make the latency change a headline changelog item. Either is fine; shipping without picking one is not. **And one prerequisite that is not optional: v2 must be in the automatic CI triggers *before* it becomes the default, not alongside.** Today `ci.yml` runs stylua + the Lua suite + `nix flake check` on every PR, which covers v2's pure-Lua half well. The engine tier runs `STDB_PROTOCOL: v1` and reaches v2 only by `workflow_dispatch`. That engine tier is the *only* coverage for `stdb/init.lua` — subprotocol negotiation, `DATA_TYPE_BINARY` frame selection, `compression=None`, and the emscripten binary path. Flip without wiring it in and the shipped default has zero automatic engine coverage while the opt-out path keeps all of it: exactly inverted. ### Grades - **Blocks:** `confirmed_reads` (undecided, silent). - **Needs pre-announcement or a shim:** `light = true` raising (the docs demonstrate that exact configuration, and the SDK's own smoke uses it on v1); `reducers`/`procedures` becoming mandatory to call anything (loud but currently unannounced, and no shim is possible — BSATN cannot encode without types). - **Acceptable on a major bump with a changelog line:** legacy `subscribe` raising, the reducer/subscribe callback `msg` changing shape, hand-written schemas and undecodable column tags degrading to skip-and-report, one-off rows becoming byte strings, raw `ret_value`. ### Two capability losses worth stating plainly - **`on_out_of_energy` cannot fire on v2 and there is no substitute signal.** The v2 outcome has four variants and no out-of-energy; `energy_quanta_used` has no v2 counterpart anywhere in the schema. A consumer showing "you're out of energy" cannot do that on v2 by any means the wire provides. Changelog line, not a heuristic on error text. - **Broadcast transactions lose their attribution** — no reducer name, no caller identity, anywhere in the v2 `TransactionUpdate`. Deserves its own changelog paragraph. ### Sequence, each gate observable 1. Decide `confirmed_reads`; assert the chosen URL in `test_init.lua`. 2. ~~Pin the `f32` divergence~~ — done in #88. 3. Wire v2 into the automatic triggers. *Gate:* two green automatic v2 runs on unrelated PRs, native and browser, no dispatch. 4. Make the HTML5 tier gate or record on #81 why a binary default ships without it (it is `continue-on-error` today). 5. Bring the v2 smoke to parity — a one-off query, a frame-budget run so slicing over BSATN is proven on an engine, and a v2 stand-in for the `light` assertion. 6. Ship `1.2.0` as pre-announcement of the loud breaks. 7. Flip as **`2.0.0`**, docs in the same commit. **Versioning:** this repo versions in `game.project` (`1.1.0`), tags `v*`, and publishes release notes from the tag annotation — `release.yml` refuses a tag without one. `Development.md` states an additive-only posture within 1.x. A flip changes observable behaviour for unchanged consumer code, so it is `2.0.0` and the upgrade note lives in the tag annotation. ### Follow-up 4 is dropped `event = true` from the generator is dead weight, verified: nothing in `stdb/` reads such a flag — the v2 event path keys off the wire variant (`TableUpdateRows::EventTable`), never the schema. The comparison to `view = true` does not hold, because that flag *is* read (`stdb/tables.lua:146-149`, `stdb/schema.lua:32-45`). The only thing it would buy is a better `require_pk` diagnostic. Not a generator work item, certainly not a flip prerequisite. ### Remaining follow-ups - [ ] Decide `confirmed_reads` for v2 (**flip gate 1**) - [ ] Wire v2 into the automatic CI triggers (**flip gate 3 — prerequisite, not companion**) - [ ] HTML5 tier: gate it, or record the exemption on #81 (**flip gate 4**) - [ ] Bring the v2 smoke to parity with the v1 smoke (**flip gate 5**) - [ ] Flip the default as `2.0.0` (**after gates 1-5**) - [ ] Emit reducer `returns` from the generator — additive, *not* a gate. Note the premise was wrong: v1 has no reducer return value at all, so v2 already gives strictly more. Worth pinning whether the schema endpoint even exposes a reducer return type before promising it. - [ ] Re-measure latency on an idle machine — docs correction, not a gate. The case for v2 rests on the byte count, which is load-independent. - [x] ~~Emit `event = true` for event tables~~ — dropped, see above. **State on `main` `b057960`:** 11 unit suites green, format gate clean, v1 and v2 native conformance both PASS, `stdb/protocol.lua` still byte-identical to its pre-stack blob.
yuris commented 2026-08-21 11:33:48 +00:00
Author
Owner
Copy link

Default flipped in v2.0.0 — connect() now negotiates v2.bsatn.spacetimedb, and connect({ protocol = "v1" }) is the JSON opt-out, CI-covered on both wires. Tracking issue: #89.

Default flipped in `v2.0.0` — `connect()` now negotiates `v2.bsatn.spacetimedb`, and `connect({ protocol = "v1" })` is the JSON opt-out, CI-covered on both wires. Tracking issue: #89.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
release/v4.0.1
fix/hold-pre-open-frames
v4.0.2
v4.0.1
v4.0.0
v3.1.0
v3.0.0
v2.0.0
v1.1.0
v1.0.0
v0.1.0
Labels
Clear labels   
area:defold
Defold engine integration

  
area:docs
Documentation and examples

  
area:infra
Flake, CI, secretspec

  
area:protocol
Wire protocol conformance

  
area:testing
Test harness and fixtures

  
kind:bug
Something broken

  
kind:chore
Maintenance

  
phase:breadth
UnsubscribeMulti, procedures, BSATN

  
phase:core
Core SDK layers (protocol/cache/connection)

  
phase:hardening
Reconnect, token persistence, errors

  
phase:slice
Vertical slice: first playable end-to-end

No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/spacetimedb-defold#9
No description provided.