protocol_v2: the v2 message schema over BSATN

yuris commented

2026-08-21 05:46:13 +00:00

Owner

Phase 2 of #9. Stacked on #82. Still inert — no connection speaks it yet.

Encodes the five ClientMessage variants, decodes all eight ServerMessage ones, and adds the typed row decode that is the binary twin of protocol.decode_typed. Rows leave a BsatnRowList still encoded so the cache decodes them, which keeps row slicing and the frame budget untouched.

Two places the binary codec cannot mirror the JSON one, both documented at the site:

An unknown type tag raises instead of passing the value through. BSATN carries no widths, so a tag the decoder does not know means it can no longer find the next column.
Identity renders as the byte-reverse of the wire: crates/lib/src/identity.rs stores it little-endian while to_hex encodes the big-endian array.

Verified: 10 passed, 0 failed on this branch alone. The differential test decodes the same all_types row through both codecs and deep-equals — that is the value-surface proof, and it is non-vacuous (20 scalar columns differ between the two rows exercised).

Phase 2 of #9. Stacked on #82. Still inert — no connection speaks it yet. Encodes the five ClientMessage variants, decodes all eight ServerMessage ones, and adds the typed row decode that is the binary twin of `protocol.decode_typed`. Rows leave a `BsatnRowList` still encoded so the cache decodes them, which keeps row slicing and the frame budget untouched. Two places the binary codec cannot mirror the JSON one, both documented at the site: - An unknown type tag raises instead of passing the value through. BSATN carries no widths, so a tag the decoder does not know means it can no longer find the next column. - Identity renders as the byte-reverse of the wire: `crates/lib/src/identity.rs` stores it little-endian while `to_hex` encodes the big-endian array. **Verified:** 10 passed, 0 failed on this branch alone. The differential test decodes the same `all_types` row through both codecs and deep-equals — that is the value-surface proof, and it is non-vacuous (20 scalar columns differ between the two rows exercised).

yuris added 1 commit

2026-08-21 05:46:13 +00:00

protocol_v2: the v2 message schema over BSATN

ci / ci (pull_request) Successful in 1m35s

Details

conformance / conformance (pull_request) Successful in 2m28s

Details

19e5325a6b

Encodes the five ClientMessage variants and decodes all eight
ServerMessage ones, plus the typed row decode that is the binary twin of
protocol.decode_typed. Rows come out of a BsatnRowList still encoded:
the cache decodes them, so row slicing and the frame budget keep working
untouched.

Two places the binary codec cannot mirror the JSON one. An unknown type
tag raises instead of passing the value through, because BSATN carries
no widths and a tag the decoder does not know means it can no longer
find the next column. And identity renders as the byte-reverse of the
wire: crates/lib/src/identity.rs stores it little-endian and to_hex
encodes the big-endian array.

yuris referenced this pull request

2026-08-21 05:46:14 +00:00

stdb: connect({ protocol = "v2" }), opt-in #84

yuris added 1 commit

2026-08-21 06:39:13 +00:00

protocol_v2: validate what the wire claims about itself

ci / ci (pull_request) Successful in 20s

Details

conformance / conformance (pull_request) Successful in 2m44s

Details

b543ee236e

A row-offset list was trusted verbatim. Overlapping offsets make every
row a near-copy of the whole buffer, so decoded bytes grew as the square
of the frame: a 232 KB frame produced 388 MB of Lua heap, and the model
puts 1 MB at ~15 GB. Non-descending and in-bounds makes the rows a
partition of rows_data, which bounds the total by the frame however the
offsets are arranged, and keeps an offset past 2^53 away from
string.sub's undefined double-to-ptrdiff_t cast -- that one returned the
whole buffer on x86-64 while arm64 and wasm32 saturate the other way,
which would have decoded identical bytes to different values per
platform.

Array counts are bounded by the bytes remaining, because a vec of a
zero-width element type turned four wire bytes into two million tables.
A decoded frame must be fully consumed: v2 carries one message per
frame, and cursor exhaustion is the only structural self-check BSATN
admits.

An unknown message ordinal now reports itself instead of raising, so v2
survives what v1 already shrugs off. A tag whose width cannot be known
is classified once per relation through decodable_tag rather than
raising per row, and reducer args gained the wide types -- identity and
the 128/256-bit widths are ordinary parameters that v1 could already
send. A trailing nil Option argument now encodes: # cannot see it, so
the arg count is an upper bound and required-ness is per tag.

yuris referenced this pull request

2026-08-21 06:51:55 +00:00

bsatn: a pure-Lua BSATN primitive codec #82

yuris commented

2026-08-21 06:51:55 +00:00

Author

Owner

Review round: one BLOCKER, fixed, plus six smaller findings

BLOCKER — unvalidated `RowOffsets` gave quadratic memory amplification

BsatnRowList in RowOffsets mode carried a server-supplied array<u64> of row starts that the
decoder used without checking. The offsets were trusted to be ordered and in bounds; they are neither
guaranteed nor were they verified.

Measured, not theorised: 232 KB of frame produced 388 MB of Lua heap. The mechanism is that
overlapping offsets make every row a near-copy of the whole buffer rather than a slice of it, so cost
grows as frame²/64 — which extrapolates to roughly 15 GB at a 1 MB frame. A single hostile or
merely buggy frame was an OOM.

Fixed by enforcing the offsets as a non-descending, in-bounds partition of rows_data. That is
the same validation that closes a second, quieter hole: an offset past 2^53 reaches string.sub as a
double that cannot be represented as a ptrdiff_t, and that cast is undefined. It was not
uniformly wrong — on x86-64 it returned the whole buffer, while arm64 and wasm32 saturate the other
way. Identical bytes would have decoded to different values per platform, which on an HTML5 target is
not a hypothetical tier.

The rest

Array counts are now bounded by remaining bytes. A vec of a zero-width type turned 4 wire
bytes into 2,000,000 Lua tables — the count was read and trusted with nothing to spend it against.
Full-frame consumption is asserted. Cursor exhaustion is the only end-of-message signal BSATN
offers; a frame that decoded and left bytes behind used to pass silently.
A trailing nil Option argument now encodes. It was being lost to Lua array-length semantics,
so the last argument of a reducer call could vanish.
Wide reducer-argument types added, with the wide-type writers they need (see the note on #82 —
they live on this branch).
An unknown sum ordinal reports instead of raising, matching how the SDK already degrades on a
type it cannot handle rather than killing the connection.
M.encode_value deleted — unused, and an exported encoder with no caller is a maintenance
liability, not an API.

Benchmark correction (applies to the numbers this stack published): the 10k apply-latency pair of
20.0ms / 10.5ms and its ratio of 0.53 were measured with the machine's load unrecorded. A re-run of the
same commit measured every native absolute 2-3x higher (v1 median 61.7ms, v2 44.6ms over 5 runs) at
loadavg 11.06 on 16 cores, with the harness flagging 47% spread on two sweeps, and put the ratio at
0.59-0.65, not 0.53. Direction holds, the value does not — #9 and the wiki now state the conditions
instead of the bare digits, and a clean re-measure is an open follow-up. The wire-byte tables are
unaffected and stand as published (v2 ~46% of v1 on row payloads, 31% on client frames): they are byte
counts, not timings.

Verification state at the stack tip (bsatn-phase5-docs): unit suite 11 passed, 0 failed;
all four conformance suites PASS end to end — v1 native, v2 native, v1 HTML5, v2 HTML5 — under the
new protocol-specific gates, whose negative controls were proven non-vacuous; stylua --check stdb dev/test
clean.

## Review round: one BLOCKER, fixed, plus six smaller findings ### BLOCKER — unvalidated `RowOffsets` gave quadratic memory amplification `BsatnRowList` in `RowOffsets` mode carried a server-supplied `array<u64>` of row starts that the decoder used without checking. The offsets were trusted to be ordered and in bounds; they are neither guaranteed nor were they verified. Measured, not theorised: **232 KB of frame produced 388 MB of Lua heap.** The mechanism is that overlapping offsets make every row a near-copy of the whole buffer rather than a slice of it, so cost grows as *frame²/64* — which extrapolates to roughly **15 GB at a 1 MB frame**. A single hostile or merely buggy frame was an OOM. Fixed by enforcing the offsets as a **non-descending, in-bounds partition of `rows_data`**. That is the same validation that closes a second, quieter hole: an offset past 2^53 reaches `string.sub` as a double that cannot be represented as a `ptrdiff_t`, and **that cast is undefined**. It was not uniformly wrong — on x86-64 it returned the whole buffer, while arm64 and wasm32 saturate the other way. Identical bytes would have decoded to different values per platform, which on an HTML5 target is not a hypothetical tier. ### The rest - **Array counts are now bounded by remaining bytes.** A `vec` of a zero-width type turned **4 wire bytes into 2,000,000 Lua tables** — the count was read and trusted with nothing to spend it against. - **Full-frame consumption is asserted.** Cursor exhaustion is the only end-of-message signal BSATN offers; a frame that decoded and left bytes behind used to pass silently. - **A trailing `nil` Option argument now encodes.** It was being lost to Lua array-length semantics, so the last argument of a reducer call could vanish. - **Wide reducer-argument types added**, with the wide-type writers they need (see the note on #82 — they live on this branch). - **An unknown sum ordinal reports instead of raising**, matching how the SDK already degrades on a type it cannot handle rather than killing the connection. - **`M.encode_value` deleted** — unused, and an exported encoder with no caller is a maintenance liability, not an API. **Benchmark correction** (applies to the numbers this stack published): the 10k apply-latency pair of 20.0ms / 10.5ms and its ratio of 0.53 were measured with the machine's load unrecorded. A re-run of the same commit measured every native absolute 2-3x higher (v1 median 61.7ms, v2 44.6ms over 5 runs) at `loadavg` 11.06 on 16 cores, with the harness flagging 47% spread on two sweeps, and put the ratio at **0.59-0.65, not 0.53**. Direction holds, the value does not — #9 and the wiki now state the conditions instead of the bare digits, and a clean re-measure is an open follow-up. The **wire-byte tables are unaffected** and stand as published (v2 ~46% of v1 on row payloads, 31% on client frames): they are byte counts, not timings. **Verification state at the stack tip** (`bsatn-phase5-docs`): unit suite **11 passed, 0 failed**; all **four conformance suites PASS** end to end — v1 native, v2 native, v1 HTML5, v2 HTML5 — under the new protocol-specific gates, whose negative controls were proven non-vacuous; `stylua --check stdb dev/test` clean.

yuris added 1 commit

2026-08-21 07:56:24 +00:00

Call a vec of a zero-width element undecodable

ci / ci (pull_request) Successful in 20s

Details

conformance / conformance (pull_request) Has been cancelled

Details

bd2643c57e

Reader:len bounds an array count by the bytes remaining, which is sound
because a real element costs at least one byte, and which deliberately
rejects a vec whose element type is ZERO-WIDTH -- nothing bounds the
count there. decodable_tag disagreed: it resolved vec_<inner> by
recursing on inner alone, and `unit` is a scalar, so vec_unit and
vec_product_Empty both answered true. A table with such a column was
therefore not skipped by the per-relation check, and the raise escaped
through cache.decode_rows once per frame -- the exact failure mode
decodable_tag exists to keep out of the cache. An empty vec of one
decoding fine is not a reason to call the column decodable.

zero_width answers the general question rather than the two instances:
unit, a sum with no variants, and a product whose every field is
zero-width, cycle-safe on the path so sibling fields of the same type
still resolve. Only the vec branch consults it; a bare zero-width column
reads nothing and decodes as before.

Also soften decodable_tag's docstring, which promised it NEVER raises.
A chain of ~100000 named typespace entries overflows the Lua stack
inside it, because only the named-entry branch recurses for real --
opt_/vec_ tag nesting is a tail call and is fine 200000 deep. That input
is the generated schema, developer-controlled rather than wire-driven,
so it needs no guard; the docstring just must not claim otherwise.

yuris changed target branch from bsatn-phase1-codec to main

2026-08-21 07:57:12 +00:00

yuris force-pushed bsatn-phase2-protocol from bd2643c57e

ci / ci (pull_request) Successful in 20s

Details

conformance / conformance (pull_request) Has been cancelled

Details

to 841dd7ed45

conformance / conformance (pull_request) Failing after 25s

Details

ci / ci (pull_request) Successful in 1m32s

Details