conformance: run the smoke over v2, and capture the wire

yuris commented

2026-08-21 05:46:14 +00:00

Owner

Phase 4 of #9. Stacked on #84.

capture-v2.mjs drives all five client messages against the dev server and dumps every frame hex-encoded. It encodes BSATN by hand rather than through stdb/bsatn.lua on purpose: a capture written with the module under test cannot catch that module encoding wrong.

What the capture settled, each recorded at the affected decode site:

ReducerResult.timestamp is microseconds. v2.rs:378's doc comment claiming nanoseconds is wrong — measured 1787288613878948, where a nanos value would be ~1.787e18.
Identity hex is the byte-reverse of the wire, confirmed against the JWT hex_identity and the v1 render.
Option is 0=Some/1=None; Result is 0=Ok/1=Err.
Client frames are accepted with no compression tag.

The dev module gains a ping event table so the smoke can assert the one behaviour v1 has no counterpart for: a row arrives decoded as an event and is absent from the cache. STDB_PROTOCOL picks the wire on both tiers.

CI keeps running v1 automatically and takes v2 by dispatch only. The path filter stays v1-only; running both wires would double every PR for a feature with no default consumers. It flips when the default does.

Verified: four suites PASS — v1 native, v2 native, v2 HTML5, v1 HTML5. All 16 captured server frames decode with remaining() == 0. No SDK fix was needed; v2 passed on the first full harness run.

Measured: v2 is 46% of v1's row bytes and 31% on client frames; 10k rows apply in 10.5ms vs 20.0ms (native, median of 3). The big-int quoting bucket is structurally absent on v2 — that scan exists only because JSON cannot carry a u128 exactly.

Phase 4 of #9. Stacked on #84. `capture-v2.mjs` drives all five client messages against the dev server and dumps every frame hex-encoded. It encodes BSATN by hand rather than through `stdb/bsatn.lua` on purpose: a capture written with the module under test cannot catch that module encoding wrong. **What the capture settled**, each recorded at the affected decode site: - `ReducerResult.timestamp` is **microseconds**. `v2.rs:378`'s doc comment claiming nanoseconds is wrong — measured `1787288613878948`, where a nanos value would be ~1.787e18. - Identity hex is the byte-reverse of the wire, confirmed against the JWT `hex_identity` and the v1 render. - Option is 0=Some/1=None; Result is 0=Ok/1=Err. - Client frames are accepted with no compression tag. The dev module gains a `ping` event table so the smoke can assert the one behaviour v1 has no counterpart for: a row arrives decoded as an event and is absent from the cache. `STDB_PROTOCOL` picks the wire on both tiers. CI keeps running v1 automatically and takes v2 by dispatch only. The path filter stays v1-only; running both wires would double every PR for a feature with no default consumers. It flips when the default does. **Verified:** four suites PASS — v1 native, v2 native, v2 HTML5, v1 HTML5. All 16 captured server frames decode with `remaining() == 0`. No SDK fix was needed; v2 passed on the first full harness run. **Measured:** v2 is 46% of v1's row bytes and 31% on client frames; 10k rows apply in 10.5ms vs 20.0ms (native, median of 3). The big-int quoting bucket is structurally absent on v2 — that scan exists only because JSON cannot carry a u128 exactly.

yuris added 1 commit

2026-08-21 05:46:14 +00:00

conformance: run the smoke over v2, and capture the wire

ci / ci (pull_request) Successful in 1m30s

Details

conformance / conformance (pull_request) Successful in 2m37s

Details

df6a963bb9

capture-v2.mjs drives all five client messages against the dev server
and dumps every frame hex-encoded. It encodes BSATN by hand rather than
through stdb/bsatn.lua on purpose: a capture written with the module
under test cannot catch that module encoding wrong.

What the capture settled. ReducerResult.timestamp is microseconds --
v2.rs:378's doc comment claiming nanoseconds is wrong, measured
1787288613878948. Identity hex is the byte-reverse of the wire.
Option is 0=Some/1=None and Result 0=Ok/1=Err. Client frames are
accepted with no compression tag.

The module gains a ping event table so the smoke can assert the one
behaviour v1 has no counterpart for: a row arrives decoded as an event
and is absent from the cache. STDB_PROTOCOL picks the wire on both
tiers. CI keeps running v1 automatically and takes v2 by dispatch only;
the path filter flips when the default does.

yuris referenced this pull request

2026-08-21 05:46:15 +00:00

readme: two wire formats, one value surface #86

yuris force-pushed bsatn-phase4-conformance from df6a963bb9

ci / ci (pull_request) Successful in 1m30s

Details

conformance / conformance (pull_request) Successful in 2m37s

Details

to ab1d9a68c6

ci / ci (pull_request) Successful in 1m32s

Details

conformance / conformance (pull_request) Successful in 2m24s

Details

2026-08-21 06:39:13 +00:00

Compare

yuris commented

2026-08-21 06:51:55 +00:00

Author

Owner

Review round: the gates themselves were not gating

The most consequential finding in the stack

Both conformance gates accepted a run that had silently fallen back to v1. Setting
STDB_PROTOCOL=v2 and getting a PASS proved that a protocol worked, not that v2 did — so the
earlier claim that "v2 HTML5 passes" was unproven at the time it was made. A green suite that
cannot distinguish the thing under test from its predecessor is not evidence.

Fixed on both tiers, and each fix is a different kind of check because the two tiers expose different
truth:

Native now requires the protocol-specific verdict line. The run must say which wire it ran.
HTML5 now asserts the negotiated Sec-WebSocket-Protocol at every connect — the
negotiated subprotocol is the only authority on what the server agreed to speak, and asserting it
per-connect also covers reconnects, where a fallback would otherwise slip in mid-run.

Negative controls prove both catch a v1-masquerading-as-v2 run that the old gates passed. That is
the part that makes this a fix rather than another assertion of good intent: the new gates were shown
to fail on the exact scenario the old ones let through.

The rest

The capture's pins now FAIL the run instead of narrating it. Four pinned wire answers were being
printed and ignored; a pin that only logs is a comment.
The committed fixture had no EventTable frame. It does now — the one wire shape unique to v2 was
absent from the only file that records the wire.
Partial captures no longer overwrite the fixture. A capture that died halfway used to replace a
good file with a truncated one.
New test_fixtures_v2.lua replays every captured frame. Before this, nothing read the
capture — it was a file the suite produced and never consumed, so a regression in v2 decoding
could not be caught by the fixture it was supposedly pinned to.

Benchmark correction (applies to the numbers this stack published): the 10k apply-latency pair of
20.0ms / 10.5ms and its ratio of 0.53 were measured with the machine's load unrecorded. A re-run of the
same commit measured every native absolute 2-3x higher (v1 median 61.7ms, v2 44.6ms over 5 runs) at
loadavg 11.06 on 16 cores, with the harness flagging 47% spread on two sweeps, and put the ratio at
0.59-0.65, not 0.53. Direction holds, the value does not — #9 and the wiki now state the conditions
instead of the bare digits, and a clean re-measure is an open follow-up. The wire-byte tables are
unaffected and stand as published (v2 ~46% of v1 on row payloads, 31% on client frames): they are byte
counts, not timings.

Verification state at the stack tip (bsatn-phase5-docs): unit suite 11 passed, 0 failed;
all four conformance suites PASS end to end — v1 native, v2 native, v1 HTML5, v2 HTML5 — under the
new protocol-specific gates, whose negative controls were proven non-vacuous; stylua --check stdb dev/test
clean.

## Review round: the gates themselves were not gating ### The most consequential finding in the stack **Both conformance gates accepted a run that had silently fallen back to v1.** Setting `STDB_PROTOCOL=v2` and getting a PASS proved that *a* protocol worked, not that **v2** did — so the earlier claim that "v2 HTML5 passes" was **unproven at the time it was made**. A green suite that cannot distinguish the thing under test from its predecessor is not evidence. Fixed on both tiers, and each fix is a different kind of check because the two tiers expose different truth: - **Native** now requires the **protocol-specific verdict line**. The run must say which wire it ran. - **HTML5** now asserts the negotiated **`Sec-WebSocket-Protocol`** at *every* connect — the negotiated subprotocol is the only authority on what the server agreed to speak, and asserting it per-connect also covers reconnects, where a fallback would otherwise slip in mid-run. **Negative controls prove both catch a v1-masquerading-as-v2 run that the old gates passed.** That is the part that makes this a fix rather than another assertion of good intent: the new gates were shown to fail on the exact scenario the old ones let through. ### The rest - **The capture's pins now FAIL the run instead of narrating it.** Four pinned wire answers were being printed and ignored; a pin that only logs is a comment. - **The committed fixture had no EventTable frame.** It does now — the one wire shape unique to v2 was absent from the only file that records the wire. - **Partial captures no longer overwrite the fixture.** A capture that died halfway used to replace a good file with a truncated one. - **New `test_fixtures_v2.lua` replays every captured frame.** Before this, **nothing read the capture** — it was a file the suite produced and never consumed, so a regression in v2 decoding could not be caught by the fixture it was supposedly pinned to. **Benchmark correction** (applies to the numbers this stack published): the 10k apply-latency pair of 20.0ms / 10.5ms and its ratio of 0.53 were measured with the machine's load unrecorded. A re-run of the same commit measured every native absolute 2-3x higher (v1 median 61.7ms, v2 44.6ms over 5 runs) at `loadavg` 11.06 on 16 cores, with the harness flagging 47% spread on two sweeps, and put the ratio at **0.59-0.65, not 0.53**. Direction holds, the value does not — #9 and the wiki now state the conditions instead of the bare digits, and a clean re-measure is an open follow-up. The **wire-byte tables are unaffected** and stand as published (v2 ~46% of v1 on row payloads, 31% on client frames): they are byte counts, not timings. **Verification state at the stack tip** (`bsatn-phase5-docs`): unit suite **11 passed, 0 failed**; all **four conformance suites PASS** end to end — v1 native, v2 native, v1 HTML5, v2 HTML5 — under the new protocol-specific gates, whose negative controls were proven non-vacuous; `stylua --check stdb dev/test` clean.

yuris force-pushed bsatn-phase4-conformance from ab1d9a68c6

ci / ci (pull_request) Successful in 1m32s

Details

conformance / conformance (pull_request) Successful in 2m24s

Details

to b2c12d8713

conformance / conformance (pull_request) Has been cancelled

Details

ci / ci (pull_request) Has been cancelled

Details

2026-08-21 07:56:24 +00:00

Compare