cache: route rows through the typed decoder so an Identity key works #48

Merged

yuris merged 2 commits from feat/35-identity-pk into main 2026-08-20 03:22:25 +00:00

Conversation 1 Commits 2 Files changed 17 +1073 -107

yuris commented 2026-08-20 02:16:57 +00:00

Owner

Copy link

Closes #35 — the last of the three PRs that each routed around this and left a comment saying so.

protocol.decode_row never converted special-type columns, so an Identity column decoded to a fresh wrapper table per row. A table key only ever matches itself, so an identity-keyed table — the canonical SpacetimeDB game pattern — was broken in three ways at once: cache.get/cache.find could never resolve it, store/remove treated every arrival as a new key so the row set grew without bound, and #21's pairing never fired so it kept emitting delete + insert.

cache.apply_database_update now routes a table's rows through protocol.decode_typed when its schema entry carries types, and through decode_row when it does not. A generated schema gets every column coerced against its declared type — an Identity column becomes the server's 0x hex string, and therefore a usable primary key — while a hand-written schema is byte-identically unchanged.

Design decisions

cache.new(schema, typespace) — a second argument, not schema.typespace. The schema's key space belongs to whoever wrote the server module, and 2.8.2 accepts and serves a table called typespace, so a def hidden under that key would be shadowed. stdb.schema.new also demands a columns list for every value handed to it, so schema.typespace cannot survive validation. Same argument the repo already made for putting table accessors under db.tables. Optional throughout, and what "optional" costs is narrower than first stated: exactly two things degrade without it — a module-declared sum comes back as the raw [ordinal, payload], and a nested product inside a positional row comes back as a positional array. A nested product in a named readback is unaffected, because that row carried the field names itself. Special types and scalars always decode, so the identity fix works with no typespace at all, which is pinned by a test.

The switch is the presence of def.types, per table and never per row. Sniffing the pk's type and coercing only that column was rejected: the same table would then hand out a converted key and unconverted values of the identical wire type, so row.owner ~= row.identity for one identity. A table with no schema entry at all — a view, #42 — takes the untyped branch and behaves exactly as before.

schema.new validates types at construction. types is load-bearing now and hand-writable — the example in dev/example/main/smoke.script writes one out. A skew against columns used to be caught only by decode_typed, i.e. inside apply_database_update, on the first row of that table, with no pcall between it and Defold's websocket callback. types too short, too long, types = {}, a bare string and a set-shaped table now all raise from schema.new, naming the line the author wrote. Absent types stays legal — its absence is the request for the untyped decoder, and that is asserted so the check cannot close the escape hatch.

The u64 > 2^53 limit is declined, bounded and documented. Lua 5.1 has one number type: a u64 pk above 2^53 collides, two server keys become one Lua key, the cache merges rows, last arrival wins, count is short, nothing raises. Returning digits as text would fix it and would retype message.id from number to string for every consumer of every u64 column — a larger break than the bound is worth. Documented in the README, cache.get and protocol.lua's PRECISION block, and pinned three ways: collides at 2^53/2^53+1, stays distinct at 2^53-2/2^53-1, and stays distinct at u128::MAX-1/u128::MAX because a 128-bit key is an exact decimal string.

ConnectionId and Uuid freeze as decimal digits, Identity as canonical hex. That asymmetry is now defended in the README rather than merely stated. It is not a losslessness constraint — dec_to_hex32 is exact to 2^128 — it is that these two are the only special types whose wire form is a number, and rendering at decode time would put a formatting choice on the frozen path plus a digit-array division per column, on a decode path already at ×2.1 for a row of that shape. connection_id_to_hex and uuid_to_string are the renderers and they are one call away.

Cost — the interesting part

Routing through decode_typed is never cheaper than decode_row. Not on any table shape, not in either encoding. An earlier draft of this body claimed player came out at ×0.76 — that does not reproduce, and it was wrong in direction, not just in magnitude.

20k-row named readback per table through cache.apply_database_update, best of 9, main (bf7c75b) against this branch, byte-identical row text into both. Two independent runs:

table	before	after
message (u64, string, string)	7.00 / 6.87 µs/row	8.55 / 8.38 µs/row	×1.22 / ×1.22
player (identity, identity, string, i32)	17.51 / 17.26 µs/row	22.78 / 23.17 µs/row	×1.30 / ×1.34
profile (identity, timestamp, opt, vec, sum, nested product)	31.03 / 30.61 µs/row	65.06 / 66.46 µs/row	×2.10 / ×2.17

Budget against profile, not against message. A table carrying a timestamp plus a sum plus a nested product is the realistic game shape and it is the real worst case at ~×2.1.

Confirmed at the decode level too, decode_row against decode_typed in one process on the same row text, so no cross-head noise: message 6.37 → 7.87 µs/row (×1.24), player 16.92 → 22.24 (×1.31), profile 31.94 → 65.22 (×2.04).

The mechanism the earlier claim rested on is also false. quote_big_ints' pattern is "(__[%a_]+__)":(%d+) — it needs an unquoted digit run after the magic key, and an Identity column's value is a quoted hex string, so on player that gsub finds nothing to replace and there is no work for the typed path to save.

The first cut was ×2.14 on message. Attribution: quote_wide_ints was a per-character Lua scan whose every rescue decode_value then undid, for a table with no 128-bit column at all. It now returns early unless the raw text contains a 16-digit run or the rescue mark — a necessary condition for the scan to change anything, since only two branches rewrite and every other path appends the exact input slice, so there are false positives and no false negatives. The guard pattern is built from WIDE_DIGITS so the two cannot drift.

What the guard is worth, measured by stripping it:

table	guard fires on	guarded	unguarded
message	0% of rows	×1.24	×2.02
player	1.9% of rows	×1.31	×1.92
profile	100% of rows	×2.04	×1.93

So it is worth ~1.5x on the first two and buys profile nothing — a micros Timestamp is always 16 digits, so the guard fires on every row and only adds a wasted find. (player's 1.9% is real, not an artifact: a full-entropy 64-hex identity contains a 16-digit run by chance about that often. Benchmarking with %064x identities instead puts 60 leading zeros in every row, trips the guard on 100% of them and inflates the typed path.)

Recorded against #30's budget question in the commit message and in decode_typed's docstring.

Verification

6 passed, 0 failed; stylua clean; native conformance PASS — all 5 phases in both engine runs, including this PR's new PHASE5, with the identity persisting across the process restart. Run 4 times in total (3 on the head before the final fixup, once on the final commit), green every time; the phase-2 flake (#39) was not hit.

Nine mutations, each reverted:

mutation	failure
always decode_row (reverts the fix)	an identity pk keys the store by its hex string: expected true, got false; cache.get resolves the row by the identity STRING: expected true, got false
always decode_typed (drops the escape hatch)	4 files fail with decode_typed needs a schema spec with columns and types
cache.new drops the typespace	and the typespace named the sum's ordinal: expected busy, got nil
connection.new drops the typespace	connection.new hands the typespace it was given to the cache: expected table, got nil + the same from test_init
init.lua drops the typespace	connect forwards the typespace object through to the cache: expected table, got nil
schema.new stops validating types	types shorter than columns is rejected at construction: expected error, call succeeded
schema.new checks types shape but not length	same
schedule_at_to_seconds loses the typed shape	typed Interval variant, the cached shape: expected interval, got nil
WIDE_RUN drifts to 17 digits	a 16-digit u128 one past 2^53 keeps its last digit
schema.new's types check loses its diagnostic noun	and says so as a types diagnosis, naming type tags rather than column names: expected true, got false

The two typespace-forwarding mutations and the three schema.new ones are new here. Forwarding was a one-line seam the whole feature hangs on and deleting it used to leave the suite green. The last one is worth its row: the first cut of the types check reused the shared list validator without its diagnostic noun, so an invalid types raised a concat error from inside schema.lua — which assert_error accepts as happily as a real diagnosis. The two shape assertions now match the message text, not just the fact that something raised.

An identity-keyed table was added to dev/server/index.js and captured as a second socket (frames 30-36), so frames 1-29 stay byte-identical and the fixture-backed test decodes real bytes rather than hand-built input.

The frozen row representation was re-verified by replaying all 26 captured database_updates through cache.apply_database_update on both heads and diffing canonical row dumps: message rows are byte-identical across the two, and the player cache key equals db.identity from the IdentityToken on the positional insert, the pk-preserving update and the named readback alike.

Closes #35 — the last of the three PRs that each routed around this and left a comment saying so. `protocol.decode_row` never converted special-type columns, so an `Identity` column decoded to a **fresh wrapper table per row**. A table key only ever matches itself, so an identity-keyed table — *the* canonical SpacetimeDB game pattern — was broken in three ways at once: `cache.get`/`cache.find` could never resolve it, `store`/`remove` treated every arrival as a new key so the row set grew without bound, and #21's pairing never fired so it kept emitting `delete` + `insert`. `cache.apply_database_update` now routes a table's rows through `protocol.decode_typed` when its schema entry carries `types`, and through `decode_row` when it does not. A generated schema gets every column coerced against its declared type — an `Identity` column becomes the server's `0x` hex string, and therefore a usable primary key — while a hand-written schema is byte-identically unchanged. ## Design decisions **`cache.new(schema, typespace)` — a second argument, not `schema.typespace`.** The schema's key space belongs to whoever wrote the server module, and 2.8.2 accepts and serves a table called `typespace`, so a def hidden under that key would be shadowed. `stdb.schema.new` also demands a `columns` list for every value handed to it, so `schema.typespace` cannot survive validation. Same argument the repo already made for putting table accessors under `db.tables`. Optional throughout, and what "optional" costs is narrower than first stated: exactly two things degrade without it — a module-declared **sum** comes back as the raw `[ordinal, payload]`, and a nested **product inside a positional row** comes back as a positional array. A nested product in a *named* readback is unaffected, because that row carried the field names itself. Special types and scalars always decode, so the identity fix works with no typespace at all, which is pinned by a test. **The switch is the presence of `def.types`, per table and never per row.** Sniffing the pk's type and coercing only that column was rejected: the same table would then hand out a converted key and unconverted values of the identical wire type, so `row.owner ~= row.identity` for one identity. A table with no schema entry at all — a view, #42 — takes the untyped branch and behaves exactly as before. **`schema.new` validates `types` at construction.** `types` is load-bearing now *and* hand-writable — the example in `dev/example/main/smoke.script` writes one out. A skew against `columns` used to be caught only by `decode_typed`, i.e. inside `apply_database_update`, on the first row of that table, with no `pcall` between it and Defold's websocket callback. `types` too short, too long, `types = {}`, a bare string and a set-shaped table now all raise from `schema.new`, naming the line the author wrote. Absent `types` stays legal — its absence *is* the request for the untyped decoder, and that is asserted so the check cannot close the escape hatch. **The `u64 > 2^53` limit is declined, bounded and documented.** Lua 5.1 has one number type: a `u64` pk above 2^53 collides, two server keys become one Lua key, the cache merges rows, last arrival wins, `count` is short, nothing raises. Returning digits as text would fix it and would retype `message.id` from number to string for every consumer of every `u64` column — a larger break than the bound is worth. Documented in the README, `cache.get` and `protocol.lua`'s PRECISION block, and pinned three ways: collides at 2^53/2^53+1, stays distinct at 2^53-2/2^53-1, and stays distinct at `u128::MAX-1`/`u128::MAX` because a 128-bit key is an exact decimal string. **`ConnectionId` and `Uuid` freeze as decimal digits, `Identity` as canonical hex.** That asymmetry is now defended in the README rather than merely stated. It is not a losslessness constraint — `dec_to_hex32` is exact to 2^128 — it is that these two are the only special types whose wire form is a *number*, and rendering at decode time would put a formatting choice on the frozen path plus a digit-array division per column, on a decode path already at ×2.1 for a row of that shape. `connection_id_to_hex` and `uuid_to_string` are the renderers and they are one call away. ## Cost — the interesting part Routing through `decode_typed` is **never cheaper** than `decode_row`. Not on any table shape, not in either encoding. An earlier draft of this body claimed `player` came out at ×0.76 — that does not reproduce, and it was wrong in direction, not just in magnitude. 20k-row named readback per table through `cache.apply_database_update`, best of 9, `main` (`bf7c75b`) against this branch, byte-identical row text into both. Two independent runs: | table | before | after | | |---|---|---|---| | `message` (u64, string, string) | 7.00 / 6.87 µs/row | 8.55 / 8.38 µs/row | **×1.22 / ×1.22** | | `player` (identity, identity, string, i32) | 17.51 / 17.26 µs/row | 22.78 / 23.17 µs/row | **×1.30 / ×1.34** | | `profile` (identity, timestamp, opt, vec, sum, nested product) | 31.03 / 30.61 µs/row | 65.06 / 66.46 µs/row | **×2.10 / ×2.17** | **Budget against `profile`, not against `message`.** A table carrying a timestamp plus a sum plus a nested product is the realistic game shape and it is the real worst case at ~×2.1. Confirmed at the decode level too, `decode_row` against `decode_typed` in one process on the same row text, so no cross-head noise: `message` 6.37 → 7.87 µs/row (×1.24), `player` 16.92 → 22.24 (×1.31), `profile` 31.94 → 65.22 (×2.04). The mechanism the earlier claim rested on is also false. `quote_big_ints`' pattern is `"(__[%a_]+__)":(%d+)` — it needs an **unquoted** digit run after the magic key, and an `Identity` column's value is a quoted hex string, so on `player` that gsub finds nothing to replace and there is no work for the typed path to save. **The first cut was ×2.14 on `message`.** Attribution: `quote_wide_ints` was a per-character Lua scan whose every rescue `decode_value` then undid, for a table with no 128-bit column at all. It now returns early unless the raw text contains a 16-digit run or the rescue mark — a *necessary* condition for the scan to change anything, since only two branches rewrite and every other path appends the exact input slice, so there are false positives and no false negatives. The guard pattern is built from `WIDE_DIGITS` so the two cannot drift. What the guard is worth, measured by stripping it: | table | guard fires on | guarded | unguarded | |---|---|---|---| | `message` | 0% of rows | ×1.24 | ×2.02 | | `player` | 1.9% of rows | ×1.31 | ×1.92 | | `profile` | **100%** of rows | ×2.04 | ×1.93 | So it is worth ~1.5x on the first two and **buys `profile` nothing** — a micros `Timestamp` is always 16 digits, so the guard fires on every row and only adds a wasted `find`. (`player`'s 1.9% is real, not an artifact: a full-entropy 64-hex identity contains a 16-digit run by chance about that often. Benchmarking with `%064x` identities instead puts 60 leading zeros in every row, trips the guard on 100% of them and inflates the typed path.) Recorded against #30's budget question in the commit message and in `decode_typed`'s docstring. ## Verification `6 passed, 0 failed`; stylua clean; native conformance PASS — all 5 phases in both engine runs, including this PR's new PHASE5, with the identity persisting across the process restart. Run 4 times in total (3 on the head before the final fixup, once on the final commit), green every time; the phase-2 flake (#39) was not hit. Nine mutations, each reverted: | mutation | failure | |---|---| | always `decode_row` (reverts the fix) | `an identity pk keys the store by its hex string: expected true, got false`; `cache.get resolves the row by the identity STRING: expected true, got false` | | always `decode_typed` (drops the escape hatch) | 4 files fail with `decode_typed needs a schema spec with columns and types` | | `cache.new` drops the typespace | `and the typespace named the sum's ordinal: expected busy, got nil` | | `connection.new` drops the typespace | `connection.new hands the typespace it was given to the cache: expected table, got nil` + the same from `test_init` | | `init.lua` drops the typespace | `connect forwards the typespace object through to the cache: expected table, got nil` | | `schema.new` stops validating `types` | `types shorter than columns is rejected at construction: expected error, call succeeded` | | `schema.new` checks `types` shape but not length | same | | `schedule_at_to_seconds` loses the typed shape | `typed Interval variant, the cached shape: expected interval, got nil` | | `WIDE_RUN` drifts to 17 digits | `a 16-digit u128 one past 2^53 keeps its last digit` | | `schema.new`'s `types` check loses its diagnostic noun | `and says so as a types diagnosis, naming type tags rather than column names: expected true, got false` | The two typespace-forwarding mutations and the three `schema.new` ones are new here. Forwarding was a one-line seam the whole feature hangs on and deleting it used to leave the suite green. The last one is worth its row: the first cut of the `types` check reused the shared list validator without its diagnostic noun, so an invalid `types` raised a *concat error from inside `schema.lua`* — which `assert_error` accepts as happily as a real diagnosis. The two shape assertions now match the message text, not just the fact that something raised. An identity-keyed table was added to `dev/server/index.js` and captured as a **second socket** (frames 30-36), so frames 1-29 stay byte-identical and the fixture-backed test decodes real bytes rather than hand-built input. The frozen row representation was re-verified by replaying all 26 captured `database_update`s through `cache.apply_database_update` on both heads and diffing canonical row dumps: `message` rows are **byte-identical** across the two, and the `player` cache key equals `db.identity` from the `IdentityToken` on the positional insert, the pk-preserving update and the named readback alike.

yuris added 1 commit 2026-08-20 02:16:57 +00:00

Route cached rows through the typed decoder so an Identity key works ... 2770953276

Closes #35. A table keyed by an `Identity` — `player` keyed by the
connecting identity, the canonical SpacetimeDB game pattern — did not
work in the cache at all, and the whole test corpus dodged it.

WHY IT WAS BROKEN. `cache.apply_database_update` decoded every row
through `protocol.decode_row`, which hands a column back as whatever
json.decode made of it. A special-typed column therefore became a fresh
Lua TABLE per row, and a table key only ever matches itself. So one
logical row became one cache entry per arrival, the row set grew without
bound, `get`/`find` missed every spelling of the identity a consumer
could hold — the string `IdentityToken` gives you, a re-wrapped
`{__identity__=…}`, anything — and #21's delete/insert pairing never
fired, so an identity-keyed table kept emitting despawn/respawn.

WHY IT HAD TO LAND BEFORE 1.0. Fixing it changes the TYPE of row field
values, which 1.0 freezes: `row.owner` was a table and is now a string.
Three merged PRs each routed around this and left a comment saying so
(#37 on `unwrap`, #46 on `decode_typed`, #44's caveat on `find`). After
1.0 it costs a major version.

WHAT THE CONTRACT IS NOW. One switch, stated once: a table whose schema
entry carries `types` decodes through `protocol.decode_typed`; a table
without decodes through `decode_row`, byte-identically to before. The
generator emits `types`, so a generated schema gets coercion and a
hand-written schema is untouched — that is the compatibility path, and
it needs no edit from anyone. README's new "Row value types" section is
the full per-tag table, and both halves are asserted against captured
`player` frames rather than hand-built input.

THE TYPESPACE IS A SECOND ARGUMENT, not `schema.typespace`. The schema's
key space belongs to whoever wrote the server module: 2.8.2 will serve a
table called `typespace`, and a def hidden under that key would shadow
it. `stdb.schema.new` would reject the entry anyway, since it demands a
`columns` list of everything handed to it. Same argument that put table
accessors under `db.tables`. It stays optional: only `sum_<Name>` and
`product_<Name>` columns need it, and omitting it degrades those alone.

THE u64 CEILING IS BOUNDED, NOT FIXED. Lua 5.1 has one number type, so a
`u64` primary key above 2^53 collides: two server keys become one Lua
key and the cache merges the rows, silently. Returning the digits as
text would fix it and would retype `message.id` for every consumer,
which is a bigger break than the bound is worth — so u64 stays a number
and the limit is documented in the README and `cache.get` and pinned
both ways in test_cache.lua (collides at 2^53, distinct below it,
distinct at any width for a u128 key).

COST, measured rather than assumed. Routing here coerces every column of
every row on the main thread. On a 20k-row InitialSubscription,
`apply_database_update`, best of 9:

  message (u64,string,string)          130 ms -> 167 ms   x1.28
  player  (identity,identity,...)      688 ms -> 520 ms   x0.76

The first cut of this was x2.14, and 62% of the added cost was
`quote_wide_ints` — a per-character Lua scan, as expensive as
json.decode itself, whose every rescue `decode_value` then undoes for a
table with no 128-bit column. It now returns early unless the raw text
carries a 16-digit run or the rescue mark, which is a necessary
condition for the scan to change anything: every other branch appends
the exact input slice. Noted against #30.

FIXTURES. `player` is a new table in dev/server/index.js with an
Identity primary key and a unique non-key Identity column — the second
one because the cache indexes every unique column MINUS the pk, so
`find` on the key delegates to `get` and only a non-key column exercises
the index. Frames 30-36 are a second socket appended to session.jsonl:
its own IdentityToken, then a positional insert, a pk-preserving change
in one `updates` entry, an unsubscribe, and a named readback of the same
row. Appended rather than recaptured on purpose — a full recapture moves
every frame byte-wise (identity, connection id, timestamps, table ids,
sequence counters) and would present dozens of pinned assertions as an
unreadable diff. capture-fixtures.mjs takes a block argument to replace
one block in place.

Conformance gains a phase 5: the same table through a real engine, with
the key coming from IdentityToken rather than a fixture. It is the only
coverage of stdb.connect -> cache.new -> typed decoding on a live
socket, and `message` stays hand-written there so the untyped path is
still exercised in the same run.

yuris commented 2026-08-20 02:56:00 +00:00

Author

Owner

Copy link

Review — automated reviewer, posted for the record

Verdict: APPROVE WITH CHANGES — closes #35 in full; no blocker. One accessor regression, one materially wrong cost claim, one missing validation, one untested wiring seam, one doc command broken. All five fixes are local and none touches the frozen row representation.

closes 35

{
"answer": "YES",
"acceptance_criterion": "MET, on real captured bytes. Frames 33 (join_game, positional Identity pk) then 34 (set_score, pk-preserving) through cache.apply_database_update with the generated schema: count=1 throughout, #events=1, events[1].type='update', old.score=0, row.score=42, old.identity==row.identity. Re-verified independently of the author's test in /tmp/probe.lua.",
"work_items": "All five checked. (1) representation decided + documented (README 647-783); (2) identity table works — verified; (3) player added to dev/server/index.js and captured as frames 30-36 (second socket, own IdentityToken); (4) u64>2^53 documented + pinned 3 ways; (5) README:469 update row rewritten, no stale '#35' user-facing text remains.",
"live_proof": "Native conformance on STDB_PORT=4700: PHASE5 PASS in BOTH engine runs — 'SMOKE: PASS PHASE5 identity-keyed table: one row throughout, one update event, resolved by 0xc2002e0a…'. Flake #39 not hit."
}

section 1 compatibility break

{
"method": "Same 25 captured database_updates from dev/test/fixtures/session.jsonl replayed through cache.apply_database_update on origin/main (d646507) and on 2770953, same generated schema, both encodings. Scripts /tmp/dump.lua, /tmp/before.txt, /tmp/after.txt.",
"message_u64_string_string": "UNCHANGED in both encodings, across all 18 captured message rows. Byte-identical dumps.",
"player_identity_identity_string_i32": "identity, sponsor: {identity='0x…'} (named) / ['0x…'] (positional) -> '0x…' string. name, score unchanged.",
"profile": "NO captured rows exist in the fixture (grep 'profile' session.jsonl -> 0). Measured on synthetic rows matching the documented wire shapes (/tmp/plumbing.lua): owner identity->string, joined timestamp->number, nickname opt_string->value-or-nil, tags vec_string->unchanged, status sum->{tag='busy'}, home product->{x,y} from both encodings.",
"all_types_full_table": [
{
"col": "id, flag, i_8..u_32, f_32, f_64, text, words",
"before": "number/boolean/string/array",
"after": "identical",
"changed": false
},
{
"col": "i_64",
"before": "number(-9.2233720368547758e+18)",
"after": "identical — still rounds",
"changed": false
},
{
"col": "u_64",
"before": "number(1.8446744073709552e+19)",
"after": "identical — still rounds",
"changed": false
},
{
"col": "i_128",
"before": "number(-1.7014118346046923e+38)",
"after": "string('-170141183460469231731687303715884105728')",
"changed": true
},
{
"col": "u_128",
"before": "number(3.4028236692093846e+38)",
"after": "string('340282366920938463463374607431768211455')",
"changed": true
},
{
"col": "i_256, u_256",
"before": "string('0x…') verbatim",
"after": "identical",
"changed": false
},
{
"col": "bytes (vec_u8)",
"before": "string('0001feff') — hex TEXT, 8 chars",
"after": "4 RAW BYTES 00 01 fe ff (verified #v==4, v:byte(i))",
"changed": true,
"note": "anyone printing a bytes column now prints binary"
},
{
"col": "status (sum_Status)",
"before": "[1,{}] named / [1,[]] positional",
"after": "{tag='busy'}",
"changed": true
},
{
"col": "kind (sum_Kind)",
"before": "[2,{x=5,y=-6}] named / [2,[5,-6]] positional",
"after": "{tag='pointy',value={x=5,y=-6}}",
"changed": true
},
{
"col": "point (product_Point)",
"before": "{x=3,y=-4} named / [3,-4] positional",
"after": "{x=3,y=-4} from BOTH",
"changed": "positional only — named was already this"
},
{
"col": "opt_int (opt_i32)",
"before": "[0,7]",
"after": "7",
"changed": true
},
{
"col": "opt_point",
"before": "[0,{x=8,y=9}] named / [0,[8,9]] positional",
"after": "{x=8,y=9}",
"changed": true
},
{
"col": "opt_opt (None)",
"before": "[1,{}]",
"after": "nil — the field is ABSENT from the row table",
"changed": true
},
{
"col": "owner (identity)",
"before": "{identity='0x…'} named / ['0x…'] positional",
"after": "'0x…' string",
"changed": true
},
{
"col": "conn (connection_id)",
"before": "{connection_id='218…537'} named / [2.1803579067259356e+38] positional — ROUNDED",
"after": "'218035790672593550386615283461651752537' exact digits",
"changed": true,
"note": "positional case GAINS precision it never had"
},
{
"col": "at (timestamp)",
"before": "{timestamp_micros_since_unix_epoch=n} / [n]",
"after": "n",
"changed": true
},
{
"col": "span (time_duration)",
"before": "{time_duration_micros=n} / [n]",
"after": "n",
"changed": true
},
{
"col": "sched (schedule_at)",
"before": "[0,{time_duration_micros=1500000}] / [0,[1500000]]",
"after": "{tag='Interval',value=1500000}",
"changed": true
},
{
"col": "uid (uuid)",
"before": "{uuid='1512…040'} / [1.5123660752041709e+36] ROUNDED",
"after": "'1512366075204170947332355369683137040' exact",
"changed": true,
"note": "positional gains precision"
}
],
"identity_vs_db_identity": "MATCHES EXACTLY. connection.lua:541 sets c.identity = protocol.unwrap(msg.identity) -> the bare hex string. Driven through handle_frame on real frames 30 then 33 (/tmp/final.lua): db.identity = '0xc200ff15…085b4' (string, 66 chars); cache.get(state,'player',db.identity).name = 'yuris'; row.identity == db.identity -> true; same-length -> true. Named readback (frame 36) lands on the SAME key, count stays 1. find(sponsor, db.identity) and find(identity, db.identity) both resolve. Verdict: RIGHT representation to freeze.",
"sums_and_nested_products": "NAMED now, both of them. Sums -> {tag='', value=} with the server's lowercased variant names ('idle','busy','nothing','pointy'); ScheduleAt uses library-cased 'Interval'/'Time'. Nested products -> {x=,y=} from BOTH encodings, so the positional/named asymmetry documented at README 619-625 is erased at the consumer. Correct and the whole point of #46.",
"accessor_matrix": {
"method": "/tmp/accessors.lua and /tmp/final.lua — every #23/#37 accessor against the old NAMED shape, the old POSITIONAL shape, and the new typed shape, on real fixture bytes for all_types in both encodings.",
"timestamp_to_seconds": "WORKS on the typed shape. 1787163313.2782 from all three. scalar() passes a non-table straight through.",
"duration_to_seconds": "WORKS. 90061 from all three.",
"identity_to_hex": "WORKS, idempotent — '0x…' in, '0x…' out (matches ^0x%x+$).",
"connection_id_to_hex": "WORKS on typed, and now works where it used to fail: untyped positional returned nil, typed returns 0xa4082a73357fa6a74fad283d8974e659. Strict improvement.",
"uuid_to_string": "WORKS on typed; same improvement (untyped positional -> nil, typed -> 01234567-89ab-cdef-fedc-ba9876543210).",
"schedule_at_to_seconds": "*** BROKEN ON THE TYPED SHAPE *** — returns nil in BOTH encodings. See finding SF-1.",
"unwrap": "no-op passthrough on typed values. Fine.",
"verdict": "5 of 6 accessors survive; connection_id/uuid get strictly better. Exactly one regresses."
},
"usable_without_further_conversion": {
"identity": "YES — canonical 0x hex",
"timestamp": "YES — micros number",
"time_duration": "YES — micros number",
"schedule_at": "shape yes, but the accessor rejects it (SF-1)",
"connection_id": "NO — exact DECIMAL digits; needs connection_id_to_hex to be human/comparable-with-server",
"uuid": "NO — exact DECIMAL digits; needs uuid_to_string",
"raised_for_decision": "This asymmetry is being frozen: identity decodes to canonical text but connection_id/uuid decode to decimal digits. It is NOT a losslessness constraint — dec_to_hex32 (protocol.lua:351-378) is exact to 2^128, so decode_typed could return '0x…' / hyphenated form losslessly and remove the only two tags needing a second call. Cost is one digit-array division loop per such column, on columns that are rare. Worth deciding NOW rather than after 1.0."
}
}

section 2 escape hatch

{
"byte_identity": "VERIFIED. Hand-written defs (column names copied from the generated schema, types removed) replayed over all 25 fixture database_updates on main and on the branch; full row dumps compared with cmp -> IDENTICAL, 25/25 lines. /tmp/probe.lua, /tmp/m2.txt vs /tmp/b2.txt.",
"boundary_cases": [
{
"case": "no types",
"main": "OK, wrapper preserved",
"branch": "OK, identical",
"verdict": "escape hatch intact"
},
{
"case": "types full length",
"main": "OK (types ignored)",
"branch": "OK, coerced",
"verdict": "as designed"
},
{
"case": "types TOO SHORT (2 for 4 columns)",
"main": "OK, silently ignored",
"branch": "RAISES 'stdb.protocol: schema spec has 4 columns but 2 types' from cache.lua:216",
"verdict": "NEW raise — see SF-3"
},
{
"case": "types TOO LONG (5 for 4)",
"main": "OK",
"branch": "RAISES same",
"verdict": "NEW raise"
},
{
"case": "types = {} (present but EMPTY)",
"main": "OK",
"branch": "RAISES 'schema spec has 4 columns but 0 types'",
"verdict": "NEW raise; 'presence of types is the whole test' is literally true and an empty list is not the no-op an author would expect"
},
{
"case": "table absent from schema (a view, #42)",
"main": "ins=1, row = {} (empty)",
"branch": "ins=1, row = {} — IDENTICAL",
"verdict": "correct, unchanged, #42 untouched"
}
],
"mixed_schema": "BEHAVES PER-TABLE AS CLAIMED. One state, two defs, same bytes into both: typed.owner = 'string:0xaa' and get(typed,'0xaa') resolves; untyped.owner = {1='0xaa'} and get(untyped,'0xaa') misses. On main both were the wrapper and both missed. Routing is per-table, never per-row (cache.lua:210-221)."
}

section 3 typespace signature

{
"claim_1_server_accepts_a_table_named_typespace": "TRUE, verified against a live 2.8.2 on port 4600 with a purpose-built module. spacetime publish exit 0, no warning. Schema JSON lists typespace in tables with primary_key [0], its btree index and its Unique constraint intact. The server SERVES it in all three shapes: InitialSubscription (named row), TransactionUpdate (positional row), OneOffQueryResponse (named). SELECT * FROM typespace accepted, no SubscriptionError.",
"claim_2_generator_emits_both": "TRUE and it is the crux — dev/tools/gen-schema.sh emits tables['typespace'] = { columns={'id','label'}, types={'u64','string'}, pk='id', … } at line 59 AND a top-level typespace = { Kind=…, Point=…, Status=… } at line 131 of the SAME generated file. No name-mangling anywhere.",
"claim_3_schema_new_rejects_it": "TRUE. defs.typespace = gen.typespace; schema.new(defs) raises 'stdb.schema: table "typespace" needs a columns list (order matters)' (schema.lua:62-64). Control run without the injected key passes, so the failure is caused by the key alone.",
"claim_4_shadowing_is_real": "TRUE. Plain table assignment, no metatable, no error at assignment: defs.typespace goes from the real table def (columns={'id','label'}, pk='id') to the typespace object; .columns and .pk become nil and the real def is unreachable. Ordering only picks which of the two is destroyed. So the rejected design cannot even be constructed — one branch raises, the other drops data. The two-argument signature is justified by facts, not a hypothetical.",
"plumbing": "VERIFIED END TO END (/tmp/plumbing.lua). connection.new({typespace=gen.typespace}) -> state.typespace is the same object; frames driven through handle_frame produce typed rows; profile.status.tag='busy', profile.home='1,2', profile.joined is a number. init.lua:244 forwards typespace = opts.typespace and init.lua:139 documents it.",
"no_typespace_degradation": "CLAIM HOLDS, with one correction. Generated schema, typespace=nil: identity pk still resolves (get by hex string -> true), so the #35 fix does NOT depend on the typespace. Sums degrade to the raw [ordinal,payload] (status.tag=nil, status[1]=1). BUT a NAMED nested product does not degrade — profile.home.x=1 still works, because the named row already carries field names and the untypespaced product_ path passes it through. Only a POSITIONAL nested product actually degrades. README:682 and cache.lua:79-80 say 'sum_ / product_ columns degrade' without that qualifier. Nit.",
"tables_facades": "SEE THE SAME DECODED ROWS. conn.tables.player:on_insert fires with row.identity as a 'string' starting 0xffff; :find('sponsor', db.identity) and :find('identity', db.identity) both resolve; :count() correct. No divergence between the cache and the facade."
}

section 4 u64 bound

{
"pinned_cases_reproduced": [
{
"case": "2^53 and 2^53+1 (u64 pk)",
"result": "count=1, inserts=2, events=2, get(9007199254740992).note='second', NOTHING RAISES"
},
{
"case": "2^53-2 and 2^53-1 (u64 pk)",
"result": "count=2 — the boundary is real, not a blanket 'u64 is broken'"
},
{
"case": "u128::MAX-1 and u128::MAX (u128 pk)",
"result": "count=2, get by the exact 39-digit string -> 'second'. On main the same input gave count=1 and get->nil, so this is a fix, not just a workaround."
}
],
"extra_probe": "u64::MAX-1 and u64::MAX also collide: count=1, later row wins. Lua 5.1 confirms 2^53 == 2^53+1 -> true and tonumber('9007199254740993') == 2^53 -> true.",
"consequence_confirmed": "Two server keys -> one Lua key; cache merges; last arrival wins; cache.count is 1 while counts.inserts is 2, so the consumer's own event stream and the cache disagree and nothing says why.",
"judgement": "The REPRESENTATION decision is right — retyping every u64 column to string to fix a 9.0e15-row ceiling is a worse trade, and the README's snowflake/hash caveat is the correct guidance. The SILENCE is wrong to freeze. This repo's stated posture is that failures are loud or absent (README:45-49 'it is silently wrong rather than loud' is listed as a TRAP; cache.find raises rather than returning nil for a non-unique column precisely so a lookup that can never succeed says so). A silent row merge with a count mismatch is exactly the class of thing the repo elsewhere refuses to ship.",
"cheapest_loud_alternative": "A one-time warning, costing nothing on the happy path, at protocol.lua:957-965. The NUMERIC branch already holds BOTH the exact digit string and the double it produced: local digits = rescued_digits(v); if digits then return tonumber(digits) end. Add one format+compare there: if string.format('%.0f', tonumber(digits)) ~= digits then the double lost a digit. VERIFIED EXACT in Lua 5.1 (/tmp/loud.lua, /tmp/loud2.lua): fires on 9007199254740993, 18446744073709551615, -9007199254740993, -9223372036854775807; does NOT fire on 9007199254740992 (2^53, representable), 9007199254740994, 10000000000000000, 1234567890123457; zero false positives over 5000 exactly-representable integers; fires on exactly the odd integers above 2^53. Cost is bounded to values the rescue already caught (>=16 digits), which the quote_wide_ints guard already establishes is rare — so a table of plain scalars pays nothing. Sharper than the README's framing too: it flags actual precision loss, not 'above 2^53'. Recommend emitting it once per process (or as an on_event) rather than raising, so an existing consumer is not broken by a diagnostic."
}

section 5 cost claims

{
"method": "20k-row named subscription readback per table, best-of-9 os.clock, same generated schema and same rows against origin/main (before) and 2770953 (after). Realistic full-entropy 64-hex-char identities, NOT %064x (60 leading zeros is a 16-digit run and spuriously triggers the guard — my first attempt made that mistake and inflated the typed path). /tmp/bench2.lua, two independent runs.",
"measured": [
{
"table": "message (u64,string,string)",
"main": "6.39 / 6.30 us/row",
"branch": "8.16 / 7.81 us/row",
"ratio": "x1.28 / x1.24",
"author_claim": "x1.28",
"verdict": "REPRODUCES EXACTLY"
},
{
"table": "player (identity,identity,string,i32)",
"main": "17.36 / 17.01 us/row",
"branch": "23.68 / 23.71 us/row",
"ratio": "x1.36 / x1.39",
"author_claim": "x0.76 — CHEAPER",
"verdict": "DOES NOT REPRODUCE. Wrong by a factor of ~1.8 and wrong in DIRECTION."
},
{
"table": "profile (identity, timestamp, opt, vec, sum, nested product)",
"main": "29.44 / 28.84 us/row",
"branch": "66.46 / 66.50 us/row",
"ratio": "x2.26 / x2.31",
"author_claim": "absent from the PR's table",
"verdict": "the realistic worst case, and undocumented"
}
],
"decode_level_on_real_fixture_bytes": "/tmp/bench4.lua, decode_row vs decode_typed on the actual captured rows: message positional x1.19, message named x1.27, player positional x1.33, player named x1.26, all_types positional x1.94, all_types named x1.85. decode_typed is NEVER cheaper, in any table, in either encoding. The author's 'decode_row 28.18 vs decode_typed 22.10 us/row' does not reproduce; I also swept identity shapes and encodings (/tmp/bench3.lua: real-id named x1.27, random-id named x1.21, %064x named x1.82, real-id positional x1.32, %064x positional x1.91) and found no configuration where the typed path wins. The stated mechanism ('quote_big_ints gsub costs more than the guarded scan plus per-column coercion') is false for player: quote_big_ints' pattern '"([%a_]+)":(%d+)' cannot even match an identity column (the value is quoted), so it does no replacement work there, while typed decoding still pays two special-type coercions.",
"first_cut_x2_14": "DIRECTIONALLY CONFIRMED, magnitude differs. Guard stripped from quote_wide_ints (/tmp/noguard): message 11.89 us/row = x1.86 vs main; player 35.31 = x2.03. So the guard is worth ~1.5x on message and player and the 'x2.14 before the guard' story is real. Note it buys profile nothing — 61.73 unguarded vs 66.50 guarded, because a micros timestamp is always 16 digits so the guard always fires and only adds a wasted find.",
"guard_false_negative_hunt": {
"conclusion": "NO FALSE NEGATIVE FOUND. The claim survives both a structural argument and 41,306 differential cases.",
"structural": "quote_wide_ints has exactly two rewriting branches. protocol.lua:828-829 requires digits to match ^%-?(%d+)$ with #digits>=16, i.e. >=16 CONTIGUOUS digits in text — strictly witnessed by WIDE_RUN = %d x16. protocol.lua:815-817 requires lit:sub(2,7)==SENTINEL_JSON or lit:sub(2,2)==SENTINEL — strictly witnessed by the two plain finds. Every other path appends the exact input slice verbatim (line 818 the whole string literal including an unterminated tail, line 831 the whole number literal, line 835 one character), and each advances i past exactly what it appended, so with neither trigger present the scan IS the identity function. There is no third mutation site.",
"empirical_handcrafted": "1,306 adversarial cases across 15 integer literals x 8 tags x 2 encodings, 18 string literals x 7 tags x 2 encodings, plus every string-beside-wide-int pairing in both orders, digit runs split across literal boundaries (15+15 with no 16-run), nested sums/products, and both sentinel spellings including the decoy '\\u0001' (escaped backslash). Guarded vs unguarded output IDENTICAL.",
"empirical_random": "40,000 seeded random rows, 1-4 columns, 22 tags, both encodings, grammar biased hard toward digit runs near 16 and toward both sentinel spellings. IDENTICAL. Critically, 12,271 of those rows took the guard's early return and still matched the unguarded scan exactly — so the differential genuinely exercised the guard rather than always falling through it.",
"note": "WIDE_RUN drift IS caught by the suite: rebuilding it as WIDE_DIGITS+1 fails 'a 16-digit u128 one past 2^53 keeps its last digit'. The author's 'the two cannot drift' claim is test-backed."
}
}

section 6 identity fix correctness

{
"fixture_backed": "All four required properties hold on real bytes: one logical row (count=1 after join, after set_score, and after the named readback), one cache entry, cache.get and cache.find both resolving by the identity string, and #21's pairing producing exactly one update (events=1, type='update', old.score=0 -> row.score=42, both sides same key).",
"two_identities_one_batch": "WORKS. count=2, inserts=2, 2 events, get(A).name='A', get(B).name='B', find('sponsor',A)='A', find('sponsor',B)='B'. No cross-talk.",
"same_identity_resent_unchanged": "Frame 36 (named readback of the unchanged row) -> 1 event, type='insert', count stays 1. Correct: pairing scope is one batch and frame 36 carries no delete. Synthetically pairing a delete+insert of a byte-identical row in ONE batch -> 1 event, type='update', count=1 — which is exactly the doctrine README:469 states.",
"identity_also_a_unique_constraint": "WORKS, and this is #28's headline case. player.sponsor is a non-pk unique Identity column, so find('sponsor', db.identity) goes through the real index rather than delegating to get — resolves to the right row, follows the row through set_score (score=42), and drains with it on unsubscribe. find on the identity PK also works (delegates to get). The merged README's claim that this does not work is now WRONG and the PR correctly rewrites it.",
"readme_staleness": "ALREADY HANDLED. README:946-949 rewritten to 'resolves by the identity string, the same value db.identity holds, on a generated schema'; README:469 update row rewritten and its '#35' pointer removed; cache.lua's find caveat, cache.get's docstring and tables.lua:225-233 all rewritten. No stale user-facing text found. Two leftovers, both nits: README:955 still describes schema.new as validating '{ columns, pk?, unique? }' with no mention of types, and README:876 still says t:find delegates 'including its identity-column caveat' where the caveat is now conditional."
}

section 7 sanity

{
"suite": "6 passed, 0 failed (lua dev/test/runner.lua, repo root).",
"stylua": "CLEAN. nix develop .#ci --command stylua --check stdb dev/test (the exact CI command, .forgejo/workflows/ci.yml:43) exit 0, no diff; --check . also clean.",
"conformance": "PASS on first run, no flake. All 5 phases in both engine runs, including PR #48's new PHASE5. Identity persisted across the process restart.",
"mutations": "8 run, all reverted, git status clean afterward.",
"mutation_results": [
{
"m": "always decode_row (reverts the fix)",
"result": "CAUGHT — 4 passed 2 failed: 'an identity pk keys the store by its hex string: expected true, got false' + 'cache.get resolves the row by the identity STRING: expected true, got false'",
"matches_author": "YES, verbatim"
},
{
"m": "cache.new drops the typespace",
"result": "CAUGHT — 5/1: 'and the typespace named the sum's ordinal: expected busy, got nil'",
"matches_author": "YES, verbatim"
},
{
"m": "always decode_typed (drops the escape hatch)",
"result": "CAUGHT — 2/4, four files: 'decode_typed needs a schema spec with columns and types'",
"matches_author": "YES, verbatim"
},
{
"m": "guard drops the raw-SENTINEL witness",
"result": "NOT CAUGHT — 6 passed. Untested clause, but justified in-comment (rxi's json rejects a raw control byte; the clause exists for Defold's parser) and unreachable under the test json. Nit only."
},
{
"m": "guard drops the \u0001 witness",
"result": "CAUGHT — test_protocol 'a string already carrying two'"
},
{
"m": "WIDE_RUN drifts to 17 digits",
"result": "CAUGHT — 'a 16-digit u128 one past 2^53 keeps its last digit'"
},
{
"m": "connection.new stops forwarding the typespace",
"result": "NOT CAUGHT — 6 passed, 0 failed. See SF-4."
},
{
"m": "guard always short-circuits (scan never runs)",
"result": "CAUGHT — 3/3 across cache, fixtures and protocol"
}
],
"pr_body_accuracy": "The body says 'Five mutations, each reverted' but tables three. Minor."
}

findings

[
{
"id": "SF-1",
"severity": "should-fix",
"title": "protocol.schedule_at_to_seconds returns nil on the shape a cached ScheduleAt column now has",
"where": "stdb/protocol.lua:269-286 (accessor); README:988 and README:1019-1026 (docs)",
"problem": "decode_typed returns a schedule_at as {tag='Interval'|'Time', value=micros} (protocol.lua:1012-1015, README:714). schedule_at_to_seconds only tests v.Interval, v.Time, v[1]==0 and v[1]==1, so it falls through and returns nil. Verified on real captured all_types bytes in BOTH encodings (/tmp/final.lua): untyped -> ('interval', 1.5); typed -> nil, nil. It is the ONLY one of the six accessors that regresses. The PR's own README edit at 1019-1026 explicitly enumerates which typed representations still differ from the helpers' expectations and names only Uuid and ConnectionId — the one that actually BREAKS a helper is omitted, while README:988 still lists schedule_at_to_seconds as the way to read a ScheduleAt column. Not a blocker: widening a total accessor is not a frozen-surface break, so it is fixable after 1.0. But this PR is what creates the mismatch, so it should fix it.",
"fix": "Six lines before protocol.lua:273, symmetric with how every sibling accessor already takes both wire shapes:\n\tif v.tag == "Interval" then\n\t\treturn variant("interval", M.duration_to_seconds(v.value))\n\tend\n\tif v.tag == "Time" then\n\t\treturn variant("time", M.timestamp_to_seconds(v.value))\n\tend\nVERIFIED: applied locally, typed now returns ('interval', 1.5) in both encodings, suite stays 6 passed 0 failed, stylua clean. Reverted. Also add a schedule_at row to the README:1019-1026 paragraph."
},
{
"id": "SF-2",
"severity": "should-fix",
"title": "The player x0.76 — cheaper cost claim does not reproduce, and it is the number #30 will be decided on",
"where": "PR body cost table; stdb/protocol.lua:1112-1114 ('slightly CHEAPER on a row with special types')",
"problem": "Two independent runs on this machine, same generated schema, same rows, main vs branch: message x1.28/x1.24 (reproduces exactly), player x1.36/x1.39, profile x2.26/x2.31. At decode level on the actual captured bytes decode_typed is never cheaper in any table or encoding (worst: all_types positional x1.94). I swept five identity shapes and both encodings looking for the author's result and found none. The stated mechanism is also false for player: quote_big_ints' pattern requires an unquoted digit run after the magic key, which an identity column never has, so the gsub does no replacement work to save. Documenting the typed path as free-or-cheaper for identity-heavy tables, when it is ~1.4x, will mislead #30's frame-budget decision in the one direction that matters.",
"fix": "Re-measure and restate: message ~x1.25-1.30, player ~x1.35-1.40, and ADD profile (or all_types) at ~x2.3 — a table with a timestamp plus a sum plus a nested product is the realistic game shape and the real worst case, and the guard buys it nothing (61.7 unguarded vs 66.5 guarded, because micros is always 16 digits so the guard always fires). Drop the 'slightly CHEAPER' clause in protocol.lua:1113."
},
{
"id": "SF-3",
"severity": "should-fix",
"title": "schema.new does not validate types, so a hand-written types/columns skew raises per transaction on the main thread instead of at construction",
"where": "stdb/schema.lua:56-80; raise surfaces at stdb/cache.lua:216",
"problem": "types is now load-bearing AND hand-writable — the PR's own example hand-writes it (dev/example/main/smoke.script:91-97 adds player with types = {'identity','identity','string','i32'}). schema.new checks columns presence and shape, pk membership and unique membership, but never touches types. So types too short, too long, or types = {} all construct fine and then raise 'stdb.protocol: schema spec has N columns but M types' on the first row of that table. Verified: all three raise on the branch and all three were silent no-ops on main. That raise lands in cache.apply_database_update, which has NO pcall at any of its four call sites (connection.lua:559, 592, 639, 677) and propagates through handle_frame into Defold's websocket callback (init.lua:201) — established repo behaviour for a malformed frame, but this is a static schema mistake being reported at frame time, once per transaction, forever. README:955 still documents schema.new as validating '{ columns, pk?, unique? }'.",
"fix": "One check after schema.lua:65, same shape as the pk/unique checks that already live there:\n\tif def.types ~= nil then\n\t\tcheck_names(name, "types", def.types)\n\t\tif #def.types ~= #def.columns then\n\t\t\terror('stdb.schema: table "' .. name .. '" has ' .. #def.columns .. ' columns but ' .. #def.types .. ' types', 2)\n\t\tend\n\tend\nThat moves the diagnosis to the line the author wrote, which is what schema.lua exists for. Update README:955."
},
{
"id": "SF-4",
"severity": "should-fix",
"title": "The typespace forwarding — the one-line seam the whole feature hangs on — is untested",
"where": "stdb/connection.lua:23, stdb/init.lua:244",
"problem": "Deleting opts.typespace from connection.new's cache.new call leaves the suite at 6 passed, 0 failed. No test in test_connection.lua, test_init.lua or test_tables.lua passes a typespace at all. The author's own mutation table covers cache.new dropping it (caught) but not connection.new failing to supply it (not caught). test_init.lua already drives stdb.connect through a fake websocket, so both halves are cheaply testable.",
"fix": "Two assertions. In test_connection.lua: build a conn with typespace and a def carrying a sum_, feed one TransactionUpdate, assert row..tag. In test_init.lua: assert the connection's state.typespace is the object passed to stdb.connect. I verified both currently pass (/tmp/plumbing.lua) — the gap is defence, not behaviour."
},
{
"id": "SF-5",
"severity": "should-fix",
"title": "The README's new fixture-regeneration command block does not work",
"where": "README:1050-1053",
"problem": "cd dev/server && … node capture-fixtures.mjs then lua ../test/runner.lua. Run from dev/server, cwd is dev/server, so every test file fails: require('stdb.protocol') cannot resolve, and test_fixtures.lua:13 / :128 open 'dev/test/fixtures/session.jsonl' and 'dev/test/fixtures/chat_schema.lua' relative to cwd. Measured: 0 passed, 6 failed. (Main's version was also broken — cd server names a directory that no longer exists — so the PR half-fixed it and left the runner line wrong.)",
"fix": "cd dev/server && STDB_TOKEN="$TOKEN" node capture-fixtures.mjs\ncd ../.. && lua dev/test/runner.lua\nConfirmed working from the repo root: 6 passed, 0 failed."
},
{
"id": "DEC-1",
"severity": "should-fix",
"title": "Decide now, not after 1.0: connection_id and uuid freeze as decimal digits while identity freezes as canonical text",
"where": "README:712; stdb/protocol.lua:967-988",
"problem": "After this PR a consumer reading a cached row gets canonical '0x…' for an Identity but bare decimal digits for a ConnectionId or Uuid, and must call connection_id_to_hex / uuid_to_string to get anything a human or the server would recognise. That is not forced by losslessness: dec_to_hex32 (protocol.lua:351-378) is exact to 2^128, so returning '0x…' and the hyphenated UUID form would be equally lossless and would leave zero special-ID types needing a second call. These are the only two tags in the whole table that are not directly usable. Since this PR is what puts these values in front of every cached-row consumer, it is the moment the choice becomes expensive to revisit.",
"fix": "Either return canonical text for both (one dec_to_hex32 per such column, on columns that are rare — measure it against profile's x2.3 first), or state explicitly in README:712 why digits are the frozen answer despite hex being lossless. Right now the README documents the asymmetry without defending it."
},
{
"id": "NIT-1",
"severity": "nit",
"title": "'sum_ / product_ columns degrade' without a typespace is too broad",
"where": "README:682, stdb/cache.lua:79-80, PR body",
"problem": "A NAMED nested product does not degrade — profile.home.x still reads 1 with typespace=nil, because the named row already carries field names and the untypespaced product_ path returns them. Only a POSITIONAL nested product actually loses its names. Verified in /tmp/plumbing.lua.",
"fix": "Say 'sums, and nested products inside a positional row'."
},
{
"id": "NIT-2",
"severity": "nit",
"title": "vec_u8 goes from hex text to raw bytes on the cached path, and only the per-tag table says so",
"where": "README:706",
"problem": "A bytes column was the server's 8-char hex text '0001feff' on main and is now 4 raw bytes (verified #v==4, bytes 00 01 fe ff). The per-tag table documents it correctly, but it is a visible retype for anyone who was printing or concatenating a bytes column and it is not mentioned among the compatibility notes. One clause would cover it."
},
{
"id": "NIT-3",
"severity": "nit",
"title": "Leftover doc rows not updated with the rest",
"where": "README:876, README:955",
"problem": "t:find still says it delegates 'including its identity-column caveat' where the caveat is now conditional on the schema kind; schema.new's row still lists '{ columns, pk?, unique? }' with no types. Both are one-line edits and both sit next to text this PR did rewrite."
},
{
"id": "NIT-4",
"severity": "nit",
"title": "PR body says five mutations, tables three",
"where": "PR body Verification section",
"problem": "Cosmetic, but the three tabled ones reproduce verbatim and are the load-bearing ones, so the other two are just unrecorded. Name them or say three."
}
]

unmet acceptance criteria

NONE from #35. All five work items and the acceptance criterion are met and independently verified against captured bytes plus live conformance PHASE5. Every finding above is either something this change newly introduces (SF-1, SF-3, SF-5), a claim about it that does not hold (SF-2), defence it lacks (SF-4), or a freeze-time decision it makes implicitly (DEC-1).

what is right about this pr

The core judgement is correct and well argued. Routing on def.types per table (never per row, never sniffing the pk's type) is the only rule statable in a sentence, and the rejected alternative — coercing only the key — really would have produced row.owner ~= row.identity for one identity. The escape hatch is byte-identical across all 25 captured database_updates, not merely 'unchanged in spirit'. Identity freezing to the hex string is exactly right: it is byte-identical to db.identity, verified through handle_frame on real frames. The typespace-as-second-argument argument is backed by facts I confirmed against a live 2.8.2, including the shadowing. The fixture was captured as an appended second socket so frames 1-29 stay byte-identical, which is the right call and correctly explained. The quote_wide_ints guard's no-false-negative claim survives a serious attempt to break it — structural analysis plus 41,306 differential cases with 12,271 of them actually taking the early return. And the three tabled mutations reproduce verbatim.

reviewer notes

{
"heads": "Reviewed 2770953 (feat/35-identity-pk) against origin/main d646507. PR #47's merge to bf7c75b landed mid-review; none of my probes touch set_send, close, schedule_reconnect, one_off_query or the pending-settle paths #47 rewrote — they exercise connection.new (line 20-23), handle_frame and the cache only — so no finding mixes the two heads.",
"artifacts": "/tmp/dump.lua + before.txt/after.txt (before/after table), /tmp/accessors.lua + /tmp/final.lua (accessor matrix, identity equality), /tmp/probe.lua (escape hatch, boundaries, u64, identity probes), /tmp/plumbing.lua (typespace plumbing, facades), /tmp/fuzz_guard.lua + /tmp/fuzz_rand.lua + /tmp/noguard (guard differential), /tmp/bench2.lua /tmp/bench3.lua /tmp/bench4.lua (cost), /tmp/loud.lua /tmp/loud2.lua (loud-u64 predicate), /tmp/mutate.sh (mutations).",
"hygiene": "Nothing edited, pushed, merged or commented. Local checkout /tmp/review-48 left clean (git status empty); the SF-1 patch and all 8 mutations were applied and reverted. Servers on 4600 and 4700 stopped; port 3000 never touched."
}

## Review — automated reviewer, posted for the record **Verdict:** APPROVE WITH CHANGES — closes #35 in full; no blocker. One accessor regression, one materially wrong cost claim, one missing validation, one untested wiring seam, one doc command broken. All five fixes are local and none touches the frozen row representation. ### closes 35 { "answer": "YES", "acceptance_criterion": "MET, on real captured bytes. Frames 33 (join_game, positional Identity pk) then 34 (set_score, pk-preserving) through cache.apply_database_update with the generated schema: count=1 throughout, #events=1, events[1].type='update', old.score=0, row.score=42, old.identity==row.identity. Re-verified independently of the author's test in /tmp/probe.lua.", "work_items": "All five checked. (1) representation decided + documented (README 647-783); (2) identity table works — verified; (3) `player` added to dev/server/index.js and captured as frames 30-36 (second socket, own IdentityToken); (4) u64>2^53 documented + pinned 3 ways; (5) README:469 `update` row rewritten, no stale '#35' user-facing text remains.", "live_proof": "Native conformance on STDB_PORT=4700: PHASE5 PASS in BOTH engine runs — 'SMOKE: PASS PHASE5 identity-keyed table: one row throughout, one update event, resolved by 0xc2002e0a…'. Flake #39 not hit." } ### section 1 compatibility break { "method": "Same 25 captured database_updates from dev/test/fixtures/session.jsonl replayed through cache.apply_database_update on origin/main (d646507) and on 2770953, same generated schema, both encodings. Scripts /tmp/dump.lua, /tmp/before.txt, /tmp/after.txt.", "message_u64_string_string": "UNCHANGED in both encodings, across all 18 captured message rows. Byte-identical dumps.", "player_identity_identity_string_i32": "identity, sponsor: {__identity__='0x…'} (named) / ['0x…'] (positional) -> '0x…' string. name, score unchanged.", "profile": "NO captured rows exist in the fixture (grep 'profile' session.jsonl -> 0). Measured on synthetic rows matching the documented wire shapes (/tmp/plumbing.lua): owner identity->string, joined timestamp->number, nickname opt_string->value-or-nil, tags vec_string->unchanged, status sum->{tag='busy'}, home product->{x,y} from both encodings.", "all_types_full_table": [ { "col": "id, flag, i_8..u_32, f_32, f_64, text, words", "before": "number/boolean/string/array", "after": "identical", "changed": false }, { "col": "i_64", "before": "number(-9.2233720368547758e+18)", "after": "identical — still rounds", "changed": false }, { "col": "u_64", "before": "number(1.8446744073709552e+19)", "after": "identical — still rounds", "changed": false }, { "col": "i_128", "before": "number(-1.7014118346046923e+38)", "after": "string('-170141183460469231731687303715884105728')", "changed": true }, { "col": "u_128", "before": "number(3.4028236692093846e+38)", "after": "string('340282366920938463463374607431768211455')", "changed": true }, { "col": "i_256, u_256", "before": "string('0x…') verbatim", "after": "identical", "changed": false }, { "col": "bytes (vec_u8)", "before": "string('0001feff') — hex TEXT, 8 chars", "after": "4 RAW BYTES 00 01 fe ff (verified #v==4, v:byte(i))", "changed": true, "note": "anyone printing a bytes column now prints binary" }, { "col": "status (sum_Status)", "before": "[1,{}] named / [1,[]] positional", "after": "{tag='busy'}", "changed": true }, { "col": "kind (sum_Kind)", "before": "[2,{x=5,y=-6}] named / [2,[5,-6]] positional", "after": "{tag='pointy',value={x=5,y=-6}}", "changed": true }, { "col": "point (product_Point)", "before": "{x=3,y=-4} named / [3,-4] positional", "after": "{x=3,y=-4} from BOTH", "changed": "positional only — named was already this" }, { "col": "opt_int (opt_i32)", "before": "[0,7]", "after": "7", "changed": true }, { "col": "opt_point", "before": "[0,{x=8,y=9}] named / [0,[8,9]] positional", "after": "{x=8,y=9}", "changed": true }, { "col": "opt_opt (None)", "before": "[1,{}]", "after": "nil — the field is ABSENT from the row table", "changed": true }, { "col": "owner (identity)", "before": "{__identity__='0x…'} named / ['0x…'] positional", "after": "'0x…' string", "changed": true }, { "col": "conn (connection_id)", "before": "{__connection_id__='218…537'} named / [2.1803579067259356e+38] positional — ROUNDED", "after": "'218035790672593550386615283461651752537' exact digits", "changed": true, "note": "positional case GAINS precision it never had" }, { "col": "at (timestamp)", "before": "{__timestamp_micros_since_unix_epoch__=n} / [n]", "after": "n", "changed": true }, { "col": "span (time_duration)", "before": "{__time_duration_micros__=n} / [n]", "after": "n", "changed": true }, { "col": "sched (schedule_at)", "before": "[0,{__time_duration_micros__=1500000}] / [0,[1500000]]", "after": "{tag='Interval',value=1500000}", "changed": true }, { "col": "uid (uuid)", "before": "{__uuid__='1512…040'} / [1.5123660752041709e+36] ROUNDED", "after": "'1512366075204170947332355369683137040' exact", "changed": true, "note": "positional gains precision" } ], "identity_vs_db_identity": "MATCHES EXACTLY. connection.lua:541 sets c.identity = protocol.unwrap(msg.identity) -> the bare hex string. Driven through handle_frame on real frames 30 then 33 (/tmp/final.lua): db.identity = '0xc200ff15…085b4' (string, 66 chars); cache.get(state,'player',db.identity).name = 'yuris'; row.identity == db.identity -> true; same-length -> true. Named readback (frame 36) lands on the SAME key, count stays 1. find(sponsor, db.identity) and find(identity, db.identity) both resolve. Verdict: RIGHT representation to freeze.", "sums_and_nested_products": "NAMED now, both of them. Sums -> {tag='<variant>', value=<decoded>} with the server's lowercased variant names ('idle','busy','nothing','pointy'); ScheduleAt uses library-cased 'Interval'/'Time'. Nested products -> {x=,y=} from BOTH encodings, so the positional/named asymmetry documented at README 619-625 is erased at the consumer. Correct and the whole point of #46.", "accessor_matrix": { "method": "/tmp/accessors.lua and /tmp/final.lua — every #23/#37 accessor against the old NAMED shape, the old POSITIONAL shape, and the new typed shape, on real fixture bytes for all_types in both encodings.", "timestamp_to_seconds": "WORKS on the typed shape. 1787163313.2782 from all three. `scalar()` passes a non-table straight through.", "duration_to_seconds": "WORKS. 90061 from all three.", "identity_to_hex": "WORKS, idempotent — '0x…' in, '0x…' out (matches ^0x%x+$).", "connection_id_to_hex": "WORKS on typed, and now works where it used to fail: untyped positional returned nil, typed returns 0xa4082a73357fa6a74fad283d8974e659. Strict improvement.", "uuid_to_string": "WORKS on typed; same improvement (untyped positional -> nil, typed -> 01234567-89ab-cdef-fedc-ba9876543210).", "schedule_at_to_seconds": "*** BROKEN ON THE TYPED SHAPE *** — returns nil in BOTH encodings. See finding SF-1.", "unwrap": "no-op passthrough on typed values. Fine.", "verdict": "5 of 6 accessors survive; connection_id/uuid get strictly better. Exactly one regresses." }, "usable_without_further_conversion": { "identity": "YES — canonical 0x hex", "timestamp": "YES — micros number", "time_duration": "YES — micros number", "schedule_at": "shape yes, but the accessor rejects it (SF-1)", "connection_id": "NO — exact DECIMAL digits; needs connection_id_to_hex to be human/comparable-with-server", "uuid": "NO — exact DECIMAL digits; needs uuid_to_string", "raised_for_decision": "This asymmetry is being frozen: `identity` decodes to canonical text but `connection_id`/`uuid` decode to decimal digits. It is NOT a losslessness constraint — dec_to_hex32 (protocol.lua:351-378) is exact to 2^128, so decode_typed could return '0x…' / hyphenated form losslessly and remove the only two tags needing a second call. Cost is one digit-array division loop per such column, on columns that are rare. Worth deciding NOW rather than after 1.0." } } ### section 2 escape hatch { "byte_identity": "VERIFIED. Hand-written defs (column names copied from the generated schema, `types` removed) replayed over all 25 fixture database_updates on main and on the branch; full row dumps compared with cmp -> IDENTICAL, 25/25 lines. /tmp/probe.lua, /tmp/m2.txt vs /tmp/b2.txt.", "boundary_cases": [ { "case": "no `types`", "main": "OK, wrapper preserved", "branch": "OK, identical", "verdict": "escape hatch intact" }, { "case": "`types` full length", "main": "OK (types ignored)", "branch": "OK, coerced", "verdict": "as designed" }, { "case": "`types` TOO SHORT (2 for 4 columns)", "main": "OK, silently ignored", "branch": "RAISES 'stdb.protocol: schema spec has 4 columns but 2 types' from cache.lua:216", "verdict": "NEW raise — see SF-3" }, { "case": "`types` TOO LONG (5 for 4)", "main": "OK", "branch": "RAISES same", "verdict": "NEW raise" }, { "case": "`types` = {} (present but EMPTY)", "main": "OK", "branch": "RAISES 'schema spec has 4 columns but 0 types'", "verdict": "NEW raise; 'presence of types is the whole test' is literally true and an empty list is not the no-op an author would expect" }, { "case": "table absent from schema (a view, #42)", "main": "ins=1, row = {} (empty)", "branch": "ins=1, row = {} — IDENTICAL", "verdict": "correct, unchanged, #42 untouched" } ], "mixed_schema": "BEHAVES PER-TABLE AS CLAIMED. One state, two defs, same bytes into both: typed.owner = 'string:0xaa' and get(typed,'0xaa') resolves; untyped.owner = {1='0xaa'} and get(untyped,'0xaa') misses. On main both were the wrapper and both missed. Routing is per-table, never per-row (cache.lua:210-221)." } ### section 3 typespace signature { "claim_1_server_accepts_a_table_named_typespace": "TRUE, verified against a live 2.8.2 on port 4600 with a purpose-built module. `spacetime publish` exit 0, no warning. Schema JSON lists `typespace` in `tables` with primary_key [0], its btree index and its Unique constraint intact. The server SERVES it in all three shapes: InitialSubscription (named row), TransactionUpdate (positional row), OneOffQueryResponse (named). `SELECT * FROM typespace` accepted, no SubscriptionError.", "claim_2_generator_emits_both": "TRUE and it is the crux — dev/tools/gen-schema.sh emits `tables['typespace'] = { columns={'id','label'}, types={'u64','string'}, pk='id', … }` at line 59 AND a top-level `typespace = { Kind=…, Point=…, Status=… }` at line 131 of the SAME generated file. No name-mangling anywhere.", "claim_3_schema_new_rejects_it": "TRUE. `defs.typespace = gen.typespace; schema.new(defs)` raises 'stdb.schema: table \"typespace\" needs a `columns` list (order matters)' (schema.lua:62-64). Control run without the injected key passes, so the failure is caused by the key alone.", "claim_4_shadowing_is_real": "TRUE. Plain table assignment, no metatable, no error at assignment: defs.typespace goes from the real table def (columns={'id','label'}, pk='id') to the typespace object; `.columns` and `.pk` become nil and the real def is unreachable. Ordering only picks which of the two is destroyed. So the rejected design cannot even be constructed — one branch raises, the other drops data. The two-argument signature is justified by facts, not a hypothetical.", "plumbing": "VERIFIED END TO END (/tmp/plumbing.lua). connection.new({typespace=gen.typespace}) -> state.typespace is the same object; frames driven through handle_frame produce typed rows; profile.status.tag='busy', profile.home='1,2', profile.joined is a number. init.lua:244 forwards `typespace = opts.typespace` and init.lua:139 documents it.", "no_typespace_degradation": "CLAIM HOLDS, with one correction. Generated schema, typespace=nil: identity pk still resolves (get by hex string -> true), so the #35 fix does NOT depend on the typespace. Sums degrade to the raw [ordinal,payload] (status.tag=nil, status[1]=1). BUT a NAMED nested product does not degrade — profile.home.x=1 still works, because the named row already carries field names and the untypespaced product_ path passes it through. Only a POSITIONAL nested product actually degrades. README:682 and cache.lua:79-80 say 'sum_<Name> / product_<Name> columns degrade' without that qualifier. Nit.", "tables_facades": "SEE THE SAME DECODED ROWS. conn.tables.player:on_insert fires with row.identity as a 'string' starting 0xffff; :find('sponsor', db.identity) and :find('identity', db.identity) both resolve; :count() correct. No divergence between the cache and the facade." } ### section 4 u64 bound { "pinned_cases_reproduced": [ { "case": "2^53 and 2^53+1 (u64 pk)", "result": "count=1, inserts=2, events=2, get(9007199254740992).note='second', NOTHING RAISES" }, { "case": "2^53-2 and 2^53-1 (u64 pk)", "result": "count=2 — the boundary is real, not a blanket 'u64 is broken'" }, { "case": "u128::MAX-1 and u128::MAX (u128 pk)", "result": "count=2, get by the exact 39-digit string -> 'second'. On main the same input gave count=1 and get->nil, so this is a fix, not just a workaround." } ], "extra_probe": "u64::MAX-1 and u64::MAX also collide: count=1, later row wins. Lua 5.1 confirms 2^53 == 2^53+1 -> true and tonumber('9007199254740993') == 2^53 -> true.", "consequence_confirmed": "Two server keys -> one Lua key; cache merges; last arrival wins; cache.count is 1 while counts.inserts is 2, so the consumer's own event stream and the cache disagree and nothing says why.", "judgement": "The REPRESENTATION decision is right — retyping every u64 column to string to fix a 9.0e15-row ceiling is a worse trade, and the README's snowflake/hash caveat is the correct guidance. The SILENCE is wrong to freeze. This repo's stated posture is that failures are loud or absent (README:45-49 'it is silently wrong rather than loud' is listed as a TRAP; cache.find raises rather than returning nil for a non-unique column precisely so a lookup that can never succeed says so). A silent row merge with a count mismatch is exactly the class of thing the repo elsewhere refuses to ship.", "cheapest_loud_alternative": "A one-time warning, costing nothing on the happy path, at protocol.lua:957-965. The NUMERIC branch already holds BOTH the exact digit string and the double it produced: `local digits = rescued_digits(v); if digits then return tonumber(digits) end`. Add one format+compare there: if string.format('%.0f', tonumber(digits)) ~= digits then the double lost a digit. VERIFIED EXACT in Lua 5.1 (/tmp/loud.lua, /tmp/loud2.lua): fires on 9007199254740993, 18446744073709551615, -9007199254740993, -9223372036854775807; does NOT fire on 9007199254740992 (2^53, representable), 9007199254740994, 10000000000000000, 1234567890123457; zero false positives over 5000 exactly-representable integers; fires on exactly the odd integers above 2^53. Cost is bounded to values the rescue already caught (>=16 digits), which the quote_wide_ints guard already establishes is rare — so a table of plain scalars pays nothing. Sharper than the README's framing too: it flags actual precision loss, not 'above 2^53'. Recommend emitting it once per process (or as an on_event) rather than raising, so an existing consumer is not broken by a diagnostic." } ### section 5 cost claims { "method": "20k-row named subscription readback per table, best-of-9 os.clock, same generated schema and same rows against origin/main (before) and 2770953 (after). Realistic full-entropy 64-hex-char identities, NOT %064x (60 leading zeros is a 16-digit run and spuriously triggers the guard — my first attempt made that mistake and inflated the typed path). /tmp/bench2.lua, two independent runs.", "measured": [ { "table": "message (u64,string,string)", "main": "6.39 / 6.30 us/row", "branch": "8.16 / 7.81 us/row", "ratio": "x1.28 / x1.24", "author_claim": "x1.28", "verdict": "REPRODUCES EXACTLY" }, { "table": "player (identity,identity,string,i32)", "main": "17.36 / 17.01 us/row", "branch": "23.68 / 23.71 us/row", "ratio": "x1.36 / x1.39", "author_claim": "x0.76 — CHEAPER", "verdict": "DOES NOT REPRODUCE. Wrong by a factor of ~1.8 and wrong in DIRECTION." }, { "table": "profile (identity, timestamp, opt, vec, sum, nested product)", "main": "29.44 / 28.84 us/row", "branch": "66.46 / 66.50 us/row", "ratio": "x2.26 / x2.31", "author_claim": "absent from the PR's table", "verdict": "the realistic worst case, and undocumented" } ], "decode_level_on_real_fixture_bytes": "/tmp/bench4.lua, decode_row vs decode_typed on the actual captured rows: message positional x1.19, message named x1.27, player positional x1.33, player named x1.26, all_types positional x1.94, all_types named x1.85. decode_typed is NEVER cheaper, in any table, in either encoding. The author's 'decode_row 28.18 vs decode_typed 22.10 us/row' does not reproduce; I also swept identity shapes and encodings (/tmp/bench3.lua: real-id named x1.27, random-id named x1.21, %064x named x1.82, real-id positional x1.32, %064x positional x1.91) and found no configuration where the typed path wins. The stated mechanism ('quote_big_ints gsub costs more than the guarded scan plus per-column coercion') is false for player: quote_big_ints' pattern '\"(__[%a_]+__)\":(%d+)' cannot even match an identity column (the value is quoted), so it does no replacement work there, while typed decoding still pays two special-type coercions.", "first_cut_x2_14": "DIRECTIONALLY CONFIRMED, magnitude differs. Guard stripped from quote_wide_ints (/tmp/noguard): message 11.89 us/row = x1.86 vs main; player 35.31 = x2.03. So the guard is worth ~1.5x on message and player and the 'x2.14 before the guard' story is real. Note it buys profile nothing — 61.73 unguarded vs 66.50 guarded, because a micros timestamp is always 16 digits so the guard always fires and only adds a wasted find.", "guard_false_negative_hunt": { "conclusion": "NO FALSE NEGATIVE FOUND. The claim survives both a structural argument and 41,306 differential cases.", "structural": "quote_wide_ints has exactly two rewriting branches. protocol.lua:828-829 requires `digits` to match ^%-?(%d+)$ with #digits>=16, i.e. >=16 CONTIGUOUS digits in `text` — strictly witnessed by WIDE_RUN = %d x16. protocol.lua:815-817 requires lit:sub(2,7)==SENTINEL_JSON or lit:sub(2,2)==SENTINEL — strictly witnessed by the two plain finds. Every other path appends the exact input slice verbatim (line 818 the whole string literal including an unterminated tail, line 831 the whole number literal, line 835 one character), and each advances i past exactly what it appended, so with neither trigger present the scan IS the identity function. There is no third mutation site.", "empirical_handcrafted": "1,306 adversarial cases across 15 integer literals x 8 tags x 2 encodings, 18 string literals x 7 tags x 2 encodings, plus every string-beside-wide-int pairing in both orders, digit runs split across literal boundaries (15+15 with no 16-run), nested sums/products, and both sentinel spellings including the decoy '\\\\u0001' (escaped backslash). Guarded vs unguarded output IDENTICAL.", "empirical_random": "40,000 seeded random rows, 1-4 columns, 22 tags, both encodings, grammar biased hard toward digit runs near 16 and toward both sentinel spellings. IDENTICAL. Critically, 12,271 of those rows took the guard's early return and still matched the unguarded scan exactly — so the differential genuinely exercised the guard rather than always falling through it.", "note": "WIDE_RUN drift IS caught by the suite: rebuilding it as WIDE_DIGITS+1 fails 'a 16-digit u128 one past 2^53 keeps its last digit'. The author's 'the two cannot drift' claim is test-backed." } } ### section 6 identity fix correctness { "fixture_backed": "All four required properties hold on real bytes: one logical row (count=1 after join, after set_score, and after the named readback), one cache entry, cache.get and cache.find both resolving by the identity string, and #21's pairing producing exactly one `update` (events=1, type='update', old.score=0 -> row.score=42, both sides same key).", "two_identities_one_batch": "WORKS. count=2, inserts=2, 2 events, get(A).name='A', get(B).name='B', find('sponsor',A)='A', find('sponsor',B)='B'. No cross-talk.", "same_identity_resent_unchanged": "Frame 36 (named readback of the unchanged row) -> 1 event, type='insert', count stays 1. Correct: pairing scope is one batch and frame 36 carries no delete. Synthetically pairing a delete+insert of a byte-identical row in ONE batch -> 1 event, type='update', count=1 — which is exactly the doctrine README:469 states.", "identity_also_a_unique_constraint": "WORKS, and this is #28's headline case. `player.sponsor` is a non-pk unique Identity column, so find('sponsor', db.identity) goes through the real index rather than delegating to get — resolves to the right row, follows the row through set_score (score=42), and drains with it on unsubscribe. find on the identity PK also works (delegates to get). The merged README's claim that this does not work is now WRONG and the PR correctly rewrites it.", "readme_staleness": "ALREADY HANDLED. README:946-949 rewritten to 'resolves by the identity string, the same value db.identity holds, on a generated schema'; README:469 `update` row rewritten and its '#35' pointer removed; cache.lua's find caveat, cache.get's docstring and tables.lua:225-233 all rewritten. No stale user-facing text found. Two leftovers, both nits: README:955 still describes schema.new as validating '{ columns, pk?, unique? }' with no mention of `types`, and README:876 still says t:find delegates 'including its identity-column caveat' where the caveat is now conditional." } ### section 7 sanity { "suite": "6 passed, 0 failed (lua dev/test/runner.lua, repo root).", "stylua": "CLEAN. `nix develop .#ci --command stylua --check stdb dev/test` (the exact CI command, .forgejo/workflows/ci.yml:43) exit 0, no diff; `--check .` also clean.", "conformance": "PASS on first run, no flake. All 5 phases in both engine runs, including PR #48's new PHASE5. Identity persisted across the process restart.", "mutations": "8 run, all reverted, git status clean afterward.", "mutation_results": [ { "m": "always decode_row (reverts the fix)", "result": "CAUGHT — 4 passed 2 failed: 'an identity pk keys the store by its hex string: expected true, got false' + 'cache.get resolves the row by the identity STRING: expected true, got false'", "matches_author": "YES, verbatim" }, { "m": "cache.new drops the typespace", "result": "CAUGHT — 5/1: 'and the typespace named the sum's ordinal: expected busy, got nil'", "matches_author": "YES, verbatim" }, { "m": "always decode_typed (drops the escape hatch)", "result": "CAUGHT — 2/4, four files: 'decode_typed needs a schema spec with columns and types'", "matches_author": "YES, verbatim" }, { "m": "guard drops the raw-SENTINEL witness", "result": "NOT CAUGHT — 6 passed. Untested clause, but justified in-comment (rxi's json rejects a raw control byte; the clause exists for Defold's parser) and unreachable under the test json. Nit only." }, { "m": "guard drops the \\u0001 witness", "result": "CAUGHT — test_protocol 'a string already carrying two'" }, { "m": "WIDE_RUN drifts to 17 digits", "result": "CAUGHT — 'a 16-digit u128 one past 2^53 keeps its last digit'" }, { "m": "connection.new stops forwarding the typespace", "result": "NOT CAUGHT — 6 passed, 0 failed. See SF-4." }, { "m": "guard always short-circuits (scan never runs)", "result": "CAUGHT — 3/3 across cache, fixtures and protocol" } ], "pr_body_accuracy": "The body says 'Five mutations, each reverted' but tables three. Minor." } ### findings [ { "id": "SF-1", "severity": "should-fix", "title": "protocol.schedule_at_to_seconds returns nil on the shape a cached ScheduleAt column now has", "where": "stdb/protocol.lua:269-286 (accessor); README:988 and README:1019-1026 (docs)", "problem": "decode_typed returns a schedule_at as {tag='Interval'|'Time', value=micros} (protocol.lua:1012-1015, README:714). schedule_at_to_seconds only tests v.Interval, v.Time, v[1]==0 and v[1]==1, so it falls through and returns nil. Verified on real captured all_types bytes in BOTH encodings (/tmp/final.lua): untyped -> ('interval', 1.5); typed -> nil, nil. It is the ONLY one of the six accessors that regresses. The PR's own README edit at 1019-1026 explicitly enumerates which typed representations still differ from the helpers' expectations and names only Uuid and ConnectionId — the one that actually BREAKS a helper is omitted, while README:988 still lists schedule_at_to_seconds as the way to read a ScheduleAt column. Not a blocker: widening a total accessor is not a frozen-surface break, so it is fixable after 1.0. But this PR is what creates the mismatch, so it should fix it.", "fix": "Six lines before protocol.lua:273, symmetric with how every sibling accessor already takes both wire shapes:\n\tif v.tag == \"Interval\" then\n\t\treturn variant(\"interval\", M.duration_to_seconds(v.value))\n\tend\n\tif v.tag == \"Time\" then\n\t\treturn variant(\"time\", M.timestamp_to_seconds(v.value))\n\tend\nVERIFIED: applied locally, typed now returns ('interval', 1.5) in both encodings, suite stays 6 passed 0 failed, stylua clean. Reverted. Also add a schedule_at row to the README:1019-1026 paragraph." }, { "id": "SF-2", "severity": "should-fix", "title": "The `player x0.76 — cheaper` cost claim does not reproduce, and it is the number #30 will be decided on", "where": "PR body cost table; stdb/protocol.lua:1112-1114 ('slightly CHEAPER on a row with special types')", "problem": "Two independent runs on this machine, same generated schema, same rows, main vs branch: message x1.28/x1.24 (reproduces exactly), player x1.36/x1.39, profile x2.26/x2.31. At decode level on the actual captured bytes decode_typed is never cheaper in any table or encoding (worst: all_types positional x1.94). I swept five identity shapes and both encodings looking for the author's result and found none. The stated mechanism is also false for player: quote_big_ints' pattern requires an unquoted digit run after the magic key, which an identity column never has, so the gsub does no replacement work to save. Documenting the typed path as free-or-cheaper for identity-heavy tables, when it is ~1.4x, will mislead #30's frame-budget decision in the one direction that matters.", "fix": "Re-measure and restate: message ~x1.25-1.30, player ~x1.35-1.40, and ADD profile (or all_types) at ~x2.3 — a table with a timestamp plus a sum plus a nested product is the realistic game shape and the real worst case, and the guard buys it nothing (61.7 unguarded vs 66.5 guarded, because micros is always 16 digits so the guard always fires). Drop the 'slightly CHEAPER' clause in protocol.lua:1113." }, { "id": "SF-3", "severity": "should-fix", "title": "schema.new does not validate `types`, so a hand-written types/columns skew raises per transaction on the main thread instead of at construction", "where": "stdb/schema.lua:56-80; raise surfaces at stdb/cache.lua:216", "problem": "`types` is now load-bearing AND hand-writable — the PR's own example hand-writes it (dev/example/main/smoke.script:91-97 adds player with types = {'identity','identity','string','i32'}). schema.new checks columns presence and shape, pk membership and unique membership, but never touches types. So types too short, too long, or `types = {}` all construct fine and then raise 'stdb.protocol: schema spec has N columns but M types' on the first row of that table. Verified: all three raise on the branch and all three were silent no-ops on main. That raise lands in cache.apply_database_update, which has NO pcall at any of its four call sites (connection.lua:559, 592, 639, 677) and propagates through handle_frame into Defold's websocket callback (init.lua:201) — established repo behaviour for a malformed frame, but this is a static schema mistake being reported at frame time, once per transaction, forever. README:955 still documents schema.new as validating '{ columns, pk?, unique? }'.", "fix": "One check after schema.lua:65, same shape as the pk/unique checks that already live there:\n\tif def.types ~= nil then\n\t\tcheck_names(name, \"types\", def.types)\n\t\tif #def.types ~= #def.columns then\n\t\t\terror('stdb.schema: table \"' .. name .. '\" has ' .. #def.columns .. ' columns but ' .. #def.types .. ' types', 2)\n\t\tend\n\tend\nThat moves the diagnosis to the line the author wrote, which is what schema.lua exists for. Update README:955." }, { "id": "SF-4", "severity": "should-fix", "title": "The typespace forwarding — the one-line seam the whole feature hangs on — is untested", "where": "stdb/connection.lua:23, stdb/init.lua:244", "problem": "Deleting `opts.typespace` from connection.new's cache.new call leaves the suite at 6 passed, 0 failed. No test in test_connection.lua, test_init.lua or test_tables.lua passes a typespace at all. The author's own mutation table covers cache.new dropping it (caught) but not connection.new failing to supply it (not caught). test_init.lua already drives stdb.connect through a fake websocket, so both halves are cheaply testable.", "fix": "Two assertions. In test_connection.lua: build a conn with typespace and a def carrying a sum_<Name>, feed one TransactionUpdate, assert row.<sum>.tag. In test_init.lua: assert the connection's state.typespace is the object passed to stdb.connect. I verified both currently pass (/tmp/plumbing.lua) — the gap is defence, not behaviour." }, { "id": "SF-5", "severity": "should-fix", "title": "The README's new fixture-regeneration command block does not work", "where": "README:1050-1053", "problem": "`cd dev/server && … node capture-fixtures.mjs` then `lua ../test/runner.lua`. Run from dev/server, cwd is dev/server, so every test file fails: require('stdb.protocol') cannot resolve, and test_fixtures.lua:13 / :128 open 'dev/test/fixtures/session.jsonl' and 'dev/test/fixtures/chat_schema.lua' relative to cwd. Measured: 0 passed, 6 failed. (Main's version was also broken — `cd server` names a directory that no longer exists — so the PR half-fixed it and left the runner line wrong.)", "fix": "cd dev/server && STDB_TOKEN=\"$TOKEN\" node capture-fixtures.mjs\ncd ../.. && lua dev/test/runner.lua\nConfirmed working from the repo root: 6 passed, 0 failed." }, { "id": "DEC-1", "severity": "should-fix", "title": "Decide now, not after 1.0: connection_id and uuid freeze as decimal digits while identity freezes as canonical text", "where": "README:712; stdb/protocol.lua:967-988", "problem": "After this PR a consumer reading a cached row gets canonical '0x…' for an Identity but bare decimal digits for a ConnectionId or Uuid, and must call connection_id_to_hex / uuid_to_string to get anything a human or the server would recognise. That is not forced by losslessness: dec_to_hex32 (protocol.lua:351-378) is exact to 2^128, so returning '0x…' and the hyphenated UUID form would be equally lossless and would leave zero special-ID types needing a second call. These are the only two tags in the whole table that are not directly usable. Since this PR is what puts these values in front of every cached-row consumer, it is the moment the choice becomes expensive to revisit.", "fix": "Either return canonical text for both (one dec_to_hex32 per such column, on columns that are rare — measure it against profile's x2.3 first), or state explicitly in README:712 why digits are the frozen answer despite hex being lossless. Right now the README documents the asymmetry without defending it." }, { "id": "NIT-1", "severity": "nit", "title": "'sum_<Name> / product_<Name> columns degrade' without a typespace is too broad", "where": "README:682, stdb/cache.lua:79-80, PR body", "problem": "A NAMED nested product does not degrade — profile.home.x still reads 1 with typespace=nil, because the named row already carries field names and the untypespaced product_ path returns them. Only a POSITIONAL nested product actually loses its names. Verified in /tmp/plumbing.lua.", "fix": "Say 'sums, and nested products inside a positional row'." }, { "id": "NIT-2", "severity": "nit", "title": "vec_u8 goes from hex text to raw bytes on the cached path, and only the per-tag table says so", "where": "README:706", "problem": "A `bytes` column was the server's 8-char hex text '0001feff' on main and is now 4 raw bytes (verified #v==4, bytes 00 01 fe ff). The per-tag table documents it correctly, but it is a visible retype for anyone who was printing or concatenating a bytes column and it is not mentioned among the compatibility notes. One clause would cover it." }, { "id": "NIT-3", "severity": "nit", "title": "Leftover doc rows not updated with the rest", "where": "README:876, README:955", "problem": "t:find still says it delegates 'including its identity-column caveat' where the caveat is now conditional on the schema kind; schema.new's row still lists '{ columns, pk?, unique? }' with no `types`. Both are one-line edits and both sit next to text this PR did rewrite." }, { "id": "NIT-4", "severity": "nit", "title": "PR body says five mutations, tables three", "where": "PR body Verification section", "problem": "Cosmetic, but the three tabled ones reproduce verbatim and are the load-bearing ones, so the other two are just unrecorded. Name them or say three." } ] ### unmet acceptance criteria NONE from #35. All five work items and the acceptance criterion are met and independently verified against captured bytes plus live conformance PHASE5. Every finding above is either something this change newly introduces (SF-1, SF-3, SF-5), a claim about it that does not hold (SF-2), defence it lacks (SF-4), or a freeze-time decision it makes implicitly (DEC-1). ### what is right about this pr The core judgement is correct and well argued. Routing on def.types per table (never per row, never sniffing the pk's type) is the only rule statable in a sentence, and the rejected alternative — coercing only the key — really would have produced row.owner ~= row.identity for one identity. The escape hatch is byte-identical across all 25 captured database_updates, not merely 'unchanged in spirit'. Identity freezing to the hex string is exactly right: it is byte-identical to db.identity, verified through handle_frame on real frames. The typespace-as-second-argument argument is backed by facts I confirmed against a live 2.8.2, including the shadowing. The fixture was captured as an appended second socket so frames 1-29 stay byte-identical, which is the right call and correctly explained. The quote_wide_ints guard's no-false-negative claim survives a serious attempt to break it — structural analysis plus 41,306 differential cases with 12,271 of them actually taking the early return. And the three tabled mutations reproduce verbatim. ### reviewer notes { "heads": "Reviewed 2770953 (feat/35-identity-pk) against origin/main d646507. PR #47's merge to bf7c75b landed mid-review; none of my probes touch set_send, close, schedule_reconnect, one_off_query or the pending-settle paths #47 rewrote — they exercise connection.new (line 20-23), handle_frame and the cache only — so no finding mixes the two heads.", "artifacts": "/tmp/dump.lua + before.txt/after.txt (before/after table), /tmp/accessors.lua + /tmp/final.lua (accessor matrix, identity equality), /tmp/probe.lua (escape hatch, boundaries, u64, identity probes), /tmp/plumbing.lua (typespace plumbing, facades), /tmp/fuzz_guard.lua + /tmp/fuzz_rand.lua + /tmp/noguard (guard differential), /tmp/bench2.lua /tmp/bench3.lua /tmp/bench4.lua (cost), /tmp/loud.lua /tmp/loud2.lua (loud-u64 predicate), /tmp/mutate.sh (mutations).", "hygiene": "Nothing edited, pushed, merged or commented. Local checkout /tmp/review-48 left clean (git status empty); the SF-1 patch and all 8 mutations were applied and reverted. Servers on 4600 and 4700 stopped; port 3000 never touched." }

yuris force-pushed feat/35-identity-pk from 2770953276 to bf7d84d999 2026-08-20 03:16:31 +00:00 Compare

yuris force-pushed feat/35-identity-pk from bf7d84d999 to f526caeb93 2026-08-20 03:18:28 +00:00 Compare

yuris merged commit 6908d478c3 into main 2026-08-20 03:22:25 +00:00

yuris referenced this pull request from a commit 2026-08-20 03:22:27 +00:00

cache: route rows through the typed decoder so an Identity key works (#48)

yuris referenced this pull request 2026-08-20 04:49:29 +00:00

v1.0.0 release checklist #24

yuris referenced this pull request 2026-08-20 04:49:29 +00:00

v1.0.0 release checklist #24

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area:defold

Defold engine integration

  

area:docs

Documentation and examples

  

area:infra

Flake, CI, secretspec

  

area:protocol

Wire protocol conformance

  

area:testing

Test harness and fixtures

  

kind:bug

Something broken

  

kind:chore

Maintenance

  

phase:breadth

UnsubscribeMulti, procedures, BSATN

  

phase:core

Core SDK layers (protocol/cache/connection)

  

phase:hardening

Reconnect, token persistence, errors

  

phase:slice

Vertical slice: first playable end-to-end

No labels

area:defold

area:docs

area:infra

area:protocol

area:testing

kind:bug

kind:chore

phase:breadth

phase:core

phase:hardening

phase:slice

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/spacetimedb-defold!48

No description provided.