protocol: convert special types explicitly #37

Merged

yuris merged 1 commit from feat/23-special-types into main 2026-08-19 18:00:39 +00:00

Conversation 1 Commits 1 Files changed 6 +682 -14

yuris commented 2026-08-19 16:46:45 +00:00

Owner

Copy link

Closes #23.

Six explicit conversion accessors in stdb/protocol.lua, beside unwrap and decode_payload where the rest of the wire-encoding knowledge lives:

accessor	returns
protocol.timestamp_to_seconds(v)	float seconds, for os.date / os.time
protocol.duration_to_seconds(v)	float seconds
protocol.schedule_at_to_seconds(v)	"interval" or "time", plus the converted value
protocol.connection_id_to_hex(v)	0x + 32 lowercase hex digits, exact
protocol.identity_to_hex(v)	0x… hex string
protocol.uuid_to_string(v)	01020304-0506-0708-090a-0b0c0d0e0f10

Every name states its output unit, because the unit is the point of #23.

The bug this closes

Timestamp is microseconds. Every other timestamp a Defold developer touches is seconds (socket.gettime, os.time), so os.date("%H:%M", row.created_at) does not error — it produces a date about 56 million years out (os.date("!%Y-%m-%d", 1787158263465067) → 56634760-11-18). Same trap for TimeDuration.

Two shapes, not one

A special-type column is a one-field product, and it arrives in two shapes depending on the row that carries it:

named row        "joined":{"__timestamp_micros_since_unix_epoch__":1787158263465067}
positional row   "joined" -> [1787158263465067]

A named row serializes from the row layout, so the field name survives (serialize_named_product). A positional row serializes an untyped ProductValue — TransactionUpdate's inserts: Arc<[ProductValue]> — and AlgebraicValue::Product forwards to the seq product serializer recursively, so the wrapper name is gone at every depth. Both observed live on 2.8.2, same table, same column.

Accepting only the named form meant timestamp_to_seconds(row.created_at) worked on rows from InitialSubscription and returned nil on rows from TransactionUpdate — the on_event path a game spends its life in, and exactly the path #23 exists to fix. Per #40's capture it is also every view row, subscription readbacks included.

The fix is one guard in scalar, the choke point all six accessors route through, and the arity check is load-bearing rather than defensive dressing:

if x == nil and #v == 1 and type(v[1]) ~= "table" then
	x = v[1]
end

serialize_variant's positional branch is serialize_tuple(2), so a sum is always exactly [tag, payload] and can never be mistaken for a one-field column. Without #v == 1, Option<Timestamp> None [1,{}] reads the variant tag as microseconds and returns 1e-06, and Some [0,[n]] returns 0 — silent wrong numbers, the exact class this PR exists to prevent. Tested both ways: the positional column resolves, every Option shape gives nil.

connection_id_to_hex is exact — all 32 digits

The server sends ConnectionId as a bare 128-bit JSON number, so json.decode rounds it to the nearest double and only 14 of 32 hex digits survive. Captured live from 2.8.2:

wire decimal      135936128599631372233289231990769713196
server's to_hex   0x66445897597a38fb947e3b5c1d06002c
padded to 32      0x66445897597a38000000000000000000
                                ^^^^^^^^^^^^^^^^^^ fabricated

Padding produced a string of precisely the canonical width with a fabricated tail — the shape you paste into spacetime, key a table on, or write to a save file. Two ids differing by 1 on the wire rendered identically. And the retained digits are not reliably right either: rounding is to nearest, so the committed fixture's id renders as …bef508 under the old path where the truth is …bef5077b.

This needs no BSATN codec and no big-integer JSON decoder, which is what makes it fixable rather than a documented limit: the digits are in the raw frame text. decode_server and decode_row quote u128 fields before json.decode ever runs, and the accessor converts the exact decimal by repeated division by 16 over a digit array (every intermediate stays under 160, so it is exact in a double at any width).

Verified against four ConnectionIds captured live this round plus the committed fixture, each cross-checked against Node BigInt:

decimal on the wire	hex
135936128599631372233289231990769713196	0x66445897597a38fb947e3b5c1d06002c
167170533507420896873558886572199149804	0x7dc3dfa6391cdb0016558ed835ae80ec
164534092079369019845196319157764641020	0x7bc81d0d1444b98bc7a4ae86911e50fc
10414371664771645740041721761875515547	0x07d5bc2a8a5cb45acd01357ed52e589b
304429789785738994598068895799762753346 (fixture)	0xe5070b78bef5077ba9abfbd79e381742

That is byte-for-byte the server's own output: to_hex is hex::encode(as_be_byte_array()) = u128::to_be_bytes, i.e. %032x of the decimal, so there is no byte-swap subtlety.

What it will not do is guess. nil for a number already past 2^53 (its low bits are gone and no downstream arithmetic returns them), nil past 2^128 rather than silently widening to 33 digits, and nil for a u128 column on a positional row, which carries no field name for the rescue to match. The old string branch is gone with it: a ConnectionId is never hex on this protocol (connection_id.rs:13 — bare decimal, always), and returning "hello" unchanged from a function contracted to return 32 hex digits was the failure mode most likely to be mistaken for success.

Breaking change, taken deliberately before the freeze: connection.lua's c.connection_id is now canonical hex rather than a rounded number. Exact, and the same shape c.identity already has, so the two credentials read alike.

Decisions worth a reviewer's opinion

• nil on bad input, never raise — with the real justification. The earlier "an Option column legitimately decodes to nil" was false: sums in rows are positional, so an Option column is the table [0, v] or [1, {}]. The actual reason is the split protocol.lua already keeps: frame-level functions (decode_server, decode_row, encode_client) raise, because a malformed frame is a protocol violation that should kill the connection; value-level functions (unwrap, decode_payload) are total. These six are value-level, so they match unwrap, which they sit beside. Callers check for nil. An Option<Timestamp> column needs its own unwrap step first — which unwrap cannot yet do for the positional sum form.
• Naming is consistent, not half verb half noun. schedule_at → schedule_at_to_seconds, connection_id → connection_id_to_hex. Once frozen, consistency beats any individual name.
• One string rule across the family: each accessor takes only the shapes its own wire type produces. Time types take a number. ConnectionId/Uuid take exact decimal digits, or a number within 2^53 — not hex, which never occurs. Identity takes a hex string, because that is its wire form. Nothing passes through unvalidated.
• identity_to_hex added so the family is complete, and so unwrap stays a generic wrapper-stripper rather than doubling as the Identity accessor. It also closes the two-shapes gap for Identity, where unwrap returned a bare table on a positional row. unwrap itself is deliberately not extended: it is type-blind and cannot know whether a bare number is micros or a ConnectionId.
• Uuid included rather than deferred. #23 names it, gen-schema.sh already tags __uuid__ columns, and it is the same u128 with the same precision decision being made once, here. Output is the canonical hyphenated form, matching Uuid's Display.
• schedule_at_to_seconds handles both sum encodings. Ordinals follow variant declaration order — 0 = Interval, 1 = Time, read off crates/lib/src/scheduler.rs:59-63. The positional form is what rows carry; the named { Interval = … } form is unreachable on this protocol today and is kept, with a comment saying so, for schema JSON and message envelopes.
• variant() returns a single nil, not ("interval", nil). "Could not interpret that" is then uniform across the family, so if kind then use(secs) end is always safe. The two-shapes fix removed the only shape that triggered the alternative in practice.
• No automatic per-column coercion in decode_row. That needs #25's column types and changes the type of row field values, which is frozen at 1.0 — tracked in #35. Explicit accessors only.

Precision

A current-era microsecond value (~1.787e15) is exactly representable — a double holds integers to 2^53 ≈ 9.007e15. Only the /1e6 is inexact, and whole-second resolution survives. The ceiling is 2^53 micros = 2255-06-05, computed and asserted in a test so the doc comment cannot rot.

string.format("%x", n) returns "0" for a 128-bit value in Lua 5.1, which is why the hex is hand-rolled. Note the new implementation never runs that arithmetic on an out-of-range double at all — it divides a decimal digit array, so the math.fmod negative-remainder hazard and the %x trap are structurally gone rather than guarded. (A previous revision of this description claimed a test pinned the %x result directly. It did not — the pin was indirect, via the expected hex string. Now moot.)

Discoverability

#23's point was that a helper nobody finds does not prevent the bug. The table is now reachable from where a reader actually stands:

• the microseconds trap is the third Defold-specific trap, in the section for exactly this class of finding;
• ### stdb.cache, the section on reading rows, points down to it;
• the ## Usage example says so inline, where a timestamp column would go.

The table itself is corrected: it previously claimed the decoded row value is the named wrapper, which is true only for named rows.

Verification

5 passed, 0 failed; stylua clean; native conformance PASS on the first run, both phases including identity persistence across a process restart.

Both blockers reproduced before the change and re-checked after:

	before	after
timestamp_to_seconds on a TransactionUpdate column	nil	1787158263.4651
naive positional fix, Option<Timestamp> None	1e-06	nil
naive positional fix, Option<Timestamp> Some	0	nil
ConnectionId from the live capture	0x66445897597a38000000000000000000	0x66445897597a38fb947e3b5c1d06002c
two ids one apart on the wire	identical strings	…06002c / …06002d
connection_id("hello")	"hello"	nil
connection_id(2^128)	33 digits	nil

Falsifiability by nine independent sabotages, each restored afterwards:

sabotage	failure
drop the #v == 1 arity guard	Option<Timestamp> None is not a Timestamp: expected nil, got 1e-06
drop the positional-column branch	positional Timestamp column: expected 1787157191.2252, got nil
stop quoting u128 digits in decode_server	connection_id survives json.decode as digit text: expected string, got number
stop quoting u128 digits in decode_row	ConnectionId column on a named row is exact: … got nil
restore the string pass-through	hex string is not a wire form: expected nil, got 0x0000…
drop the 32-digit ceiling	2^128 does not fit 32 digits: … got 0x1000… (33 digits)
reintroduce string.format("%x") on the whole value	low digits are real, not padding: expected true, got false
swap the ScheduleAt ordinals	ordinal 0 is Interval: expected interval, got time
/1e6 → /1e3	timestamp wrapper to seconds: expected 1787157191.2252, got 1787157191225.2

The fixture assertion is deliberately not a pinned hex literal: the server mints a fresh ConnectionId per socket, so a literal goes stale on every recapture for no behavioural reason. It asserts instead that the digits reach Lua as text, that the capture is genuinely past 2^53, and that the hex does not end in the 14 zeros the padding path produced — which is what the %x sabotage above trips. The exact-value proofs live in test_protocol.lua on frozen captures no recapture can invalidate.

Deliberately not done

No wrapper objects. No seconds→micros encode direction — nothing in the SDK sends a Timestamp yet, so the encode shape would be a guess. unwrap still cannot strip a positional Option; that is pre-existing and belongs with #26's schema-driven decoder.

Closes #23. Six explicit conversion accessors in `stdb/protocol.lua`, beside `unwrap` and `decode_payload` where the rest of the wire-encoding knowledge lives: | accessor | returns | |---|---| | `protocol.timestamp_to_seconds(v)` | float seconds, for `os.date` / `os.time` | | `protocol.duration_to_seconds(v)` | float seconds | | `protocol.schedule_at_to_seconds(v)` | `"interval"` or `"time"`, plus the converted value | | `protocol.connection_id_to_hex(v)` | `0x` + 32 lowercase hex digits, **exact** | | `protocol.identity_to_hex(v)` | `0x…` hex string | | `protocol.uuid_to_string(v)` | `01020304-0506-0708-090a-0b0c0d0e0f10` | Every name states its output unit, because the unit is the point of #23. ## The bug this closes `Timestamp` is **microseconds**. Every other timestamp a Defold developer touches is seconds (`socket.gettime`, `os.time`), so `os.date("%H:%M", row.created_at)` does not error — it produces a date about **56 million years** out (`os.date("!%Y-%m-%d", 1787158263465067)` → `56634760-11-18`). Same trap for `TimeDuration`. ## Two shapes, not one A special-type column is a one-field product, and it arrives in two shapes depending on the row that carries it: ``` named row "joined":{"__timestamp_micros_since_unix_epoch__":1787158263465067} positional row "joined" -> [1787158263465067] ``` A named row serializes from the row layout, so the field name survives (`serialize_named_product`). A positional row serializes an untyped `ProductValue` — `TransactionUpdate`'s `inserts: Arc<[ProductValue]>` — and `AlgebraicValue::Product` forwards to the *seq* product serializer **recursively**, so the wrapper name is gone at every depth. Both observed live on 2.8.2, same table, same column. Accepting only the named form meant `timestamp_to_seconds(row.created_at)` worked on rows from `InitialSubscription` and returned **`nil`** on rows from `TransactionUpdate` — the `on_event` path a game spends its life in, and exactly the path #23 exists to fix. Per #40's capture it is also every *view* row, subscription readbacks included. The fix is one guard in `scalar`, the choke point all six accessors route through, and the arity check is load-bearing rather than defensive dressing: ```lua if x == nil and #v == 1 and type(v[1]) ~= "table" then x = v[1] end ``` `serialize_variant`'s positional branch is `serialize_tuple(2)`, so a sum is *always* exactly `[tag, payload]` and can never be mistaken for a one-field column. Without `#v == 1`, `Option<Timestamp>` `None` `[1,{}]` reads the variant tag as microseconds and returns `1e-06`, and `Some` `[0,[n]]` returns `0` — silent wrong numbers, the exact class this PR exists to prevent. Tested both ways: the positional column resolves, every Option shape gives `nil`. ## `connection_id_to_hex` is exact — all 32 digits The server sends `ConnectionId` as a bare 128-bit JSON *number*, so `json.decode` rounds it to the nearest double and only 14 of 32 hex digits survive. Captured live from 2.8.2: ``` wire decimal 135936128599631372233289231990769713196 server's to_hex 0x66445897597a38fb947e3b5c1d06002c padded to 32 0x66445897597a38000000000000000000 ^^^^^^^^^^^^^^^^^^ fabricated ``` Padding produced a string of precisely the canonical width with a fabricated tail — the shape you paste into `spacetime`, key a table on, or write to a save file. Two ids differing by **1** on the wire rendered identically. And the retained digits are not reliably right either: rounding is to nearest, so the committed fixture's id renders as `…bef508` under the old path where the truth is `…bef5077b`. This needs no BSATN codec and no big-integer JSON decoder, which is what makes it fixable rather than a documented limit: **the digits are in the raw frame text**. `decode_server` and `decode_row` quote u128 fields before `json.decode` ever runs, and the accessor converts the exact decimal by repeated division by 16 over a digit array (every intermediate stays under 160, so it is exact in a double at any width). Verified against four ConnectionIds captured live this round plus the committed fixture, each cross-checked against Node `BigInt`: | decimal on the wire | hex | |---|---| | `135936128599631372233289231990769713196` | `0x66445897597a38fb947e3b5c1d06002c` | | `167170533507420896873558886572199149804` | `0x7dc3dfa6391cdb0016558ed835ae80ec` | | `164534092079369019845196319157764641020` | `0x7bc81d0d1444b98bc7a4ae86911e50fc` | | `10414371664771645740041721761875515547` | `0x07d5bc2a8a5cb45acd01357ed52e589b` | | `304429789785738994598068895799762753346` (fixture) | `0xe5070b78bef5077ba9abfbd79e381742` | That is byte-for-byte the server's own output: `to_hex` is `hex::encode(as_be_byte_array())` = `u128::to_be_bytes`, i.e. `%032x` of the decimal, so there is no byte-swap subtlety. **What it will not do is guess.** `nil` for a number already past 2^53 (its low bits are gone and no downstream arithmetic returns them), `nil` past 2^128 rather than silently widening to 33 digits, and `nil` for a u128 column on a *positional* row, which carries no field name for the rescue to match. The old string branch is gone with it: a ConnectionId is never hex on this protocol (`connection_id.rs:13` — bare decimal, always), and returning `"hello"` unchanged from a function contracted to return 32 hex digits was the failure mode most likely to be mistaken for success. **Breaking change, taken deliberately before the freeze:** `connection.lua`'s `c.connection_id` is now canonical hex rather than a rounded number. Exact, and the same shape `c.identity` already has, so the two credentials read alike. ## Decisions worth a reviewer's opinion - **`nil` on bad input, never raise — with the real justification.** The earlier "an `Option` column legitimately decodes to `nil`" was false: sums in rows are positional, so an `Option` column is the *table* `[0, v]` or `[1, {}]`. The actual reason is the split `protocol.lua` already keeps: frame-level functions (`decode_server`, `decode_row`, `encode_client`) raise, because a malformed frame is a protocol violation that should kill the connection; value-level functions (`unwrap`, `decode_payload`) are total. These six are value-level, so they match `unwrap`, which they sit beside. Callers check for `nil`. An `Option<Timestamp>` column needs its own unwrap step first — which `unwrap` cannot yet do for the positional sum form. - **Naming is consistent, not half verb half noun.** `schedule_at` → `schedule_at_to_seconds`, `connection_id` → `connection_id_to_hex`. Once frozen, consistency beats any individual name. - **One string rule across the family:** each accessor takes only the shapes its own wire type produces. Time types take a number. `ConnectionId`/`Uuid` take exact decimal digits, or a number within 2^53 — not hex, which never occurs. `Identity` takes a hex string, because that *is* its wire form. Nothing passes through unvalidated. - **`identity_to_hex` added so the family is complete**, and so `unwrap` stays a generic wrapper-stripper rather than doubling as the Identity accessor. It also closes the two-shapes gap for Identity, where `unwrap` returned a bare table on a positional row. `unwrap` itself is deliberately not extended: it is type-blind and cannot know whether a bare number is micros or a ConnectionId. - **`Uuid` included rather than deferred.** #23 names it, `gen-schema.sh` already tags `__uuid__` columns, and it is the same u128 with the same precision decision being made once, here. Output is the canonical hyphenated form, matching `Uuid`'s `Display`. - **`schedule_at_to_seconds` handles both sum encodings.** Ordinals follow variant declaration order — `0` = Interval, `1` = Time, read off `crates/lib/src/scheduler.rs:59-63`. The positional form is what rows carry; the named `{ Interval = … }` form is unreachable on this protocol today and is kept, with a comment saying so, for schema JSON and message envelopes. - **`variant()` returns a single `nil`, not `("interval", nil)`.** "Could not interpret that" is then uniform across the family, so `if kind then use(secs) end` is always safe. The two-shapes fix removed the only shape that triggered the alternative in practice. - **No automatic per-column coercion in `decode_row`.** That needs #25's column types and changes the type of row field values, which is frozen at 1.0 — tracked in #35. Explicit accessors only. ## Precision A current-era microsecond value (~1.787e15) is *exactly* representable — a double holds integers to 2^53 ≈ 9.007e15. Only the `/1e6` is inexact, and whole-second resolution survives. The ceiling is 2^53 micros = **2255-06-05**, computed and asserted in a test so the doc comment cannot rot. `string.format("%x", n)` returns `"0"` for a 128-bit value in Lua 5.1, which is why the hex is hand-rolled. Note the new implementation never runs that arithmetic on an out-of-range double at all — it divides a decimal *digit array*, so the `math.fmod` negative-remainder hazard and the `%x` trap are structurally gone rather than guarded. (A previous revision of this description claimed a test pinned the `%x` result directly. It did not — the pin was indirect, via the expected hex string. Now moot.) ## Discoverability #23's point was that a helper nobody finds does not prevent the bug. The table is now reachable from where a reader actually stands: - the microseconds trap is the **third** Defold-specific trap, in the section for exactly this class of finding; - `### stdb.cache`, the section on reading rows, points down to it; - the `## Usage` example says so inline, where a timestamp column would go. The table itself is corrected: it previously claimed the decoded row value *is* the named wrapper, which is true only for named rows. ## Verification `5 passed, 0 failed`; stylua clean; native conformance **PASS** on the first run, both phases including identity persistence across a process restart. Both blockers reproduced before the change and re-checked after: | | before | after | |---|---|---| | `timestamp_to_seconds` on a `TransactionUpdate` column | `nil` | `1787158263.4651` | | naive positional fix, `Option<Timestamp>` `None` | `1e-06` | `nil` | | naive positional fix, `Option<Timestamp>` `Some` | `0` | `nil` | | ConnectionId from the live capture | `0x66445897597a38000000000000000000` | `0x66445897597a38fb947e3b5c1d06002c` | | two ids one apart on the wire | identical strings | `…06002c` / `…06002d` | | `connection_id("hello")` | `"hello"` | `nil` | | `connection_id(2^128)` | 33 digits | `nil` | Falsifiability by nine independent sabotages, each restored afterwards: | sabotage | failure | |---|---| | drop the `#v == 1` arity guard | `Option<Timestamp> None is not a Timestamp: expected nil, got 1e-06` | | drop the positional-column branch | `positional Timestamp column: expected 1787157191.2252, got nil` | | stop quoting u128 digits in `decode_server` | `connection_id survives json.decode as digit text: expected string, got number` | | stop quoting u128 digits in `decode_row` | `ConnectionId column on a named row is exact: … got nil` | | restore the string pass-through | `hex string is not a wire form: expected nil, got 0x0000…` | | drop the 32-digit ceiling | `2^128 does not fit 32 digits: … got 0x1000…` (33 digits) | | reintroduce `string.format("%x")` on the whole value | `low digits are real, not padding: expected true, got false` | | swap the ScheduleAt ordinals | `ordinal 0 is Interval: expected interval, got time` | | `/1e6` → `/1e3` | `timestamp wrapper to seconds: expected 1787157191.2252, got 1787157191225.2` | The fixture assertion is deliberately **not** a pinned hex literal: the server mints a fresh ConnectionId per socket, so a literal goes stale on every recapture for no behavioural reason. It asserts instead that the digits reach Lua as *text*, that the capture is genuinely past 2^53, and that the hex does not end in the 14 zeros the padding path produced — which is what the `%x` sabotage above trips. The exact-value proofs live in `test_protocol.lua` on frozen captures no recapture can invalidate. ## Deliberately not done No wrapper objects. No seconds→micros encode direction — nothing in the SDK sends a `Timestamp` yet, so the encode shape would be a guess. `unwrap` still cannot strip a positional `Option`; that is pre-existing and belongs with #26's schema-driven decoder.

yuris added 1 commit 2026-08-19 16:46:45 +00:00

protocol: convert special types explicitly (#23) ... 27234934a2

Timestamp and TimeDuration are microseconds on the wire; every clock a
Defold game already touches is seconds. `os.date("%H:%M", row.created_at)`
on a raw Timestamp does not error, it prints a date ~50,000 years out.
That is the silent-wrong-answer class this SDK has been careful about
elsewhere (#8's dropped header, #17's silent new identity), and unwrap
made it worse by removing the wrapper name that hinted at the unit.

Four pure helpers beside unwrap, where the rest of the wire-encoding
knowledge lives: timestamp_to_seconds, duration_to_seconds, schedule_at
and connection_id. Each takes the wrapper table or an already-unwrapped
bare value, so running unwrap first is not punished, and each returns nil
rather than raising for nil or a wrong shape: an Option column
legitimately decodes to nil, raising would force a pcall at every call
site, and a nil errors loudly at the point of misuse instead of producing
a plausible wrong number.

schedule_at accepts both JSON sum encodings. The named { Interval = v } /
{ Time = v } form is what a hand-written fixture looks like; the wire uses
the positional ordinal form [0, v] / [1, v], and ordinals follow variant
declaration order, so 0 is Interval and 1 is Time.

connection_id needs hand-rolled hex. The server sends ConnectionId as a
bare 128-bit JSON *number*, so it is already rounded to a double before
the SDK sees it, and string.format("%x", n) on a value that large returns
"0" outright — silently, which is the whole problem. The manual digits are
the exact hex of the parsed double: high bits real, low bits zero. A test
pins that, so the "%x" trap cannot be reintroduced as a simplification.

No automatic per-column coercion in decode_row. That needs column types
the generated schema does not carry (#25) and would change decoded values
under existing consumers, which is breaking; tracked in #35.

The 2^53 exact-integer ceiling is documented in both protocol.lua and the
README, with a test asserting the date it falls on (2255-06-05) so the
comment cannot rot. Current-era values are held exactly; only the
division to seconds is inexact, and whole-second resolution survives it.

yuris commented 2026-08-19 17:05:10 +00:00

Author

Owner

Copy link

Review — automated reviewer, posted for the record

Review — PR #37 feat/23-special-types

All numeric claims checked by running them in the pinned Lua 5.1 from nix develop .#ci. Wire-shape claims checked against the pinned SpacetimeDB v2.8.2 source (/tmp/librarian-stdb, git describe = v2.8.2) and against the live server at 127.0.0.1:3000. Baseline: 5 passed, 0 failed, stylua clean.

Verdict: request changes. The arithmetic in this PR is correct — I could not break it. What is wrong is one shape assumption and one honesty problem, and both get frozen at 1.0.

---

BLOCKER 1 — timestamp_to_seconds returns nil for exactly the bug #23 exists to fix

stdb/protocol.lua:141-146 (scalar)

A special-type column arrives in two shapes depending on which frame carried the row, and the helpers accept only one of them.

• Subscription frames (InitialSubscription, Subscribe*Applied) serialize rows from a RowRef, which carries the row layout → typed → serialize_named_product → {"__timestamp_micros_since_unix_epoch__": n}.
  crates/table/src/table.rs:2034-2041 → crates/sats/src/ser/impls.rs:226-228
• TransactionUpdate serializes rows from inserts: Arc<[ProductValue]> — an untyped value. Field names are dropped, and AlgebraicValue::Product(prod) => prod.serialize(ser) means they are dropped recursively, so the one-field Timestamp product becomes the array [micros].
  crates/core/src/host/module_host.rs:127-128,159-160 → crates/sats/src/ser/impls.rs:129-131,152-156

Both shapes observed live on 2.8.2, same table, same column:

InitialSubscription (ws)  "joined":{"__timestamp_micros_since_unix_epoch__":1787158263465067}
untyped serializer (SQL)  "joined" -> [1787158263465067]        "owner" -> ["0xc200d7a1…"]
fixture session.jsonl:4   TransactionUpdate inserts -> ["[2,\"vic\",\"live insert\"]"]

Run end to end through the SDK (decode_row + the new helper):

InitialSubscription (named)      timestamp_to_seconds -> 1787158263.4651   os.date -> 2026-08-19T16:51:03
TransactionUpdate (positional)   timestamp_to_seconds -> nil
                                 unwrap(owner)        -> table: 0x…

So protocol.timestamp_to_seconds(row.created_at) works on rows from the initial subscription and returns nil on rows from live updates — the on_insert/on_event path a game actually spends its life in. That is the worst available failure mode for a helper whose whole purpose is a silent-wrong-answer bug: the helper silently declines to work on half the inputs, in a way unit tests built from hand-written wrappers cannot see.

This is not an unknown-unknown for this codebase. protocol.lua:262-264 already says decode_row "Handles both encodings the server uses: positional arrays (TransactionUpdate) and named objects (InitialSubscription)." The dual encoding is known at the row level and simply was not carried down to the column value level.

Fix — one guard in scalar, the shared choke point all four helpers route through:

local function scalar(v, field)
	if type(v) == "table" then
		local x = v[field]
		-- TransactionUpdate rows are untyped ProductValues: field names are dropped
		-- recursively, so a Timestamp column arrives as the 1-element array [micros].
		-- Guarded on #v == 1 so a positional sum ([tag, payload]) never matches.
		if x == nil and #v == 1 and type(v[1]) == "number" then
			x = v[1]
		end
		return x
	end
	return v
end

The #v == 1 guard is load-bearing, not defensive dressing. I tried the naive x = v[1] first and it produces silent wrong numbers on Option columns, which is the exact class #23 is about:

naive: Option<Timestamp> None [1,{}]     -> 1e-06   <-- reads the sum tag as micros
naive: Option<Timestamp> Some [0,[n]]    -> 0       <-- reads the sum tag as micros
guarded: both                            -> nil

Guarded version, verified:

Timestamp column (TransactionUpdate) [n]   -> 1787158263.4651
named wrapper                              -> 1787158263.4651
bare number                                -> 1787158263.4651
Option<Timestamp> None [1,{}]              -> nil
Option<Timestamp> Some [0,[n]]             -> nil
wrong wrapper {__time_duration_micros__=5} -> nil
positional ScheduleAt [0,[1500000]]        -> interval, 1.5
positional ConnectionId [42]               -> 0x…002a

Applied to a scratch clone: 5 passed, 0 failed, stylua clean. Fully backwards compatible.

The README table needs the same correction — it currently states the decoded row value is { __timestamp_micros_since_unix_epoch__ = n }, which is true only for subscription rows.

---

BLOCKER 2 — connection_id pads lossy data into a precise-looking identifier, and the lossiness is avoidable

stdb/protocol.lua:220-234

First, the author's own claims are all true. I verified each:

• string.format("%x", 36062078055823991097399387891464995893) → "0". Confirmed. (%d on it gives -9223372036854775808, also garbage.) The hand-rolled loop is genuinely necessary.
• 0x1b214db8a0d796 * 2^72 == <parsed value> → true. 0x1b214db8a0d796 is exactly 53 significant bits, as it must be.
• The hand-rolled hex is arithmetically correct. Differential test, 74 exact-integer doubles (0, 15, 16, 2^31±1, 2^32, 2^40, 2^52, 2^53, 40 random below 2^53, 20 random scaled by 2^3…2^60) against Node BigInt(...).toString(16): 0 mismatches. math.fmod/math.floor by 16 on a double are exact, as claimed.
• math.fmod(-17, 16) is indeed -1 in this Lua, so the negative-remainder hazard is real — and it is correctly closed by the v < 0 guard on line 228. -1, -0.5, -inf, and NaN all return nil. The guard is load-bearing and there is a test for it. Good.

The problem is not the arithmetic. It is that the arithmetic is applied to already-corrupted input and the result is then padded to 32 digits, which is precisely the canonical width of an exact ConnectionId.

Captured live from 2.8.2 this session:

wire decimal   135936128599631372233289231990769713196
true hex       0x66445897597a38fb947e3b5c1d06002c   <- server's ConnectionId::to_hex()
this PR emits  0x66445897597a38000000000000000000
                              ^^^^^^^^^^^^^^^^^^ 18 digits fabricated

14 of 32 digits are real. Same result on the fixture value (0x1b214db8a0d79617a81f1948ffa10435 true, 0x1b214db8a0d796000000000000000000 emitted). The true hex is %032x of the u128 — to_hex = hex::encode(as_be_byte_array()) = u128::to_be_bytes (crates/lib/src/connection_id.rs:104-119), so there is no byte-swap subtlety; the emitted string is directly comparable to the server's, and it is 56% right.

So: not honest. The value is shaped exactly like the thing you paste into spacetime, key a table on, write to a save file, or send to a support channel. 0x…000000000000000000 does not read as "unknown low bits"; it reads as a ConnectionId that happens to end in zeros. Even the server's own abbreviated form (to_abbreviated_hex, first 16 hex chars) would be wrong in its last two digits.

The docstring's mitigations don't hold up either:

• "the high ~53 bits are the wire value" — only when rounding happens to go down, as it did for the fixture. Rounding is round-to-nearest, so the retained digits can be wrong too. Demonstrated: a value whose true high digits end …d796 emits …d797.
• "Fine … to compare against another id decoded the same way" — mostly true for random ids, but not unconditionally: two ids differing by 1 on the wire produce an identical string. Verified with json.decode on …302976 and …302977.
• "That needs a BSATN codec or a JSON decoder that keeps big integers as text" — this is false, and it is what turns the finding from "document a limitation" into a blocker. Neither is needed. SpacetimeDB documents the hazard and the workaround in the same breath (crates/lib/src/connection_id.rs:13-16: "In JSON, a ConnectionId is represented as a BARE DECIMAL number. This requires some care when deserializing", linking the JS BigInt question). decode_server holds the raw frame text, so the digits are right there before json.decode ever sees them. One gsub plus ~14 lines of Lua 5.1 recovers the value exactly:

-- before json.decode, in decode_server:
data = data:gsub('("__connection_id__":)(%d+)', '%1"%2"')

-- exact decimal-string -> 32 hex digits, repeated division by 16 on a digit array
local function dec_to_hex(s)
	local d = {}
	for i = 1, #s do d[i] = tonumber(s:sub(i, i)) end
	local out = {}
	repeat
		local rem, nz = 0, false
		for i = 1, #d do
			local cur = rem * 10 + d[i]
			d[i] = math.floor(cur / 16)
			if d[i] ~= 0 then nz = true end
			rem = cur % 16
		end
		out[#out + 1] = string.format("%x", rem)
	until not nz
	local h = ""
	for i = #out, 1, -1 do h = h .. out[i] end
	return "0x" .. string.rep("0", 32 - #h) .. h
end

Verified: recovers 0x66445897597a38fb947e3b5c1d06002c from the live decimal, 0x1b214db8a0d79617a81f1948ffa10435 from the fixture, and agrees with Node BigInt on every value tried. 42 → 0x…002a, 0 → all zeros.

Recommendation, in preference order:

1. Do the gsub in decode_server and make connection_id convert the decimal string exactly. This is the only option that makes a frozen function named connection_id return an actual ConnectionId. Note it changes connection.lua:224 (c.connection_id = protocol.unwrap(...)) from a lossy double to an exact decimal string, and test_connection.lua:51,221,229 assert on the number. Pre-1.0 is the moment to take that break; after the freeze you are stuck shipping a wrong identifier.
2. If you will not do (1), then do not pad. Return the 14 real digits with a visible truncation marker, or return nil and document that exact ConnectionIds need BSATN. A short honest string is safer than a long false one.

Either way, the current combination — precise width, fabricated content, caveat buried in a docstring — is the one option that will silently produce a wrong ConnectionId in someone's save file.

---

SHOULD-FIX 3 — the string branch is dead for wire data and actively wrong for decimal strings

stdb/protocol.lua:222-224

A ConnectionId never arrives as a hex string on this protocol. crates/lib/src/connection_id.rs:13 is explicit: in JSON it is a bare decimal number, always — in the IdentityToken envelope and in row columns alike (both go through the same Serialize impl; only named-vs-positional differs). So connection_id("0xc200"), the tested case, is unreachable.

Worse, the branch that exists returns non-hex from a function whose contract is "0x" + 32 hex digits:

connection_id("135936128599631372233289231990769713196") -> "135936128599631372233289231990769713196"
connection_id("hello")                                   -> "hello"

A decimal string is the one string form that does occur — it is what you get from the gsub in Blocker 2, or from anyone who read the fixture. Passing it through unchanged is the failure mode most likely to be mistaken for success.

Make the string branch the exact path (Blocker 2 fix), or validate it (^0x%x+$, correct length) and return nil otherwise. Do not freeze a pass-through.

---

SHOULD-FIX 4 — the nil-not-raise decision is right, but its stated justification is false

On the question as posed: protocol.lua is not of two minds. The split is clean and the new helpers land on the correct side of it. Frame-level functions raise because a malformed frame is a protocol violation that should kill the connection: decode_server (242-256), decode_row (269-278), encode_client (199-215), all via err() at 65-67. Value-level functions are total and never raise: unwrap (88-111) returns its input unchanged for anything it doesn't recognise, and decode_payload returns nil, err. The four new helpers are value-level, so returning nil matches unwrap, the function they sit next to and are documented beside. Keep it.

The justification is what's wrong, and it is frozen into the README (441-444) and the code comment (135-139):

"an Option column legitimately decodes to nil"

It does not. Sums in rows are always positional [tag, payload] — even on the named row path, because serialize_row_from_page works from the row layout, which carries variant tags but not variant names, so serialize_variant gets name: None and falls to the tuple branch (crates/sats/src/ser/serde.rs:104-124). Observed live and in the SQL endpoint:

nickname Option<String> Some("nick")      -> [0,"nick"]
status                                     -> [0,{}]
st_table.primary_key Option<ColList> None  -> [1,[]]

So an Option None column decodes to the table [1,{}], never to nil:

unwrap([0,"nick"]) -> table    (not "nick")
unwrap([1,[]])     -> table    (not nil)

Two consequences. First, unwrap's some/none branches (104-109) are dead for row columns — pre-existing, out of scope, but it is the same root cause as Blocker 1 and the header comment at 19 ("Options encode as {"some": v} / {"none": []}") is wrong for rows. Second, nil inputs do occur, just for different reasons: decode_row on a positional row shorter than columns (286-288), a missing key in a named row, decode_payload(nil).

Frozen contract to state, in these words: the four helpers are total value-level accessors — they return nil for any input they cannot interpret and never raise, matching unwrap; frame-level functions raise. Callers check for nil. Then delete the Option rationale and replace it with the real one (total accessors alongside unwrap), and note that an Option<Timestamp> column needs its own unwrap step first, which unwrap cannot currently do for the positional sum form.

---

SHOULD-FIX 5 — the frozen family is inconsistent four ways

On the specific questions asked:

Ordinal mapping is correct. Verified against the pinned 2.8.2 source, not the PR's assertion: crates/lib/src/scheduler.rs:59-63 builds the type as [(SCHEDULE_AT_INTERVAL_TAG, time_duration()), (SCHEDULE_AT_TIME_TAG, timestamp())], tags "Interval"/"Time" from crates/sats/src/sum_type.rs:9-11. 0 = Interval, 1 = Time. Correct.

Both encodings are genuinely real, which vindicates handling both — though not for the stated reason. Named sums appear in message envelopes and metadata ({"Committed":…}, {"Product":…}, {"some":"id"}); positional [tag, payload] appears in row values. Since ScheduleAt only ever occurs as a table column, the named {Interval=…} branch is unreachable on this protocol today. Harmless, and it would fire if the SDK ever reads schema JSON. Worth a comment saying so rather than implying the named form is what rows carry.

Edge cases behave. All verified: [2,…] → nil; {} → nil; {Interval="soon"} → nil; {[1]=0} with no payload → nil; {[1]="0",…} → nil. A named check never misfires on a positional payload — a positional {0, {…}} has no top-level .Interval, and {[1]=1,[2]={Interval=…}} correctly yields nil. Both keys present returns interval (first-declared wins); unreachable on the wire, worth one line of doc.

What I would change before freezing:

• Naming is half verb, half noun. timestamp_to_seconds / duration_to_seconds announce their units; schedule_at / connection_id announce nothing. The unit is the point of #23, so make all four say it: timestamp_to_seconds, duration_to_seconds, schedule_at_to_seconds, connection_id_to_hex. Consistency matters more than any single name once frozen, and timestamp_to_seconds is the right instinct — do not shorten it to timestamp.
• String handling differs across the family. Three helpers reject a string; connection_id returns it. Pick one rule.
• Identity is the odd sibling. The README table routes Identity → protocol.unwrap(v) and ConnectionId → protocol.connection_id(v). The server treats them as siblings (both special types, both rendered hex by to_hex). Add protocol.identity(v) so the family is complete and unwrap stays a generic wrapper-stripper rather than doubling as the Identity accessor. This also fixes the Blocker-1 gap for Identity, where unwrap returns a bare table on TransactionUpdate rows.
• Do not extend unwrap instead. unwrap is type-blind by design — it cannot know whether a bare number is micros or a ConnectionId, and the positional row shape makes that strictly worse. A parallel family is the right call; it just needs to be a complete, consistently named one.
• Uuid is missing and unmentioned. #23 names it explicitly ("Timestamp, TimeDuration, ConnectionId, ScheduleAt or Uuid"). It exists in 2.8.2 as Uuid { __uuid__: u128 } (crates/sats/src/uuid.rs:26-27, AlgebraicType::uuid()) — the same u128 shape as ConnectionId, so the same >2^53 precision problem, and a canonical hyphenated text form. Either add uuid now while the precision decision is being made once, or say in the PR why it is out of scope. Silently dropping one of five named types is the kind of gap a freeze makes permanent.

---

Numeric claims I verified and found correct

Nothing here needs changing; recording it so it isn't re-litigated.

Claim	Result
timestamp_to_seconds(1787157191225200) → the right UTC instant	✅ 2026-08-19T16:33:11 UTC, and 1787157191225200 is exactly representable
2^53 micros ceiling = 2255-06-05	✅ 2255-06-05 23:47:34 UTC
whole-second resolution survives /1e6	✅ 1.5M samples at the current era and 1.5M just under 2^53 (offsets 0, ±1, 500000, 999999 µs), 0 mismatches — and it holds for a reason: half-ulp of the quotient is 1.19e-7 at 2026 and 9.54e-7 at the 2255 ceiling, staying just under the 1e-6 tick the whole way. Thin margin (4.6% at the top), never breached.
seconds round-trip back to micros	✅
negatives behave	✅ The wire type is signed — the live schema gives __timestamp_micros_since_unix_epoch__ as I64, so pre-1970 is legitimate. -1e15 µs → 1938-04-24; duration_to_seconds(-1455) → -0.001455.
math.fmod negative-remainder hazard	✅ Real (math.fmod(-17,16) == -1), and correctly closed by the v < 0 guard before the hex loop ever runs
string.format("%x") returns "0"	✅
hand-rolled hex is exact	✅ 74 values vs Node BigInt, 0 mismatches
0x1b214db8a0d796 * 2^72 == parsed value	✅
ScheduleAt ordinals 0=Interval, 1=Time	✅ against pinned v2.8.2 source

---

Nits

1. ~50,000 years out is wrong by three orders of magnitude — protocol.lua:118, README.md:430, PR body, inherited from #23. Measured: os.date("!%Y-%m-%d", 1787158263465067) → 56634760-11-18, about 56.6 million years. It does not error, as claimed. Amusingly, ~50,000 years is the error you'd get if the value were milliseconds — right number, wrong unit, in the document whose entire subject is unit confusion. Say ~56 million years.
2. Fixed-width contract breaks above 2^128 — protocol.lua:233. string.rep("0", 32 - #digits) with #digits > 32 returns "" in Lua 5.1, so output silently grows: connection_id(2^128) → 0x1000…000, 33 digits. Reachable, barely — a u128 within 2^75 of the top rounds up to 2^128 (p ≈ 2^-53). Return nil when #digits > 32.
3. variant() throws away a tag it knows — protocol.lua:172-177. When the payload fails to convert, schedule_at returns a single nil, discarding the variant it had already identified with certainty. Returning "interval", nil would let a caller distinguish "not a ScheduleAt" from "an Interval I could not convert". Blocker 1's fix removes the common trigger; still worth deciding before the return shape freezes.
4. PR body slightly overstates a test — "A test pins the %x result so the shortcut cannot be reintroduced." No test asserts on string.format("%x", …) directly; the pin is indirect, via the expected hex string. It does work (the sabotage table shows it), but the claim as written describes a test that isn't there.

---

Does it prevent the bug it exists for?

Not yet, on both counts.

Mechanically, no — Blocker 1. On the live-update path timestamp_to_seconds(row.created_at) returns nil. That is better than a date 56 million years out (a nil into os.date raises at the call site, which is the failure mode the PR argues for), but a helper that works on subscription rows and not on update rows will read as flaky rather than as a units lesson.

Discoverability: placed correctly, signposted nowhere. #23 asked for the table "in the API reference next to the cache section." It is at README.md:426, inside ### stdb.schema and stdb.protocol (413), one subsection after ### stdb.cache (399). Letter satisfied. But the two places a reader actually stands when they touch a timestamp column both stay silent:

• ### stdb.cache (399-411) is the section on reading rows. It documents cache.get/cache.iter and says nothing about units, with no pointer 15 lines down.
• The ## Usage example (167-187) prints e.row.sender and e.row.text — both strings, no trap. Someone who adds created_at gets no signal at all.
• ### Two Defold-specific traps (33-45) is the natural home for a trap of exactly this class — "found by running the real thing, silently wrong rather than loud" — and this isn't in it.

The table itself is good: correct units, both ceilings explained, the os.date trap named. It just needs to be reachable from where the reader is. Cheapest fix, and I'd call it required given #23 says outright that "a helper nobody knows about does not prevent the os.date bug":

• One line under ### stdb.cache: row values keep wire units — Timestamp/TimeDuration are microseconds; see Special types and their units.
• One comment in the Usage example where a timestamp column would go.
• Promote the microseconds trap into ### Two Defold-specific traps (it becomes three), linking down to the table.

---

Does it close #23?

No, not in substance. Against #23's acceptance list:

Criterion	Status
Unit tests with a real captured value, round-tripping against a known date	✅ Met, and well done — MICROS, the os.date assertion, and the round-trip back to micros
Both ScheduleAt variants tested	✅ Met, both encodings, ordinals verified correct against source
Explicit precision test; current-era timestamp keeps seconds resolution	✅ Met in substance, and the PR is right to re-frame #23's "32-bit int build" worry as the 2^53 double ceiling — Lua 5.1's number type is a double on Defold's targets regardless of pointer width, so #23's premise was itself mistaken. The literal "on a 32-bit Lua build" cannot be tested here; the 2^53 assertion is the correct substitute.
README table of special types and units, in the API reference by the cache section	⚠️ Present and correctly placed, but unreachable from where readers are, and one row is wrong (row values are not always the named wrapper)
protocol.connection_id(v) → hex string "matching unwrap's treatment of identity"	❌ Not met — 14 of 32 digits are real, and unwrap's Identity treatment is exact where this is not
Stops os.date(row.created_at) being silently wrong	❌ Not met on the TransactionUpdate path, where the helper returns nil
Uuid, named in #23	❌ Not addressed, not excluded

With Blocker 1 fixed (a nine-line guard, already validated: 5 passed, stylua clean), Blocker 2 resolved either way, and the cross-references added, this closes #23 properly. The shape of the change is right — pure functions in protocol.lua, no automatic coercion in decode_row, explicit units in the names, all correctly argued. It is the wire-shape assumption and the ConnectionId honesty that need to be settled before the surface freezes.

---

No edits, comments, or pushes were made to the PR. Verified in a throwaway clone at /tmp/review-37 plus a scratch copy at /tmp/fixcheck for the proposed fix. On the live server I only read: HTTP GET/SQL SELECT and short-lived WebSocket subscriptions. No databases created or modified.

---

Verdict: Request changes. Two blockers. The arithmetic is right; the shape assumptions and the ConnectionId honesty are not. Does not close #23 in substance.

## Review — automated reviewer, posted for the record # Review — PR #37 `feat/23-special-types` All numeric claims checked by running them in the pinned Lua 5.1 from `nix develop .#ci`. Wire-shape claims checked against the pinned SpacetimeDB **v2.8.2** source (`/tmp/librarian-stdb`, `git describe` = `v2.8.2`) and against the live server at `127.0.0.1:3000`. Baseline: `5 passed, 0 failed`, stylua clean. **Verdict: request changes.** The arithmetic in this PR is correct — I could not break it. What is wrong is one shape assumption and one honesty problem, and both get frozen at 1.0. --- ## BLOCKER 1 — `timestamp_to_seconds` returns `nil` for exactly the bug #23 exists to fix `stdb/protocol.lua:141-146` (`scalar`) A special-type column arrives in **two** shapes depending on which frame carried the row, and the helpers accept only one of them. - **Subscription frames** (`InitialSubscription`, `Subscribe*Applied`) serialize rows from a `RowRef`, which carries the row layout → typed → `serialize_named_product` → `{"__timestamp_micros_since_unix_epoch__": n}`. `crates/table/src/table.rs:2034-2041` → `crates/sats/src/ser/impls.rs:226-228` - **`TransactionUpdate`** serializes rows from `inserts: Arc<[ProductValue]>` — an *untyped* value. Field names are dropped, and `AlgebraicValue::Product(prod) => prod.serialize(ser)` means they are dropped **recursively**, so the one-field `Timestamp` product becomes the array `[micros]`. `crates/core/src/host/module_host.rs:127-128,159-160` → `crates/sats/src/ser/impls.rs:129-131,152-156` Both shapes observed live on 2.8.2, same table, same column: ``` InitialSubscription (ws) "joined":{"__timestamp_micros_since_unix_epoch__":1787158263465067} untyped serializer (SQL) "joined" -> [1787158263465067] "owner" -> ["0xc200d7a1…"] fixture session.jsonl:4 TransactionUpdate inserts -> ["[2,\"vic\",\"live insert\"]"] ``` Run end to end through the SDK (`decode_row` + the new helper): ``` InitialSubscription (named) timestamp_to_seconds -> 1787158263.4651 os.date -> 2026-08-19T16:51:03 TransactionUpdate (positional) timestamp_to_seconds -> nil unwrap(owner) -> table: 0x… ``` So `protocol.timestamp_to_seconds(row.created_at)` works on rows from the initial subscription and returns `nil` on rows from live updates — the `on_insert`/`on_event` path a game actually spends its life in. That is the worst available failure mode for a helper whose whole purpose is a silent-wrong-answer bug: the helper silently declines to work on half the inputs, in a way unit tests built from hand-written wrappers cannot see. This is not an unknown-unknown for this codebase. `protocol.lua:262-264` already says decode_row "Handles both encodings the server uses: positional arrays (TransactionUpdate) and named objects (InitialSubscription)." The dual encoding is known at the *row* level and simply was not carried down to the *column value* level. **Fix** — one guard in `scalar`, the shared choke point all four helpers route through: ```lua local function scalar(v, field) if type(v) == "table" then local x = v[field] -- TransactionUpdate rows are untyped ProductValues: field names are dropped -- recursively, so a Timestamp column arrives as the 1-element array [micros]. -- Guarded on #v == 1 so a positional sum ([tag, payload]) never matches. if x == nil and #v == 1 and type(v[1]) == "number" then x = v[1] end return x end return v end ``` The `#v == 1` guard is load-bearing, not defensive dressing. I tried the naive `x = v[1]` first and it produces **silent wrong numbers** on Option columns, which is the exact class #23 is about: ``` naive: Option<Timestamp> None [1,{}] -> 1e-06 <-- reads the sum tag as micros naive: Option<Timestamp> Some [0,[n]] -> 0 <-- reads the sum tag as micros guarded: both -> nil ``` Guarded version, verified: ``` Timestamp column (TransactionUpdate) [n] -> 1787158263.4651 named wrapper -> 1787158263.4651 bare number -> 1787158263.4651 Option<Timestamp> None [1,{}] -> nil Option<Timestamp> Some [0,[n]] -> nil wrong wrapper {__time_duration_micros__=5} -> nil positional ScheduleAt [0,[1500000]] -> interval, 1.5 positional ConnectionId [42] -> 0x…002a ``` Applied to a scratch clone: **`5 passed, 0 failed`, stylua clean.** Fully backwards compatible. The README table needs the same correction — it currently states the decoded row value *is* `{ __timestamp_micros_since_unix_epoch__ = n }`, which is true only for subscription rows. --- ## BLOCKER 2 — `connection_id` pads lossy data into a precise-looking identifier, and the lossiness is avoidable `stdb/protocol.lua:220-234` First, the author's own claims are all true. I verified each: - `string.format("%x", 36062078055823991097399387891464995893)` → **`"0"`**. Confirmed. (`%d` on it gives `-9223372036854775808`, also garbage.) The hand-rolled loop is genuinely necessary. - `0x1b214db8a0d796 * 2^72 == <parsed value>` → **true**. `0x1b214db8a0d796` is exactly 53 significant bits, as it must be. - The hand-rolled hex is arithmetically **correct**. Differential test, 74 exact-integer doubles (0, 15, 16, 2^31±1, 2^32, 2^40, 2^52, 2^53, 40 random below 2^53, 20 random scaled by 2^3…2^60) against Node `BigInt(...).toString(16)`: **0 mismatches**. `math.fmod`/`math.floor` by 16 on a double are exact, as claimed. - `math.fmod(-17, 16)` is indeed **`-1`** in this Lua, so the negative-remainder hazard is real — and it is correctly closed by the `v < 0` guard on line 228. `-1`, `-0.5`, `-inf`, and NaN all return `nil`. The guard is load-bearing and there is a test for it. Good. The problem is not the arithmetic. It is that the arithmetic is applied to already-corrupted input and the result is then padded to 32 digits, which is precisely the canonical width of an exact ConnectionId. Captured live from 2.8.2 this session: ``` wire decimal 135936128599631372233289231990769713196 true hex 0x66445897597a38fb947e3b5c1d06002c <- server's ConnectionId::to_hex() this PR emits 0x66445897597a38000000000000000000 ^^^^^^^^^^^^^^^^^^ 18 digits fabricated ``` **14 of 32 digits are real.** Same result on the fixture value (`0x1b214db8a0d79617a81f1948ffa10435` true, `0x1b214db8a0d796000000000000000000` emitted). The true hex is `%032x` of the u128 — `to_hex` = `hex::encode(as_be_byte_array())` = `u128::to_be_bytes` (`crates/lib/src/connection_id.rs:104-119`), so there is no byte-swap subtlety; the emitted string is directly comparable to the server's, and it is 56% right. So: not honest. The value is shaped exactly like the thing you paste into `spacetime`, key a table on, write to a save file, or send to a support channel. `0x…000000000000000000` does not read as "unknown low bits"; it reads as a ConnectionId that happens to end in zeros. Even the server's own abbreviated form (`to_abbreviated_hex`, first 16 hex chars) would be wrong in its last two digits. The docstring's mitigations don't hold up either: - *"the high ~53 bits are the wire value"* — only when rounding happens to go down, as it did for the fixture. Rounding is round-to-nearest, so the retained digits can be wrong too. Demonstrated: a value whose true high digits end `…d796` emits `…d797`. - *"Fine … to compare against another id decoded the same way"* — mostly true for random ids, but not unconditionally: two ids differing **by 1** on the wire produce an identical string. Verified with `json.decode` on `…302976` and `…302977`. - *"That needs a BSATN codec or a JSON decoder that keeps big integers as text"* — **this is false**, and it is what turns the finding from "document a limitation" into a blocker. Neither is needed. SpacetimeDB documents the hazard and the workaround in the same breath (`crates/lib/src/connection_id.rs:13-16`: *"In JSON, a `ConnectionId` is represented as a BARE DECIMAL number. This requires some care when deserializing"*, linking the JS BigInt question). `decode_server` holds the **raw frame text**, so the digits are right there before `json.decode` ever sees them. One gsub plus ~14 lines of Lua 5.1 recovers the value exactly: ```lua -- before json.decode, in decode_server: data = data:gsub('("__connection_id__":)(%d+)', '%1"%2"') -- exact decimal-string -> 32 hex digits, repeated division by 16 on a digit array local function dec_to_hex(s) local d = {} for i = 1, #s do d[i] = tonumber(s:sub(i, i)) end local out = {} repeat local rem, nz = 0, false for i = 1, #d do local cur = rem * 10 + d[i] d[i] = math.floor(cur / 16) if d[i] ~= 0 then nz = true end rem = cur % 16 end out[#out + 1] = string.format("%x", rem) until not nz local h = "" for i = #out, 1, -1 do h = h .. out[i] end return "0x" .. string.rep("0", 32 - #h) .. h end ``` Verified: recovers `0x66445897597a38fb947e3b5c1d06002c` from the live decimal, `0x1b214db8a0d79617a81f1948ffa10435` from the fixture, and agrees with Node `BigInt` on every value tried. `42` → `0x…002a`, `0` → all zeros. **Recommendation, in preference order:** 1. Do the gsub in `decode_server` and make `connection_id` convert the decimal string exactly. This is the only option that makes a frozen function named `connection_id` return an actual ConnectionId. Note it changes `connection.lua:224` (`c.connection_id = protocol.unwrap(...)`) from a lossy double to an exact decimal string, and `test_connection.lua:51,221,229` assert on the number. Pre-1.0 is the moment to take that break; after the freeze you are stuck shipping a wrong identifier. 2. If you will not do (1), then **do not pad**. Return the 14 real digits with a visible truncation marker, or return `nil` and document that exact ConnectionIds need BSATN. A short honest string is safer than a long false one. Either way, the current combination — precise width, fabricated content, caveat buried in a docstring — is the one option that will silently produce a wrong ConnectionId in someone's save file. --- ## SHOULD-FIX 3 — the string branch is dead for wire data and actively wrong for decimal strings `stdb/protocol.lua:222-224` A ConnectionId never arrives as a hex string on this protocol. `crates/lib/src/connection_id.rs:13` is explicit: in JSON it is a bare decimal number, always — in the `IdentityToken` envelope and in row columns alike (both go through the same `Serialize` impl; only named-vs-positional differs). So `connection_id("0xc200")`, the tested case, is unreachable. Worse, the branch that exists returns non-hex from a function whose contract is `"0x" + 32 hex digits`: ``` connection_id("135936128599631372233289231990769713196") -> "135936128599631372233289231990769713196" connection_id("hello") -> "hello" ``` A decimal string is the one string form that *does* occur — it is what you get from the gsub in Blocker 2, or from anyone who read the fixture. Passing it through unchanged is the failure mode most likely to be mistaken for success. Make the string branch the exact path (Blocker 2 fix), or validate it (`^0x%x+$`, correct length) and return `nil` otherwise. Do not freeze a pass-through. --- ## SHOULD-FIX 4 — the `nil`-not-raise decision is right, but its stated justification is false On the question as posed: **`protocol.lua` is not of two minds.** The split is clean and the new helpers land on the correct side of it. Frame-level functions raise because a malformed frame is a protocol violation that should kill the connection: `decode_server` (`242-256`), `decode_row` (`269-278`), `encode_client` (`199-215`), all via `err()` at `65-67`. Value-level functions are total and never raise: `unwrap` (`88-111`) returns its input unchanged for anything it doesn't recognise, and `decode_payload` returns `nil, err`. The four new helpers are value-level, so returning `nil` matches `unwrap`, the function they sit next to and are documented beside. Keep it. The **justification** is what's wrong, and it is frozen into the README (`441-444`) and the code comment (`135-139`): > "an `Option` column legitimately decodes to `nil`" It does not. Sums in rows are always positional `[tag, payload]` — even on the *named* row path, because `serialize_row_from_page` works from the row layout, which carries variant tags but not variant names, so `serialize_variant` gets `name: None` and falls to the tuple branch (`crates/sats/src/ser/serde.rs:104-124`). Observed live and in the SQL endpoint: ``` nickname Option<String> Some("nick") -> [0,"nick"] status -> [0,{}] st_table.primary_key Option<ColList> None -> [1,[]] ``` So an Option `None` column decodes to the **table** `[1,{}]`, never to `nil`: ``` unwrap([0,"nick"]) -> table (not "nick") unwrap([1,[]]) -> table (not nil) ``` Two consequences. First, `unwrap`'s `some`/`none` branches (`104-109`) are dead for row columns — pre-existing, out of scope, but it is the same root cause as Blocker 1 and the header comment at `19` ("Options encode as `{"some": v}` / `{"none": []}`") is wrong for rows. Second, `nil` inputs do occur, just for different reasons: `decode_row` on a positional row shorter than `columns` (`286-288`), a missing key in a named row, `decode_payload(nil)`. **Frozen contract to state, in these words:** *the four helpers are total value-level accessors — they return `nil` for any input they cannot interpret and never raise, matching `unwrap`; frame-level functions raise. Callers check for `nil`.* Then delete the Option rationale and replace it with the real one (total accessors alongside `unwrap`), and note that an `Option<Timestamp>` column needs its own unwrap step first, which `unwrap` cannot currently do for the positional sum form. --- ## SHOULD-FIX 5 — the frozen family is inconsistent four ways On the specific questions asked: **Ordinal mapping is correct.** Verified against the pinned 2.8.2 source, not the PR's assertion: `crates/lib/src/scheduler.rs:59-63` builds the type as `[(SCHEDULE_AT_INTERVAL_TAG, time_duration()), (SCHEDULE_AT_TIME_TAG, timestamp())]`, tags `"Interval"`/`"Time"` from `crates/sats/src/sum_type.rs:9-11`. **0 = Interval, 1 = Time.** Correct. **Both encodings are genuinely real**, which vindicates handling both — though not for the stated reason. Named sums appear in message envelopes and metadata (`{"Committed":…}`, `{"Product":…}`, `{"some":"id"}`); positional `[tag, payload]` appears in row values. Since `ScheduleAt` only ever occurs as a *table column*, the named `{Interval=…}` branch is unreachable on this protocol today. Harmless, and it would fire if the SDK ever reads schema JSON. Worth a comment saying so rather than implying the named form is what rows carry. **Edge cases behave.** All verified: `[2,…]` → `nil`; `{}` → `nil`; `{Interval="soon"}` → `nil`; `{[1]=0}` with no payload → `nil`; `{[1]="0",…}` → `nil`. A named check never misfires on a positional payload — a positional `{0, {…}}` has no top-level `.Interval`, and `{[1]=1,[2]={Interval=…}}` correctly yields `nil`. Both keys present returns `interval` (first-declared wins); unreachable on the wire, worth one line of doc. What I would change before freezing: - **Naming is half verb, half noun.** `timestamp_to_seconds` / `duration_to_seconds` announce their units; `schedule_at` / `connection_id` announce nothing. The unit *is* the point of #23, so make all four say it: `timestamp_to_seconds`, `duration_to_seconds`, `schedule_at_to_seconds`, `connection_id_to_hex`. Consistency matters more than any single name once frozen, and `timestamp_to_seconds` is the right instinct — do not shorten it to `timestamp`. - **String handling differs across the family.** Three helpers reject a string; `connection_id` returns it. Pick one rule. - **`Identity` is the odd sibling.** The README table routes `Identity` → `protocol.unwrap(v)` and `ConnectionId` → `protocol.connection_id(v)`. The server treats them as siblings (both special types, both rendered hex by `to_hex`). Add `protocol.identity(v)` so the family is complete and `unwrap` stays a generic wrapper-stripper rather than doubling as the Identity accessor. This also fixes the Blocker-1 gap for Identity, where `unwrap` returns a bare table on TransactionUpdate rows. - **Do not extend `unwrap` instead.** `unwrap` is type-blind by design — it cannot know whether a bare number is micros or a ConnectionId, and the positional row shape makes that strictly worse. A parallel family is the right call; it just needs to be a complete, consistently named one. - **`Uuid` is missing and unmentioned.** #23 names it explicitly ("Timestamp, TimeDuration, ConnectionId, ScheduleAt or Uuid"). It exists in 2.8.2 as `Uuid { __uuid__: u128 }` (`crates/sats/src/uuid.rs:26-27`, `AlgebraicType::uuid()`) — the same u128 shape as ConnectionId, so the same >2^53 precision problem, and a canonical hyphenated text form. Either add `uuid` now while the precision decision is being made once, or say in the PR why it is out of scope. Silently dropping one of five named types is the kind of gap a freeze makes permanent. --- ## Numeric claims I verified and found correct Nothing here needs changing; recording it so it isn't re-litigated. | Claim | Result | |---|---| | `timestamp_to_seconds(1787157191225200)` → the right UTC instant | ✅ `2026-08-19T16:33:11` UTC, and `1787157191225200` is exactly representable | | 2^53 micros ceiling = `2255-06-05` | ✅ `2255-06-05 23:47:34` UTC | | whole-second resolution survives `/1e6` | ✅ 1.5M samples at the current era **and** 1.5M just under 2^53 (offsets 0, ±1, 500000, 999999 µs), **0 mismatches** — and it holds for a reason: half-ulp of the quotient is 1.19e-7 at 2026 and 9.54e-7 at the 2255 ceiling, staying just under the 1e-6 tick the whole way. Thin margin (4.6% at the top), never breached. | | seconds round-trip back to micros | ✅ | | negatives behave | ✅ The wire type is signed — the live schema gives `__timestamp_micros_since_unix_epoch__` as `I64`, so pre-1970 is legitimate. `-1e15` µs → `1938-04-24`; `duration_to_seconds(-1455)` → `-0.001455`. | | `math.fmod` negative-remainder hazard | ✅ Real (`math.fmod(-17,16) == -1`), and correctly closed by the `v < 0` guard before the hex loop ever runs | | `string.format("%x")` returns `"0"` | ✅ | | hand-rolled hex is exact | ✅ 74 values vs Node `BigInt`, 0 mismatches | | `0x1b214db8a0d796 * 2^72` == parsed value | ✅ | | ScheduleAt ordinals 0=Interval, 1=Time | ✅ against pinned v2.8.2 source | --- ## Nits 1. **`~50,000 years out` is wrong by three orders of magnitude** — `protocol.lua:118`, `README.md:430`, PR body, inherited from #23. Measured: `os.date("!%Y-%m-%d", 1787158263465067)` → **`56634760-11-18`**, about 56.6 *million* years. It does not error, as claimed. Amusingly, ~50,000 years is the error you'd get if the value were **milliseconds** — right number, wrong unit, in the document whose entire subject is unit confusion. Say `~56 million years`. 2. **Fixed-width contract breaks above 2^128** — `protocol.lua:233`. `string.rep("0", 32 - #digits)` with `#digits > 32` returns `""` in Lua 5.1, so output silently grows: `connection_id(2^128)` → `0x1000…000`, 33 digits. Reachable, barely — a u128 within 2^75 of the top rounds *up* to 2^128 (p ≈ 2^-53). Return `nil` when `#digits > 32`. 3. **`variant()` throws away a tag it knows** — `protocol.lua:172-177`. When the payload fails to convert, `schedule_at` returns a single `nil`, discarding the variant it had already identified with certainty. Returning `"interval", nil` would let a caller distinguish "not a ScheduleAt" from "an Interval I could not convert". Blocker 1's fix removes the common trigger; still worth deciding before the return shape freezes. 4. **PR body slightly overstates a test** — "A test pins the `%x` result so the shortcut cannot be reintroduced." No test asserts on `string.format("%x", …)` directly; the pin is indirect, via the expected hex string. It does work (the sabotage table shows it), but the claim as written describes a test that isn't there. --- ## Does it prevent the bug it exists for? Not yet, on both counts. **Mechanically, no** — Blocker 1. On the live-update path `timestamp_to_seconds(row.created_at)` returns `nil`. That is better than a date 56 million years out (a `nil` into `os.date` raises at the call site, which is the failure mode the PR argues for), but a helper that works on subscription rows and not on update rows will read as flaky rather than as a units lesson. **Discoverability: placed correctly, signposted nowhere.** #23 asked for the table "in the API reference next to the cache section." It is at `README.md:426`, inside `### stdb.schema and stdb.protocol` (413), one subsection after `### stdb.cache` (399). Letter satisfied. But the two places a reader actually stands when they touch a timestamp column both stay silent: - `### stdb.cache` (399-411) is the section on reading rows. It documents `cache.get`/`cache.iter` and says nothing about units, with no pointer 15 lines down. - The `## Usage` example (167-187) prints `e.row.sender` and `e.row.text` — both strings, no trap. Someone who adds `created_at` gets no signal at all. - `### Two Defold-specific traps` (33-45) is the natural home for a trap of exactly this class — "found by running the real thing, silently wrong rather than loud" — and this isn't in it. The table itself is good: correct units, both ceilings explained, the `os.date` trap named. It just needs to be reachable from where the reader is. Cheapest fix, and I'd call it required given #23 says outright that "a helper nobody knows about does not prevent the `os.date` bug": - One line under `### stdb.cache`: *row values keep wire units — `Timestamp`/`TimeDuration` are microseconds; see Special types and their units.* - One comment in the Usage example where a timestamp column would go. - Promote the microseconds trap into `### Two Defold-specific traps` (it becomes three), linking down to the table. --- ## Does it close #23? **No, not in substance.** Against #23's acceptance list: | Criterion | Status | |---|---| | Unit tests with a real captured value, round-tripping against a known date | ✅ Met, and well done — `MICROS`, the `os.date` assertion, and the round-trip back to micros | | Both `ScheduleAt` variants tested | ✅ Met, both encodings, ordinals verified correct against source | | Explicit precision test; current-era timestamp keeps seconds resolution | ✅ Met in substance, and the PR is right to re-frame #23's "32-bit `int` build" worry as the 2^53 double ceiling — Lua 5.1's number type is a double on Defold's targets regardless of pointer width, so #23's premise was itself mistaken. The literal "on a 32-bit Lua build" cannot be tested here; the 2^53 assertion is the correct substitute. | | README table of special types and units, in the API reference by the cache section | ⚠️ Present and correctly placed, but unreachable from where readers are, and one row is wrong (row values are not always the named wrapper) | | `protocol.connection_id(v)` → hex string "matching `unwrap`'s treatment of identity" | ❌ Not met — 14 of 32 digits are real, and `unwrap`'s Identity treatment is exact where this is not | | Stops `os.date(row.created_at)` being silently wrong | ❌ Not met on the `TransactionUpdate` path, where the helper returns `nil` | | `Uuid`, named in #23 | ❌ Not addressed, not excluded | With Blocker 1 fixed (a nine-line guard, already validated: 5 passed, stylua clean), Blocker 2 resolved either way, and the cross-references added, this closes #23 properly. The shape of the change is right — pure functions in `protocol.lua`, no automatic coercion in `decode_row`, explicit units in the names, all correctly argued. It is the wire-shape assumption and the ConnectionId honesty that need to be settled before the surface freezes. --- *No edits, comments, or pushes were made to the PR. Verified in a throwaway clone at `/tmp/review-37` plus a scratch copy at `/tmp/fixcheck` for the proposed fix. On the live server I only read: HTTP `GET`/`SQL SELECT` and short-lived WebSocket subscriptions. No databases created or modified.* --- **Verdict:** Request changes. Two blockers. The arithmetic is right; the shape assumptions and the ConnectionId honesty are not. Does not close #23 in substance.

yuris force-pushed feat/23-special-types from 27234934a2 to 5a460435b4 2026-08-19 17:52:43 +00:00 Compare

yuris force-pushed feat/23-special-types from 5a460435b4 to ac32a1ae3f 2026-08-19 17:56:54 +00:00 Compare

yuris force-pushed feat/23-special-types from ac32a1ae3f to fb1be0b206 2026-08-19 17:58:44 +00:00 Compare

yuris merged commit 6efcfdd274 into main 2026-08-19 18:00:39 +00:00

yuris referenced this pull request from a commit 2026-08-19 18:00:40 +00:00

protocol: convert special types explicitly (#37)

yuris referenced this pull request 2026-08-19 18:20:44 +00:00

protocol: decode every AlgebraicType from the schema tags #46

yuris referenced this pull request 2026-08-19 18:24:26 +00:00

Identity-keyed tables are broken in the cache (pk decodes to a table) #35

yuris referenced this pull request 2026-08-19 18:26:34 +00:00

v1.0.0 release checklist #24

yuris referenced this pull request from a commit 2026-08-19 18:29:01 +00:00

cache: index rows by every single-column unique constraint (#28)

yuris referenced this pull request 2026-08-20 01:20:24 +00:00

protocol: decode every AlgebraicType from the schema tags #46

yuris referenced this pull request from a commit 2026-08-20 01:40:53 +00:00

protocol: make the 128-bit rescue mark unforgeable, and stop fabricating digits

yuris referenced this pull request from a commit 2026-08-20 01:41:44 +00:00

protocol: make the 128-bit rescue mark unforgeable, and stop fabricating digits

yuris referenced this pull request from a commit 2026-08-20 01:42:01 +00:00

protocol: make the 128-bit rescue mark unforgeable, and stop fabricating digits

yuris referenced this pull request 2026-08-20 01:45:51 +00:00

v1.0.0 release checklist #24

yuris referenced this pull request from a commit 2026-08-20 02:15:04 +00:00

Route cached rows through the typed decoder so an Identity key works

yuris referenced this pull request from a commit 2026-08-20 03:16:31 +00:00

Route cached rows through the typed decoder so an Identity key works

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area:defold

Defold engine integration

  

area:docs

Documentation and examples

  

area:infra

Flake, CI, secretspec

  

area:protocol

Wire protocol conformance

  

area:testing

Test harness and fixtures

  

kind:bug

Something broken

  

kind:chore

Maintenance

  

phase:breadth

UnsubscribeMulti, procedures, BSATN

  

phase:core

Core SDK layers (protocol/cache/connection)

  

phase:hardening

Reconnect, token persistence, errors

  

phase:slice

Vertical slice: first playable end-to-end

No labels

area:defold

area:docs

area:infra

area:protocol

area:testing

kind:bug

kind:chore

phase:breadth

phase:core

phase:hardening

phase:slice

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/spacetimedb-defold!37

No description provided.