Decode every AlgebraicType, with a fixture module that exercises all of them #26

New issue

Closed

opened 2026-08-19 15:42:13 +00:00 by yuris · 0 comments

yuris commented

2026-08-19 15:42:13 +00:00

Owner

The SDK decodes rows structurally and hands values through untouched. That is correct for scalars and strings — every column the chat test module has — and wrong or unusable for everything else. Nobody has hit it because the only module under test has three columns: u64, string, string.

What breaks today

From v2.8.2 source plus a compiled probe of the sats serializer:

Sum types / enums arrive as positional [tag, value] arrays, because rows serialized from table pages call serialize_variant(tag, None, value) with no variant name (crates/table/src/bflatn_from.rs, serialize_sum). An enum column therefore reaches the consumer as {1, {...}} — a Lua array with a zero-based tag and no variant name anywhere. Unusable without the schema.
Option<T> is a sum of some/none, and the deserializer also accepts JSON null for none. Two encodings for one concept, neither normalised.
i256 / u256 come across as hex strings ("-0x18ee90ff..."), not numbers.
u128 / i128 are emitted as full-precision JSON numbers via serde arbitrary precision. Lua 5.1's number is a double, so u128::MAX silently loses precision on decode. This one cannot be fixed by decoding alone — it needs a documented representation choice (string? hi/lo pair?).
Special types are single-field products, so a Timestamp column decodes to { __timestamp_micros_since_unix_epoch__ = 1700000000123456 } rather than anything a game can use. That is #23 seen from the wire side.

The real problem: no coverage

dev/server/index.js exercises three scalar columns. Every claim above about how the server encodes these types is read from source, not from captured bytes — and this project's history is a list of cases where the wire disagreed with the documentation: the SubscribeApplied vs SubscribeMultiApplied nesting, the double-encoded procedure return, and upstream's own doc comment being wrong about overlapping subscribes.

So the work is fixture-first, mirroring the strategy the Godot SDK uses (a Rust test module deliberately full of exotic types):

Add a table to dev/server/index.js with a column of every type: all int widths including 128/256-bit, floats, bool, string, bytes, an enum, Option<T> of a scalar and of a product, an array, a nested product, Identity, ConnectionId, Timestamp, TimeDuration, ScheduleAt, Uuid
Capture a fixture of that table's rows from a live server into dev/test/fixtures/
Write the decoder against the captured bytes, one type at a time, driven by the schema types from the codegen issue
Document the representation choice for 128/256-bit integers, where a Lua double cannot hold the value

Acceptance

A fixture-replay test that decodes every type from real captured server output and asserts the resulting Lua value. For enums that means a named variant ({ tag = "Ready", value = ... }), not a bare positional array. For 128-bit integers it means an explicit, tested, documented lossy-or-string decision rather than silent precision loss.

Priority

Blocker for 1.0. A consumer whose module has an enum or a timestamp column currently gets garbage, and the SDK's own test module is too simple to notice.

The SDK decodes rows structurally and hands values through untouched. That is correct for scalars and strings — every column the `chat` test module has — and wrong or unusable for everything else. Nobody has hit it because the only module under test has three columns: `u64`, `string`, `string`. ## What breaks today From v2.8.2 source plus a compiled probe of the sats serializer: - **Sum types / enums** arrive as **positional `[tag, value]` arrays**, because rows serialized from table pages call `serialize_variant(tag, None, value)` with no variant name (`crates/table/src/bflatn_from.rs`, `serialize_sum`). An enum column therefore reaches the consumer as `{1, {...}}` — a Lua array with a zero-based tag and no variant name anywhere. Unusable without the schema. - **`Option<T>`** is a sum of `some`/`none`, and the deserializer also accepts JSON `null` for none. Two encodings for one concept, neither normalised. - **`i256` / `u256`** come across as **hex strings** (`"-0x18ee90ff..."`), not numbers. - **`u128` / `i128`** are emitted as full-precision JSON numbers via serde arbitrary precision. Lua 5.1's number is a double, so `u128::MAX` silently loses precision on decode. This one cannot be fixed by decoding alone — it needs a documented representation choice (string? hi/lo pair?). - **Special types** are single-field products, so a `Timestamp` column decodes to `{ __timestamp_micros_since_unix_epoch__ = 1700000000123456 }` rather than anything a game can use. That is #23 seen from the wire side. ## The real problem: no coverage `dev/server/index.js` exercises three scalar columns. Every claim above about how the server encodes these types is read from source, not from captured bytes — and this project's history is a list of cases where the wire disagreed with the documentation: the `SubscribeApplied` vs `SubscribeMultiApplied` nesting, the double-encoded procedure return, and upstream's own doc comment being wrong about overlapping subscribes. So the work is fixture-first, mirroring the strategy the Godot SDK uses (a Rust test module deliberately full of exotic types): - [ ] Add a table to `dev/server/index.js` with a column of every type: all int widths including 128/256-bit, floats, bool, string, bytes, an enum, `Option<T>` of a scalar and of a product, an array, a nested product, `Identity`, `ConnectionId`, `Timestamp`, `TimeDuration`, `ScheduleAt`, `Uuid` - [ ] Capture a fixture of that table's rows from a live server into `dev/test/fixtures/` - [ ] Write the decoder against the captured bytes, one type at a time, driven by the schema types from the codegen issue - [ ] Document the representation choice for 128/256-bit integers, where a Lua double cannot hold the value ## Acceptance A fixture-replay test that decodes every type from real captured server output and asserts the resulting Lua value. For enums that means a named variant (`{ tag = "Ready", value = ... }`), not a bare positional array. For 128-bit integers it means an explicit, tested, documented lossy-or-string decision rather than silent precision loss. ## Priority Blocker for 1.0. A consumer whose module has an enum or a timestamp column currently gets garbage, and the SDK's own test module is too simple to notice.

yuris referenced this issue

2026-08-19 15:43:23 +00:00

v1.0.0 release checklist #24

yuris referenced this issue

2026-08-19 16:17:35 +00:00

codegen: keep the module's type information #36

yuris referenced this issue

2026-08-19 16:38:45 +00:00

codegen: keep the module's type information #36

yuris referenced this issue from a commit

2026-08-19 17:00:08 +00:00

codegen: keep the module's type information (#25)