yuris/spacetimedb-defold

Fork 0

docs: state the real reason column order is load-bearing #40

Merged

yuris merged 1 commit from feat/32-column-order into main

2026-08-19 17:48:24 +00:00

yuris commented

2026-08-19 17:17:46 +00:00

Owner

Closes #32 — and corrects the issue itself, which was half wrong.

The issue's premise was wrong in an instructive way

#32 claimed the doctrine "TransactionUpdate rows are positional arrays" was false, citing a captured row from session.jsonl:

{"id":1,"sender":"yuris","text":"seeded before subscribe"}

That row is from an InitialSubscription. The claim it was cited against is about TransactionUpdate. gen-schema.sh was right: TransactionUpdate rows are positional.

What was actually wrong is subtler and worth more: the doctrine generalised one message's encoding to all rows, and it led with the weakest of three reasons the declared order matters.

What the wire actually does

Captured from a live 2.8.2 server via a throwaway probe32 module carrying a table and a view whose column order deliberately differs. Driven twice — one sequential sweep, then each output reached first on a fresh connection so overlapping-subscription dedup could not mask a row list.

message	table rows	view rows
`TransactionUpdate` (inserts and deletes)	positional	positional
`InitialSubscription`	named	positional
`SubscribeApplied` / `SubscribeMultiApplied`	named	positional
`UnsubscribeApplied` / `UnsubscribeMultiApplied`	named	positional
`OneOffQueryResponse`	named	positional

Two things here are new information for this repo:

OneOffQueryResponse is named. The committed fixture only ever captured an empty one, so this was genuinely open.
A view is positional in every message, subscription readbacks included. That confirms #32's hypothesis that a view projects through its own path rather than the table's schema-aware one.

Independently corroborated: the #21 branch captured a pk row change as one updates entry with both deletes and inserts positional, inside a TransactionUpdate. Consistent.

What changed

The generator's justification is now the accurate one, in strength order:

columns[i] ↔ types[i] pairing — the order is what aligns a column with its type tag
the typespace's variant and field order, without which sum_ and product_ columns are undecodable at all
row shape — the lesser reason, and the only one the file used to give

decode_row gains a record of which messages hit which branch, and states plainly that the positional branch is neither dead nor legacy: it serves all incremental table traffic and every row of a view. Behaviour unchanged.

Also corrected: the protocol.lua and schema.lua headers, the README's wire-fact #3 and schema-generation section, capture-fixtures.mjs's protocol-facts comment, and the stamped generated-file header. chat_schema.lua regenerated — diff is comment text only, zero data lines changed, verified.

A new Limitations bullet records that views are not emitted by the generator, and that views are exactly the case that is positional everywhere.

The column-order test was kept, and now says why

Frames 4/5/8/13 of session.jsonl are TransactionUpdates whose rows are bare arrays decoded through exactly that column list, so the assertion has a real dependency — and its message now names those frames instead of asserting a general truth.

Added the dependency nothing was checking: columns and types are equal-length parallel arrays, on both generated tables.

Verification

5 passed, 0 failed; stylua clean. Each assertion proven to fire:

sabotage	failure
reorder `message` columns	`generated column order matches the module, which decodes the TransactionUpdate rows in frames 4/5/8/13: expected id,sender,text, got id,text,sender`
drop a type tag	`message: one type tag per column … expected 3, got 2`
drop a `profile` column	`profile: one type tag per column … expected 8, got 9`

probe32 was deleted after capture; its files never entered the repo.

Closes #32 — and corrects the issue itself, which was half wrong. ## The issue's premise was wrong in an instructive way #32 claimed the doctrine "TransactionUpdate rows are positional arrays" was false, citing a captured row from `session.jsonl`: ```json {"id":1,"sender":"yuris","text":"seeded before subscribe"} ``` That row is from an **InitialSubscription**. The claim it was cited against is about **TransactionUpdate**. `gen-schema.sh` was right: TransactionUpdate rows *are* positional. What was actually wrong is subtler and worth more: the doctrine generalised one message's encoding to all rows, and it led with the weakest of three reasons the declared order matters. ## What the wire actually does Captured from a live 2.8.2 server via a throwaway `probe32` module carrying a table **and a view** whose column order deliberately differs. Driven twice — one sequential sweep, then each output reached first on a fresh connection so overlapping-subscription dedup could not mask a row list. | message | table rows | view rows | |---|---|---| | `TransactionUpdate` (inserts and deletes) | **positional** | positional | | `InitialSubscription` | named | positional | | `SubscribeApplied` / `SubscribeMultiApplied` | named | positional | | `UnsubscribeApplied` / `UnsubscribeMultiApplied` | named | positional | | `OneOffQueryResponse` | **named** | positional | Two things here are new information for this repo: - **`OneOffQueryResponse` is named.** The committed fixture only ever captured an *empty* one, so this was genuinely open. - **A view is positional in every message**, subscription readbacks included. That confirms #32's hypothesis that a view projects through its own path rather than the table's schema-aware one. Independently corroborated: the #21 branch captured a pk row change as one `updates` entry with both deletes and inserts positional, inside a `TransactionUpdate`. Consistent. ## What changed The generator's justification is now the accurate one, in strength order: 1. `columns[i]` ↔ `types[i]` pairing — the order is what aligns a column with its type tag 2. the typespace's variant and field order, without which `sum_` and `product_` columns are undecodable at all 3. row shape — the *lesser* reason, and the only one the file used to give `decode_row` gains a record of which messages hit which branch, and states plainly that the positional branch is neither dead nor legacy: it serves all incremental table traffic and every row of a view. Behaviour unchanged. Also corrected: the `protocol.lua` and `schema.lua` headers, the README's wire-fact #3 and schema-generation section, `capture-fixtures.mjs`'s protocol-facts comment, and the stamped generated-file header. `chat_schema.lua` regenerated — **diff is comment text only, zero data lines changed**, verified. A new `Limitations` bullet records that views are not emitted by the generator, and that views are exactly the case that is positional everywhere. ## The column-order test was kept, and now says why Frames 4/5/8/13 of `session.jsonl` are `TransactionUpdate`s whose rows are bare arrays decoded through exactly that column list, so the assertion has a real dependency — and its message now names those frames instead of asserting a general truth. Added the dependency nothing was checking: `columns` and `types` are equal-length parallel arrays, on both generated tables. ## Verification `5 passed, 0 failed`; stylua clean. Each assertion proven to fire: | sabotage | failure | |---|---| | reorder `message` columns | `generated column order matches the module, which decodes the TransactionUpdate rows in frames 4/5/8/13: expected id,sender,text, got id,text,sender` | | drop a type tag | `message: one type tag per column … expected 3, got 2` | | drop a `profile` column | `profile: one type tag per column … expected 8, got 9` | `probe32` was deleted after capture; its files never entered the repo.

yuris added 1 commit

2026-08-19 17:17:46 +00:00

docs: state the real reason column order is load-bearing (#32 )

ci / ci (pull_request) Successful in 20s

Details

9cbfd72c05

The doctrine said table rows arrive as positional JSON arrays, so column
order is what gives their fields names. That was the justification for the
generator existing, for the "reorder a column and every client mis-maps
fields" warning, and for a fixture test asserting column order.

It is half true, and the half it gets wrong is the half it leads with.
Captured off a live 2.8.2 v1.json.spacetimedb socket, per message rather
than per table:

  TransactionUpdate            table rows POSITIONAL, inserts and deletes
  InitialSubscription          table rows NAMED
  Subscribe{,Multi}Applied     table rows NAMED
  Unsubscribe{,Multi}Applied   table rows NAMED
  OneOffQueryResponse          table rows NAMED

and a VIEW is positional in every one of those, subscription readbacks
included, because a view projects through its own path rather than the
table's schema-aware one. The repo fixture only ever captured an empty
OneOffQueryResponse and had no view at all, so both were open questions;
they were settled by publishing a throwaway module with a view and probing
each output first on a fresh connection, so overlapping-subscription dedup
could not mask a row list.

So "rows are positional" was never true as stated, and "rows are named" is
not true either. Neither is the strongest reason to keep the generator.
That reason is typing: `columns[i]` is paired with `types[i]` by position
alone, and the variant order of a sum / the field order of a nested product
in `typespace` is what makes those columns decodable at all, since both
arrive positionally carrying no names. Row shape is the lesser reason and
now reads as the lesser reason.

decode_row needed no change; it was already right. It gains the capture
instead, because it is the single place the two encodings are reconciled,
and a future reader deleting the "legacy" positional branch would break all
incremental table traffic and every view.

The fixture test keeps its column-order assertion, because something really
does depend on it: frames 4/5/8/13 are TransactionUpdates whose message rows
are bare arrays, decoded through exactly that list. The assertion now names
those frames, so the failure explains itself. Added alongside it the
dependency nothing was checking — that columns and types are equal-length
parallel arrays, on both generated tables.

Also recorded, since it is the one gap the capture exposed: views are not
emitted by the generator, and they are precisely the case whose rows are
positional everywhere.

yuris commented

2026-08-19 17:29:38 +00:00

Author

Owner

Review — automated reviewer, posted for the record

Verdict: Request changes. The table is correct — I reproduced every cell independently — but the PR introduces one NEW wrong wire claim (the fourth) and one wrong claim about its own test, both stamped into generated output and user-facing failure text.

Closes #32: Not yet. Items 2 and 3 of #32's checklist are done well. Item 1 is done except a mis-premised sub-item. But F1 below is exactly the failure mode #32 exists to prevent, so merging as-is re-opens the ticket in substance.

Method

Published a throwaway probe40 module: table item (id, alpha, beta) plus view flip returning (beta, alpha, id) — deliberately reversed. Drove v1.json.spacetimedb with EVERY scenario on its own fresh connection (stricter than the author's sequential-sweep-then-fresh-connection design; dedup could not mask anything because no query was ever subscribed twice on one socket). Then a second probe against the live chat db to test a row with non-scalar columns. probe40 deleted afterwards (schema endpoint now 404); repo tree left pristine.

Confirmed correct

pr_table — Every cell reproduced. TransactionUpdate table rows positional for BOTH inserts ([3,"a3","b3"]) and deletes ([1,"a1","b1"], [2,"a2","b2"], [3,"a3","b3"]). InitialSubscription / SubscribeApplied / SubscribeMultiApplied / UnsubscribeApplied / UnsubscribeMultiApplied all named for a table. View positional in all seven message types.
oneoffquery_named — CONFIRMED, and this is genuinely new — the committed fixture only ever captured an empty one (session.jsonl frame 7, "rows":[]). Against a NON-EMPTY table: OneOffQueryResponse{..."rows":["{\"id\":1,\"alpha\":\"a1\",\"beta\":\"b1\"}",...]}. Named.
views_positional_in_readbacks — CONFIRMED, and stronger than the PR claims. The view's rows arrive as ["b1","a1",1] — the VIEW's own declared order (beta, alpha, id), not the table's (id, alpha, beta). So it is not merely 'positional', it is positional in an order a client cannot derive from the table at all. Same in InitialSubscription, SubscribeApplied, UnsubscribeApplied, SubscribeMultiApplied, UnsubscribeMultiApplied, OneOffQueryResponse and TransactionUpdate.
dedup_workaround — The author's method was sufficient. Dedup is real (fixture frame 15: second identical SubscribeMulti returns "tables":[]). My design never reused a query on a connection, so it could not be fooled, and it reproduced the author's results exactly.
frame_numbers — Frames 4, 5, 8, 13 ARE the TransactionUpdates carrying message rows as bare arrays (4/8/13 inserts, 5 deletes). Frame 2 is a TransactionUpdate with "tables":[], correctly excluded.
sabotage — All three PR-body failure strings reproduce verbatim: reorder -> FAIL generated column order matches the module, which decodes the TransactionUpdate rows in frames 4/5/8/13: expected id,sender,text, got id,text,sender; drop type tag -> expected 3, got 2; drop profile column -> expected 8, got 9. Suite: 5 passed, 0 failed.
dependency_is_real — Neutered the order assertion, reordered the fixture columns, re-ran: fails at FAIL positional row decoded and stored. So the TransactionUpdate assertions genuinely mis-map on a reorder — via frames 4 and 5.
fixture_regen — chat_schema.lua diff is comment-only — 0 non-comment changed lines, 8 insertions all --. Regenerated from the live server through stylua: BYTE-IDENTICAL. Re-verified a second time after unrelated data writes: still byte-identical.
views_not_emitted — TRUE. Ran the generator against probe40: output contains item only, no flip. The view lives in misc_exports[].View with return_type: {Array: {Ref: 3}}, a shape the generator's .tables[] walk never visits. Its row type (typespace ref 3, [beta, alpha, id]) is also dropped from the emitted typespace, since that is seeded only from tables/reducers/procedures.

Findings

F1 — HIGH — blocking: The nested-product half of the new reason (2) is false, and it is stamped into every generated file

dev/tools/gen-schema.sh:15-18, dev/tools/gen-schema.sh:356-361 (the stamped header), dev/test/fixtures/chat_schema.lua:4-7, README.md:312-315, README.md:387-389, stdb/schema.lua:3-8 (by reference)

Claim: 'A sum_<Name> column arrives as [ORDINAL, payload] and a product_<Name> column as a positional array, neither carrying a name. The typespace variant and field order is therefore what makes those columns decodable at all.'

Evidence:

Same `profile` row, both encodings, captured minutes apart off 2.8.2.

NAMED readback (identical in InitialSubscription, SubscribeMultiApplied and OneOffQueryResponse):
  {"realm":"eu","handle":"yuris","slug":"yuris-eu","owner":{"__identity__":"0xc200…"},"joined":{"__timestamp_micros_since_unix_epoch__":1787160311786122},"nickname":[0,"nick"],"tags":["a","b"],"status":[0,{}],"home":{"x":0,"y":0}}

POSITIONAL TransactionUpdate:
  ["na","vic","vic-na",["0xc200…"],[1787160338272838],[0,"v"],["z"],[0,[]],[0,0]]

Analysis: The two encodings differ RECURSIVELY, not just at the top level. home (nested product Point) is {"x":0,"y":0} in a readback and [0,0] in a TransactionUpdate. So a nested product does NOT 'arrive positionally, carrying no names' — it does so only inside a positional row. The sum half survives: status is [0,{}] / [0,[]], ordinal-keyed in BOTH encodings, so variant order genuinely is required unconditionally. Only the product half is wrong. This matters structurally: the PR's ranking argument rests on reason (2) being independent of row shape, and for nested products it is not — it collapses back into reason (3). Also note the payload itself differs ({} vs []), which no comment mentions.

Fix: Split the sum and product cases. Sum: variant order required in both encodings (true, keep as is). Nested product: field order required only in the positional encoding. Then drop the 'has nothing to do with row shape' framing, since half of reason (2) is row shape.

F2 — MEDIUM-HIGH — blocking: The kept test names frames 8 and 13 as decoded; the test never decodes them

dev/test/test_fixtures.lua:88-98

Claim: 'frames 4, 5, 8 and 13 are TransactionUpdates whose message rows arrive as bare arrays … and those are decoded below through exactly this list.' The assertion message repeats it: '… which decodes the TransactionUpdate rows in frames 4/5/8/13'.

Evidence:

`grep -on 'frames\[[0-9]*\]' dev/test/test_fixtures.lua` — the file references frames 1,3,4,5,6,7,9,10,11,12,14,15,16,17,18,19,20,21. Frames 2, 8 and 13 are never decoded. Line 234 says so explicitly for frame 8: 'Frame 8 seeded one row; drive it through a fresh cache so the counts below are about the subscription frames alone.' Confirmed behaviourally: with the order assertion neutered, the reorder fails at frame 4's assertion (`positional row decoded and stored`), never reaching 8 or 13.

Analysis: The WIRE fact is right — 8 and 13 do carry bare arrays. The claim about the TEST is wrong. This ships in a user-facing failure string, so a maintainer chasing it opens the file and finds two of the four frames absent. Precisely the confident-comment-that-does-not-match-the-code pattern that #32 was filed about.

Fix: 'frames 4 and 5 are TransactionUpdates whose message rows arrive as bare arrays and are decoded below through exactly this list (4 inserts, 5 deletes); 8 and 13 carry the same encoding but are not replayed here.' Trim the assertion message to … in frames 4/5.

F3 — MEDIUM: 'Strongest reason' / 'descending order of how badly it breaks' is inverted for the code as it stands today

dev/tools/gen-schema.sh:8-11, README.md:309-316, dev/test/test_fixtures.lua:78-82

Evidence:

`grep -rn 'types' stdb/` returns ZERO hits. `stdb/schema.lua:19-42` validates only `columns` and `pk`. Nothing in `stdb/` reads `types` or `typespace`; the only consumers repo-wide are the generator that emits them and the assertions in test_fixtures.lua. Meanwhile `columns` is consumed live by `protocol.decode_row` (protocol.lua:173-183) via `cache.apply_database_update` (cache.lua:93-108).

Analysis: Reason (3), row shape, is the ONLY reason that breaks anything today — the sabotage proves it (positional row decoded and stored fails). Reasons (1) and (2) are dependencies of a typed decoder that does not exist yet. gen-schema.sh's 'get one wrong and a client mis-maps or mis-types fields with no error anywhere' — no client in this repo types fields. 'the difference between decoding and not' — nothing decodes sums structurally. The file-level property (parallel arrays; typespace order) is real, so the assertions added are worth keeping; the RANKING and the present-tense breakage language are not.

Fix: Say plainly that (1) and (2) are future dependencies for typed decoding and (3) is the one with a live consumer today, or drop the ordering claim. The assignment's rule applies verbatim: a stated reason that is a future dependency should say so.

F4 — MEDIUM: Completeness sweep missed dev/server/index.js — the exact unconditional claim the PR set out to remove

dev/server/index.js:7-9

Claim: 'message is byte-frozen: the captured fixtures, both conformance harnesses and the cache tests all decode its rows positionally, so reordering or retyping a column there breaks them.'

Analysis: Half wrong, unconditionally stated. The fixture test decodes frame 3 (InitialSubscription) as a NAMED object and asserts on it (test_fixtures.lua:188-194); test_cache.lua:53-55 covers the named encoding too; dev/example/main/smoke.script:92 subscribes, so its readback is named. The conclusion (reordering breaks them) survives, the reason given does not. Sweep otherwise clean: I checked every tracked file — .forgejo/workflows/*, dev/example/, dev/tools/conformance.sh, dev/tools/html5-conformance.mjs, stdb/cache.lua:10, stdb/connection.lua, patches/. No other miss.

Fix: '… decode its rows through the declared column order — positionally in the TransactionUpdate paths, by name in the subscription readbacks.'

F5 — MEDIUM: README presents a positional profile row as 'captured off a live subscription', contradicting the PR's own new wire-fact #3

README.md:378-389

Claim: 'Captured off a live v1.json.spacetimedb subscription, a profile row whose status is Idle … ["eu","yuris","yuris-eu",["0xc200…"],[1787158263465067],[0,"nick"],["a","b"],[0,[]],[0,0]]'

Evidence:

I subscribed to `profile` on the live chat db three ways. All three readbacks are named: `{"realm":"eu",…,"status":[0,{}],"home":{"x":0,"y":0}}`. The positional form appears only in TransactionUpdate. So that README row is a TransactionUpdate row.

Analysis: Loosely true ('you get it because you subscribed') but it now collides head-on with the new #3 at README:21-27, three hundred lines earlier, which says every subscription readback sends named objects. Line 388's 'a nested product is a positional array ([0,0])' is the same F1 defect. Reconciling exactly this kind of collision is the PR's job.

Fix: Relabel as 'Captured from a TransactionUpdate on a live v1.json.spacetimedb socket', and add the named counterpart so the sum-vs-product asymmetry from F1 is visible in one place.

F6 — LOW — over-correction: Headline denies the axis the body then reintroduces

README.md:21, stdb/protocol.lua:14

Claim: 'Row shape depends on the message, not the table.' / 'Row shape depends on the MESSAGE, not the table'

Analysis: Two lines later both say a view is positional in every message. Shape depends on the message AND on whether the relation is a table or a view. As written the bold clause is contradicted by its own paragraph. This is the only over-correction in the diff — I found no comment claiming rows are always named.

Fix: 'Row shape depends on the message and on whether the relation is a table or a view — never on the table alone.'

dev/tools/gen-schema.sh:131-135, stdb/cache.lua:90-108, stdb/protocol.lua:173-183

Analysis: The bullet is accurate but frames the gap as 'not emitted'. It is worse than that. A view arrives under its own table_name/table_id (my probe: table_name:"flip", table_id:4097) and routes through the same cache.apply_database_update path as a table. table_def returns {} for an unknown relation, def.columns is nil, decode_row(encoded, nil) sets columns = {}, the mapping loop runs zero times and returns {}. So subscribing to a view today stores an EMPTY row and counts it as an insert, with no error anywhere. And because the view's positional order is its own (beta, alpha, id) and not the table's (id, alpha, beta), a client cannot work around it by reusing the table's column list. Verdict on the assignment's question: yes, worth its own issue, and it should be filed as a silent-empty-row bug rather than a generator nicety. Views are the case where column order is most load-bearing and the only case the generator does not cover — the generated header now tells you views are positional while withholding the one thing that would let you decode them.

F8 — Follow-up issue — out of PR scope but undercuts its thesis: undefined

stdb/protocol.lua:90-113

Analysis: unwrap only recognises the NAMED wrapper forms (__identity__, __connection_id__, __timestamp_micros_since_unix_epoch__, __time_duration_micros__, some, none). In a positional row the same values arrive as ["0xc200…"], [1787160338272838], [0,"nick"], so unwrap returns the raw array. Pre-existing, no live consumer (the only positional table in the fixtures, message, is all scalars; profile is never subscribed). Relevant here because the PR now documents the positional path as 'the live path for all incremental table traffic and for views everywhere' without noting the SDK's own unwrap does not understand it. Another reason row shape is not 'the lesser reason'.

F9 — LOW: An #32 checklist item is silently dropped

Analysis: #32 asked to determine empirically whether 'older ?version= schema endpoints' change anything. There is no older one: ?version=8 returns 400 (unknown variant \8`, expected `9` or `10`). There is a NEWER ?version=10with asections[]shape — the onespacetime describe --json` emits and that schema.lua:7-8 and README already reference. The sub-item was mis-premised, and dropping it is right, but a PR saying 'Closes #32' should say so in one line rather than leave the box unaccounted for.

Non-blocking nits

[
"README.md:400-403 wraps the sample failure across two lines; the real output is one line, so a reader grepping the pasted string will not match.",
"dev/tools/gen-schema.sh emits params = { } / types = { } (double space) for a zero-arg reducer — pre-existing, stylua fixes it in the dev-server.sh path, cosmetic only.",
"protocol.lua:157-158 'a known column name being present is the only signal' is accurate but worth noting it is a heuristic: a row that decodes to an empty table takes the positional branch and yields {}. That is the mechanism behind F7."
]

Merge recommendation

Fix F1 and F2, then merge. F3 and F4 should go in the same pass — F3 because 'this comment justifies a mechanism no code exercises' is the literal text of #32, and F4 because it is the one file the completeness sweep missed. F5 and F6 are cheap wording fixes worth taking while the file is open. F7 and F8 as separate issues; F7 should be filed as a bug, not an enhancement.

## Review — automated reviewer, posted for the record **Verdict:** Request changes. The table is correct — I reproduced every cell independently — but the PR introduces one NEW wrong wire claim (the fourth) and one wrong claim about its own test, both stamped into generated output and user-facing failure text. **Closes #32:** Not yet. Items 2 and 3 of #32's checklist are done well. Item 1 is done except a mis-premised sub-item. But F1 below is exactly the failure mode #32 exists to prevent, so merging as-is re-opens the ticket in substance. ### Method Published a throwaway `probe40` module: table `item` (id, alpha, beta) plus view `flip` returning (beta, alpha, id) — deliberately reversed. Drove `v1.json.spacetimedb` with EVERY scenario on its own fresh connection (stricter than the author's sequential-sweep-then-fresh-connection design; dedup could not mask anything because no query was ever subscribed twice on one socket). Then a second probe against the live `chat` db to test a row with non-scalar columns. `probe40` deleted afterwards (schema endpoint now 404); repo tree left pristine. ### Confirmed correct - **pr_table** — Every cell reproduced. TransactionUpdate table rows positional for BOTH inserts (`[3,"a3","b3"]`) and deletes (`[1,"a1","b1"]`, `[2,"a2","b2"]`, `[3,"a3","b3"]`). InitialSubscription / SubscribeApplied / SubscribeMultiApplied / UnsubscribeApplied / UnsubscribeMultiApplied all named for a table. View positional in all seven message types. - **oneoffquery_named** — CONFIRMED, and this is genuinely new — the committed fixture only ever captured an empty one (session.jsonl frame 7, `"rows":[]`). Against a NON-EMPTY table: `OneOffQueryResponse{..."rows":["{\"id\":1,\"alpha\":\"a1\",\"beta\":\"b1\"}",...]}`. Named. - **views_positional_in_readbacks** — CONFIRMED, and stronger than the PR claims. The view's rows arrive as `["b1","a1",1]` — the VIEW's own declared order (beta, alpha, id), not the table's (id, alpha, beta). So it is not merely 'positional', it is positional in an order a client cannot derive from the table at all. Same in InitialSubscription, SubscribeApplied, UnsubscribeApplied, SubscribeMultiApplied, UnsubscribeMultiApplied, OneOffQueryResponse and TransactionUpdate. - **dedup_workaround** — The author's method was sufficient. Dedup is real (fixture frame 15: second identical SubscribeMulti returns `"tables":[]`). My design never reused a query on a connection, so it could not be fooled, and it reproduced the author's results exactly. - **frame_numbers** — Frames 4, 5, 8, 13 ARE the TransactionUpdates carrying `message` rows as bare arrays (4/8/13 inserts, 5 deletes). Frame 2 is a TransactionUpdate with `"tables":[]`, correctly excluded. - **sabotage** — All three PR-body failure strings reproduce verbatim: reorder -> `FAIL generated column order matches the module, which decodes the TransactionUpdate rows in frames 4/5/8/13: expected id,sender,text, got id,text,sender`; drop type tag -> `expected 3, got 2`; drop profile column -> `expected 8, got 9`. Suite: 5 passed, 0 failed. - **dependency_is_real** — Neutered the order assertion, reordered the fixture columns, re-ran: fails at `FAIL positional row decoded and stored`. So the TransactionUpdate assertions genuinely mis-map on a reorder — via frames 4 and 5. - **fixture_regen** — chat_schema.lua diff is comment-only — 0 non-comment changed lines, 8 insertions all `--`. Regenerated from the live server through stylua: BYTE-IDENTICAL. Re-verified a second time after unrelated data writes: still byte-identical. - **views_not_emitted** — TRUE. Ran the generator against probe40: output contains `item` only, no `flip`. The view lives in `misc_exports[].View` with `return_type: {Array: {Ref: 3}}`, a shape the generator's `.tables[]` walk never visits. Its row type (typespace ref 3, `[beta, alpha, id]`) is also dropped from the emitted `typespace`, since that is seeded only from tables/reducers/procedures. ### Findings #### F1 — HIGH — blocking: The nested-product half of the new reason (2) is false, and it is stamped into every generated file `dev/tools/gen-schema.sh:15-18`, `dev/tools/gen-schema.sh:356-361 (the stamped header)`, `dev/test/fixtures/chat_schema.lua:4-7`, `README.md:312-315`, `README.md:387-389`, `stdb/schema.lua:3-8 (by reference)` **Claim:** 'A `sum_<Name>` column arrives as [ORDINAL, payload] and a `product_<Name>` column as a positional array, neither carrying a name. The `typespace` variant and field order is therefore what makes those columns decodable at all.' **Evidence:** ``` Same `profile` row, both encodings, captured minutes apart off 2.8.2. NAMED readback (identical in InitialSubscription, SubscribeMultiApplied and OneOffQueryResponse): {"realm":"eu","handle":"yuris","slug":"yuris-eu","owner":{"__identity__":"0xc200…"},"joined":{"__timestamp_micros_since_unix_epoch__":1787160311786122},"nickname":[0,"nick"],"tags":["a","b"],"status":[0,{}],"home":{"x":0,"y":0}} POSITIONAL TransactionUpdate: ["na","vic","vic-na",["0xc200…"],[1787160338272838],[0,"v"],["z"],[0,[]],[0,0]] ``` **Analysis:** The two encodings differ RECURSIVELY, not just at the top level. `home` (nested product Point) is `{"x":0,"y":0}` in a readback and `[0,0]` in a TransactionUpdate. So a nested product does NOT 'arrive positionally, carrying no names' — it does so only inside a positional row. The sum half survives: `status` is `[0,{}]` / `[0,[]]`, ordinal-keyed in BOTH encodings, so variant order genuinely is required unconditionally. Only the product half is wrong. This matters structurally: the PR's ranking argument rests on reason (2) being independent of row shape, and for nested products it is not — it collapses back into reason (3). Also note the payload itself differs (`{}` vs `[]`), which no comment mentions. **Fix:** Split the sum and product cases. Sum: variant order required in both encodings (true, keep as is). Nested product: field order required only in the positional encoding. Then drop the 'has nothing to do with row shape' framing, since half of reason (2) is row shape. #### F2 — MEDIUM-HIGH — blocking: The kept test names frames 8 and 13 as decoded; the test never decodes them `dev/test/test_fixtures.lua:88-98` **Claim:** 'frames 4, 5, 8 and 13 are TransactionUpdates whose `message` rows arrive as bare arrays … and those are decoded below through exactly this list.' The assertion message repeats it: '… which decodes the TransactionUpdate rows in frames 4/5/8/13'. **Evidence:** ``` `grep -on 'frames\[[0-9]*\]' dev/test/test_fixtures.lua` — the file references frames 1,3,4,5,6,7,9,10,11,12,14,15,16,17,18,19,20,21. Frames 2, 8 and 13 are never decoded. Line 234 says so explicitly for frame 8: 'Frame 8 seeded one row; drive it through a fresh cache so the counts below are about the subscription frames alone.' Confirmed behaviourally: with the order assertion neutered, the reorder fails at frame 4's assertion (`positional row decoded and stored`), never reaching 8 or 13. ``` **Analysis:** The WIRE fact is right — 8 and 13 do carry bare arrays. The claim about the TEST is wrong. This ships in a user-facing failure string, so a maintainer chasing it opens the file and finds two of the four frames absent. Precisely the confident-comment-that-does-not-match-the-code pattern that #32 was filed about. **Fix:** 'frames 4 and 5 are TransactionUpdates whose `message` rows arrive as bare arrays and are decoded below through exactly this list (4 inserts, 5 deletes); 8 and 13 carry the same encoding but are not replayed here.' Trim the assertion message to `… in frames 4/5`. #### F3 — MEDIUM: 'Strongest reason' / 'descending order of how badly it breaks' is inverted for the code as it stands today `dev/tools/gen-schema.sh:8-11`, `README.md:309-316`, `dev/test/test_fixtures.lua:78-82` **Evidence:** ``` `grep -rn 'types' stdb/` returns ZERO hits. `stdb/schema.lua:19-42` validates only `columns` and `pk`. Nothing in `stdb/` reads `types` or `typespace`; the only consumers repo-wide are the generator that emits them and the assertions in test_fixtures.lua. Meanwhile `columns` is consumed live by `protocol.decode_row` (protocol.lua:173-183) via `cache.apply_database_update` (cache.lua:93-108). ``` **Analysis:** Reason (3), row shape, is the ONLY reason that breaks anything today — the sabotage proves it (`positional row decoded and stored` fails). Reasons (1) and (2) are dependencies of a typed decoder that does not exist yet. gen-schema.sh's 'get one wrong and a client mis-maps or mis-types fields with no error anywhere' — no client in this repo types fields. 'the difference between decoding and not' — nothing decodes sums structurally. The file-level property (parallel arrays; typespace order) is real, so the assertions added are worth keeping; the RANKING and the present-tense breakage language are not. **Fix:** Say plainly that (1) and (2) are future dependencies for typed decoding and (3) is the one with a live consumer today, or drop the ordering claim. The assignment's rule applies verbatim: a stated reason that is a future dependency should say so. #### F4 — MEDIUM: Completeness sweep missed dev/server/index.js — the exact unconditional claim the PR set out to remove `dev/server/index.js:7-9` **Claim:** '`message` is byte-frozen: the captured fixtures, both conformance harnesses and the cache tests all decode its rows positionally, so reordering or retyping a column there breaks them.' **Analysis:** Half wrong, unconditionally stated. The fixture test decodes frame 3 (InitialSubscription) as a NAMED object and asserts on it (test_fixtures.lua:188-194); test_cache.lua:53-55 covers the named encoding too; dev/example/main/smoke.script:92 subscribes, so its readback is named. The conclusion (reordering breaks them) survives, the reason given does not. Sweep otherwise clean: I checked every tracked file — .forgejo/workflows/*, dev/example/, dev/tools/conformance.sh, dev/tools/html5-conformance.mjs, stdb/cache.lua:10, stdb/connection.lua, patches/. No other miss. **Fix:** '… decode its rows through the declared column order — positionally in the TransactionUpdate paths, by name in the subscription readbacks.' #### F5 — MEDIUM: README presents a positional profile row as 'captured off a live subscription', contradicting the PR's own new wire-fact #3 `README.md:378-389` **Claim:** 'Captured off a live `v1.json.spacetimedb` subscription, a `profile` row whose `status` is `Idle` … ["eu","yuris","yuris-eu",["0xc200…"],[1787158263465067],[0,"nick"],["a","b"],[0,[]],[0,0]]' **Evidence:** ``` I subscribed to `profile` on the live chat db three ways. All three readbacks are named: `{"realm":"eu",…,"status":[0,{}],"home":{"x":0,"y":0}}`. The positional form appears only in TransactionUpdate. So that README row is a TransactionUpdate row. ``` **Analysis:** Loosely true ('you get it because you subscribed') but it now collides head-on with the new #3 at README:21-27, three hundred lines earlier, which says every subscription readback sends named objects. Line 388's 'a nested product is a positional array ([0,0])' is the same F1 defect. Reconciling exactly this kind of collision is the PR's job. **Fix:** Relabel as 'Captured from a TransactionUpdate on a live `v1.json.spacetimedb` socket', and add the named counterpart so the sum-vs-product asymmetry from F1 is visible in one place. #### F6 — LOW — over-correction: Headline denies the axis the body then reintroduces `README.md:21`, `stdb/protocol.lua:14` **Claim:** 'Row shape depends on the message, not the table.' / 'Row shape depends on the MESSAGE, not the table' **Analysis:** Two lines later both say a view is positional in every message. Shape depends on the message AND on whether the relation is a table or a view. As written the bold clause is contradicted by its own paragraph. This is the only over-correction in the diff — I found no comment claiming rows are always named. **Fix:** 'Row shape depends on the message and on whether the relation is a table or a view — never on the table alone.' #### F7 — Follow-up issue — recommend filing, and the Limitations bullet understates it: undefined `dev/tools/gen-schema.sh:131-135`, `stdb/cache.lua:90-108`, `stdb/protocol.lua:173-183` **Analysis:** The bullet is accurate but frames the gap as 'not emitted'. It is worse than that. A view arrives under its own `table_name`/`table_id` (my probe: `table_name:"flip"`, `table_id:4097`) and routes through the same `cache.apply_database_update` path as a table. `table_def` returns `{}` for an unknown relation, `def.columns` is nil, `decode_row(encoded, nil)` sets `columns = {}`, the mapping loop runs zero times and returns `{}`. So subscribing to a view today stores an EMPTY row and counts it as an insert, with no error anywhere. And because the view's positional order is its own (beta, alpha, id) and not the table's (id, alpha, beta), a client cannot work around it by reusing the table's column list. Verdict on the assignment's question: yes, worth its own issue, and it should be filed as a silent-empty-row bug rather than a generator nicety. Views are the case where column order is most load-bearing and the only case the generator does not cover — the generated header now tells you views are positional while withholding the one thing that would let you decode them. #### F8 — Follow-up issue — out of PR scope but undercuts its thesis: undefined `stdb/protocol.lua:90-113` **Analysis:** `unwrap` only recognises the NAMED wrapper forms (`__identity__`, `__connection_id__`, `__timestamp_micros_since_unix_epoch__`, `__time_duration_micros__`, `some`, `none`). In a positional row the same values arrive as `["0xc200…"]`, `[1787160338272838]`, `[0,"nick"]`, so `unwrap` returns the raw array. Pre-existing, no live consumer (the only positional table in the fixtures, `message`, is all scalars; `profile` is never subscribed). Relevant here because the PR now documents the positional path as 'the live path for all incremental table traffic and for views everywhere' without noting the SDK's own unwrap does not understand it. Another reason row shape is not 'the lesser reason'. #### F9 — LOW: An #32 checklist item is silently dropped **Analysis:** #32 asked to determine empirically whether 'older `?version=` schema endpoints' change anything. There is no older one: `?version=8` returns 400 (`unknown variant \`8\`, expected \`9\` or \`10\``). There is a NEWER `?version=10` with a `sections[]` shape — the one `spacetime describe --json` emits and that schema.lua:7-8 and README already reference. The sub-item was mis-premised, and dropping it is right, but a PR saying 'Closes #32' should say so in one line rather than leave the box unaccounted for. ### Non-blocking nits [ "README.md:400-403 wraps the sample failure across two lines; the real output is one line, so a reader grepping the pasted string will not match.", "dev/tools/gen-schema.sh emits `params = { }` / `types = { }` (double space) for a zero-arg reducer — pre-existing, stylua fixes it in the dev-server.sh path, cosmetic only.", "protocol.lua:157-158 'a known column name being present is the only signal' is accurate but worth noting it is a heuristic: a row that decodes to an empty table takes the positional branch and yields `{}`. That is the mechanism behind F7." ] ### Merge recommendation Fix F1 and F2, then merge. F3 and F4 should go in the same pass — F3 because 'this comment justifies a mechanism no code exercises' is the literal text of #32, and F4 because it is the one file the completeness sweep missed. F5 and F6 are cheap wording fixes worth taking while the file is open. F7 and F8 as separate issues; F7 should be filed as a bug, not an enhancement.

yuris referenced this pull request

2026-08-19 17:29:48 +00:00

Subscribing to a view silently stores empty rows #42

yuris force-pushed feat/32-column-order from 9cbfd72c05

ci / ci (pull_request) Successful in 20s

Details

to 42bc3e2fbf

ci / ci (pull_request) Successful in 19s

Details

2026-08-19 17:43:22 +00:00

Compare

yuris force-pushed feat/32-column-order from 42bc3e2fbf