yuris/spacetimedb-defold

Fork 0

feat: reachable schema generation, and a README that describes the SDK #97

Merged

yuris merged 4 commits from schema-reachable into main

2026-08-21 13:35:06 +00:00

yuris commented

2026-08-21 13:02:18 +00:00

Owner

Closes nothing on its own; it ticks the first DX items on #77, which this session rewrote from the stale Roadmap: what comes after v1.1.0 into Developer experience and documentation. The wiki half of that work is already pushed as 662d17c in the wiki repo.

The defect. The SDK's headline decoding path needs a generated schema, and since 2.0.0 flipped the default to v2 a table with no generated types cannot be decoded at all — it is skipped and reported as undescribed. The generator the docs told a consumer to run was dev/tools/gen-schema.sh: bash, curl and jq. .gitattributes export-ignores /dev, so git archive HEAD hands a consumer 13 entries and none of them is that script. The one artefact the default wire cannot work without was the one artefact we did not deliver. Nothing about it was reachable — not the tool, not a fallback, not a documented way to hand-write around it.

Two reachable paths, one transform. stdb/typetags.lua is the jq walker of gen-schema.sh:171-459 ported to Lua, and it is the only copy of those rules that ships. stdb.schema.fetch(opts, cb) reads <base>/v1/database/<db>/schema?version=9 through the http_seam and calls back a ready gen table — that is the development path, no external tool at all. stdb.schema.to_lua(gen) serialises the same table to a loadable module with the same -- GENERATED header the shell script writes — that is the release path, a file you commit and pin. Both go through typetags.build, so the dev path and the pinned path cannot describe the schema differently.

gen-schema.sh stays exactly where it is and keeps its existing job: conformance.sh:61-76 regenerates and diffs it against dev/test/fixtures/chat_schema.lua on every conformance run. It is the contributor's drift check against a live server, not a consumer deliverable, and it never was one.

The seam grew a get. http_seam exposed post only, because stdb.auth.exchange was the only caller. stdb/auth.lua now also exposes get(url, cb), same (status, body) callback and same status == 0 meaning nothing answered. Calling http.request straight out of schema.lua would have been one line shorter and would have thrown away the injection point the unit tests need. No headers parameter: nothing that gets sends any, and an argument every caller passes as nil is a lie about the interface.

One click in the editor. stdb/spacetimedb.editor_script adds Project → SpacetimeDB: Regenerate schema. It lives under stdb/ because include_dirs = stdb is the only directory Defold exposes to a consumer project, it reads its defaults from spacetimedb.base_url and spacetimedb.database in the consumer's game.project, fetches with editor.execute("curl", ...), and writes /my_schema.lua. It carries its own JSON decoder, ~60 lines, because the editor's Lua 5.2 runtime has no JSON module — and requiring jq in the editor would have reinstated the exact dependency this PR exists to remove. That decoder is editor-only and never reaches a game bundle; the runtime path uses the engine's builtin, injected as before.

Shipping the shell script and shelling out to it was the obvious shortcut and is not possible: the manual is explicit that a dependency's files "do not exist as files (they are entries in a zip archive)", and only ext.manifest and plugins/bin/${platform} are ever extracted. There is nothing on disk to execute.

README. Now describes the SDK instead of the project's status. It leads with what the thing is, then a ~20-line example that runs verbatim on the default wire via stdb.schema.fetch — the README had no code at all, which was the worst omission for anyone deciding whether to use this. Feature bullets, Install unchanged, Layout now states delivered-vs-repo, and a Documentation section that splits the wiki by audience. The two-wire narrative, the f32 mantissa paragraph and the capture-provenance sentence moved to the wiki rather than being cut. No #N reference survives in it.

The pin, in dev/test/test_fixtures.lua. A port with no adversary is a guess, so dev/test/fixtures/chat_schema.json is now committed — the raw schema body — and the new cases decode it, run typetags.build, and deep-equal the result against the chat_schema.lua the jq walker produced. That closes the loop: conformance.sh pins jq to the live server, this pins Lua to jq's committed output, and the two walkers cannot drift apart in silence. A to_lua round-trip case asserts load(to_lua(build(d)))() deep-equals build(d). New cases went into the existing file — runner.lua counts suites, and the count stays 11.

Verified

lua dev/test/runner.lua — 11/11.
stylua --check — clean, now including the editor script, which is Lua the checker was walking past because it is not named *.lua.
Conformance PASS on both wires, v2 and v1, against a live server.
schema.fetch driven against that live server returns { "u64", "string", "string" } for message — the value committed at chat_schema.lua:106.
The pin fails when the walker is broken. Corrupting one scalar mapping in typetags.lua makes the fixture case fail by name; reverting it goes green. A pin that cannot fail is decoration.

Not verifiable here. The editor command cannot be unit-tested — there is no editor to drive from CI, and its whole surface is editor.execute/editor.save. It needs a hand check: open a scratch project that depends on this SDK, Fetch Libraries, then Project → SpacetimeDB: Regenerate schema, and confirm the written /my_schema.lua requires cleanly and drives a connect. If the command does not appear in the menu, the thing to check first is whether an .editor_script in a library can require a .lua module from that same library; the fallback, if it cannot, is to inline the walker there and pin both copies equal in the fixture test.

Closes nothing on its own; it ticks the first DX items on #77, which this session rewrote from the stale `Roadmap: what comes after v1.1.0` into `Developer experience and documentation`. The wiki half of that work is already pushed as `662d17c` in the wiki repo. **The defect.** The SDK's headline decoding path needs a generated schema, and since 2.0.0 flipped the default to v2 a table with no generated `types` cannot be decoded at all — it is skipped and reported as `undescribed`. The generator the docs told a consumer to run was `dev/tools/gen-schema.sh`: bash, `curl` and `jq`. `.gitattributes` export-ignores `/dev`, so `git archive HEAD` hands a consumer 13 entries and none of them is that script. The one artefact the default wire cannot work without was the one artefact we did not deliver. Nothing about it was reachable — not the tool, not a fallback, not a documented way to hand-write around it. **Two reachable paths, one transform.** `stdb/typetags.lua` is the jq walker of `gen-schema.sh:171-459` ported to Lua, and it is the only copy of those rules that ships. `stdb.schema.fetch(opts, cb)` reads `<base>/v1/database/<db>/schema?version=9` through the `http_seam` and calls back a ready `gen` table — that is the development path, no external tool at all. `stdb.schema.to_lua(gen)` serialises the same table to a loadable module with the same `-- GENERATED` header the shell script writes — that is the release path, a file you commit and pin. Both go through `typetags.build`, so the dev path and the pinned path cannot describe the schema differently. `gen-schema.sh` stays exactly where it is and keeps its existing job: `conformance.sh:61-76` regenerates and diffs it against `dev/test/fixtures/chat_schema.lua` on every conformance run. It is the contributor's drift check against a live server, not a consumer deliverable, and it never was one. **The seam grew a `get`.** `http_seam` exposed `post` only, because `stdb.auth.exchange` was the only caller. `stdb/auth.lua` now also exposes `get(url, cb)`, same `(status, body)` callback and same `status == 0` meaning nothing answered. Calling `http.request` straight out of `schema.lua` would have been one line shorter and would have thrown away the injection point the unit tests need. No headers parameter: nothing that gets sends any, and an argument every caller passes as nil is a lie about the interface. **One click in the editor.** `stdb/spacetimedb.editor_script` adds **Project → SpacetimeDB: Regenerate schema**. It lives under `stdb/` because `include_dirs = stdb` is the only directory Defold exposes to a consumer project, it reads its defaults from `spacetimedb.base_url` and `spacetimedb.database` in the consumer's `game.project`, fetches with `editor.execute("curl", ...)`, and writes `/my_schema.lua`. It carries **its own JSON decoder**, ~60 lines, because the editor's Lua 5.2 runtime has no JSON module — and requiring `jq` in the editor would have reinstated the exact dependency this PR exists to remove. That decoder is editor-only and never reaches a game bundle; the runtime path uses the engine's builtin, injected as before. Shipping the shell script and shelling out to it was the obvious shortcut and is not possible: the manual is explicit that a dependency's files "do not exist as files (they are entries in a zip archive)", and only `ext.manifest` and `plugins/bin/${platform}` are ever extracted. There is nothing on disk to execute. **README.** Now describes the SDK instead of the project's status. It leads with what the thing is, then a ~20-line example that runs verbatim on the default wire via `stdb.schema.fetch` — the README had no code at all, which was the worst omission for anyone deciding whether to use this. Feature bullets, `Install` unchanged, `Layout` now states delivered-vs-repo, and a `Documentation` section that splits the wiki by audience. The two-wire narrative, the `f32` mantissa paragraph and the capture-provenance sentence moved to the wiki rather than being cut. No `#N` reference survives in it. **The pin, in `dev/test/test_fixtures.lua`.** A port with no adversary is a guess, so `dev/test/fixtures/chat_schema.json` is now committed — the raw schema body — and the new cases decode it, run `typetags.build`, and deep-equal the result against the `chat_schema.lua` the jq walker produced. That closes the loop: `conformance.sh` pins jq to the live server, this pins Lua to jq's committed output, and the two walkers cannot drift apart in silence. A `to_lua` round-trip case asserts `load(to_lua(build(d)))()` deep-equals `build(d)`. New cases went into the existing file — `runner.lua` counts suites, and the count stays 11. **Verified** - `lua dev/test/runner.lua` — 11/11. - `stylua --check` — clean, now including the editor script, which is Lua the checker was walking past because it is not named `*.lua`. - Conformance PASS on both wires, v2 and v1, against a live server. - `schema.fetch` driven against that live server returns `{ "u64", "string", "string" }` for `message` — the value committed at `chat_schema.lua:106`. - **The pin fails when the walker is broken.** Corrupting one scalar mapping in `typetags.lua` makes the fixture case fail by name; reverting it goes green. A pin that cannot fail is decoration. **Not verifiable here.** The editor command cannot be unit-tested — there is no editor to drive from CI, and its whole surface is `editor.execute`/`editor.save`. It needs a hand check: open a scratch project that depends on this SDK, **Fetch Libraries**, then **Project → SpacetimeDB: Regenerate schema**, and confirm the written `/my_schema.lua` `require`s cleanly and drives a `connect`. If the command does not appear in the menu, the thing to check first is whether an `.editor_script` in a library can `require` a `.lua` module from that same library; the fallback, if it cannot, is to inline the walker there and pin both copies equal in the fixture test.

yuris added 2 commits

2026-08-21 13:02:19 +00:00

feat: reachable schema generation, and a README that describes the SDK d9a756a66d

Two paths to a generated schema, one Lua transform. stdb/typetags.lua ports
the jq walker in dev/tools/gen-schema.sh, pinned against the same fixture the
shell generator produces, so the two cannot drift. stdb.schema.fetch reads the
schema off a running server for development; stdb.schema.to_lua writes the
pinned module for release. stdb/spacetimedb.editor_script exposes both as one
Project menu command, which is what a consumer can actually reach - dev/ is
export-ignored, so the shell generator never lands on their disk.

The seam in stdb/auth.lua gained get(url, cb); fetch injects through it, so the
unit tests drive it without the engine.

README now describes the SDK and carries a working example.

ci: format-check the editor script too, since stylua only walks *.lua

ci / ci (pull_request) Successful in 1m28s

Details

conformance / conformance (v1) (pull_request) Successful in 2m57s

Details

conformance / conformance (v2) (pull_request) Successful in 2m58s

Details

5df3f66f32

yuris added 1 commit

2026-08-21 13:29:04 +00:00

fix: close the gaps four reviews found in the generator path

ci / ci (pull_request) Successful in 20s

Details

conformance / conformance (v1) (pull_request) Successful in 2m54s

Details

conformance / conformance (v2) (pull_request) Successful in 2m20s

Details

6dcbdaf1c3

A mutation audit killed 24 of 36 single-line bugs, and the misses were all on
the periphery rather than in the walker. The blocker was the new get seam in
stdb/auth.lua: swapping its (status, body) order left the whole suite green,
so the default transport of the headline path was pinned by nothing. It is
exercised now, and so are the status window's edges, the guard messages the
raises() cases only assumed, and the sort to_lua depends on for a diffable
module.

Security: to_lua interpolated its source argument raw into the generated
module, so a newline in it broke out of the comment and became code the
consumer's game runs on require - the one string in the emitter that did not
go through %q. The editor command validated no scheme, and curl reads a
leading dash as an option, so base_url = -K. was a file read primitive from
one menu click. The editor's \u decoder took a hex quad from tonumber, which
honours a sign and leading space, and diverged from json.decode on input the
two are meant to read identically.

Also: goto is reserved on LuaJIT, so a relation named that emitted as a bare
key and broke the generated module on device but not on HTML5; a column
ordinal off the wire is now checked, because a hole in the list serialised to
a string key and stopped the module loading back equal; and conformance.sh
refreshes both fixtures, so the raw capture cannot go stale behind the .lua.

yuris added 1 commit

2026-08-21 13:34:49 +00:00

fix: take the four nits the merge gate found

ci / ci (pull_request) Successful in 21s

Details

conformance / conformance (v1) (pull_request) Successful in 2m29s

Details

conformance / conformance (v2) (pull_request) Failing after 26s

Details

b60ca5d715

An uppercase scheme in game.project was refused by a guard that exists to
block a leading dash, not a capital H. The collapsed fetch guard cases tested
two of the or-chain's four clauses, so deleting either of the other two left
the suite green - proven, and now covered. The empty-module case lost the
assertion that every section is still a table, which is what a consumer on a
fresh database iterates without checking. And schema.endpoint, now public,
raises like every other entry point instead of failing as a concat error
inside the library.

yuris merged commit 9e8cffc73e into main

2026-08-21 13:35:06 +00:00

yuris referenced this pull request from a commit

2026-08-21 13:35:06 +00:00

feat: reachable schema generation, and a README that describes the SDK (#97)

yuris referenced this pull request

2026-08-21 19:52:36 +00:00

Developer experience and documentation #77