yuris/spacetimedb-defold
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 9 Packages Wiki Activity Actions

feat: near parity with the official SDKs, typed query builder included (3.0.0) #102

Merged
yuris merged 15 commits from feat-parity into main 2026-08-21 19:51:34 +00:00
Conversation 0 Commits 15 Files changed 25 +2502 -90
yuris commented 2026-08-21 19:45:53 +00:00
Owner
Copy link

What this is

3.0.0. The SDK reaches near feature parity with the official SpacetimeDB 2.x clients, with the
names a reader of the official docs already knows. The one gap the comparison table called
impossible — a typed query builder — turned out to be possible, and is the largest piece here.

Built from the spacetimedb npm package 2.8.2's own source rather than from the docs site, so
every name below mirrors something real. The research is on the branch's commits; the two
grounding facts that shaped the work:

  • The official query builder compiles to a SQL string in the client (lib/query.ts's
    toSql, consumed by subscription_builder_impl.ts:122-126). Queries always leave the client
    as text. That is why a Lua port is a string templater rather than a query planner.
  • The subscription language is small and fully specified: SELECT * FROM t [JOIN t2 ON col = col] [WHERE pred], ops = != < <= > >=, literals INTEGER FLOAT STRING HEX TRUE FALSE, '
    escaped by doubling, identifiers quotable with ". No projection, no ordering, no limit, at
    most two tables, both join columns indexed.

The breaking change

Every row and lifecycle callback takes a context first, as the official SDKs do:

db.tables.message:on_insert(function(row) end)        -- 2.x
db.tables.message:on_insert(function(ctx, row) end)   -- 3.0.0

Clean cutover: the conformance harness, every test, the README, the wiki and the new example are
all migrated in this branch. No shim, no arity sniffing. ctx carries db, reducers,
procedures, event, and is_active/disconnect/subscription_builder.

What shipped
Typed query builder stdb/query.lua. tables.message:where(function(r) return r.sender:eq("yuris") end) → SELECT * FROM "message" WHERE "message"."sender" = 'yuris'. Operators eq ne lt lte gt gte, combinators and_ or_ not_, left_semijoin/right_semijoin. Unknown column, wrong value type, a u64 above 2^53 passed as a number, a non-hex value for an Identity column, an unindexed join column — each raises at the callsite naming what was wrong. not_ works despite the grammar having no NOT, by inverting comparisons and applying De Morgan.
Contexts, builder, accessors ctx first in every callback; stdb.builder():with_url(...):with_database(...):with_schema(gen):build(); db.reducers.<name>(args, cb) and db.procedures.<name>(args, cb) generated from the schema, raising on a typo with the real names listed; db:subscription_builder():on_applied(...):on_error(...):subscribe(...) with is_active/is_ended/unsubscribe/unsubscribe_then; db.db; db:disconnect().
Connection state and latency db:state() → connecting/connected/reconnecting/closed, assigned only at the five points the SDK already learns of a transition. db:latency() is an EMA at weight 1/8 (TCP's own SRTT smoothing) over request/response pairs the SDK already exchanges — no new traffic, no heartbeat. db:ping(cb) forces one measurement with a one-off query against st_table filtered to match nothing.
Coroutine mode stdb.await(fn, ...) and stdb.await_event(db, type), following nakama-defold's opt-in wrapper rather than auto-detecting. A truthy err raises inside the coroutine instead of being returned for the caller to forget. The state guard is what stops resume on a coroutine that never yielded.
An openable example dev/example-chat/, a real game.project with a GUI, plus a headless entry point that proves the wiring: it sends a message and only prints PASS when that row comes back through on_insert.
.script_api: answered, not built The Defold manual scopes .script_api to native extensions and separately says the editor completes Lua modules a script requires. A pure-Lua library needs none. Recorded on the wiki roadmap as a decision.

Two real defects found on the way

  • The README quickstart omitted reducers/procedures, so following it and calling a reducer
    failed on the default wire with no schema entry for reducer send_message. Found by building
    the example against the documented path.
  • Both quickstarts then called a reducer immediately after connect, where there is no socket
    yet: stdb.connect's pre-socket seam raises send before the socket was created. Sends now sit
    in the connected path, where the conformance harness always had them.

Verification

  • lua dev/test/runner.lua → 12 passed, 0 failed (a new test_query.lua).
  • stylua --check stdb dev/test stdb/spacetimedb.editor_script clean.
  • Native conformance PASS on v2 and on v1; HTML5 conformance PASS on v2 and on v1. The
    harness passing is what proves the ctx migration is complete rather than merely compiling.
  • Six generated queries POSTed to the live server, all accepted, including a join and
    sender = ''' OR 1=1 --' — the escaped injection attempt returns 200 with no rows instead of
    executing.
  • The example built and run headless against a live server: connected, subscribed, sent, and the
    row returned through on_insert.
  • Every new behaviour carries a test proven failable by the mutation it targets: 8 for the
    ergonomics layer, 6 for the query builder, 4 for state/latency, 2 for the coroutine wrapper,
    each reverted to green.

Not in this branch

Frame compression stays out for the reasons #77 records. A CI job for the example's headless proof
is worth adding and is not here; the exact verified command is in the example's README.

Wiki updated in the same shape: c68d269. Refs #77.

## What this is 3.0.0. The SDK reaches near feature parity with the official SpacetimeDB 2.x clients, with the names a reader of the official docs already knows. The one gap the comparison table called impossible — a typed query builder — turned out to be possible, and is the largest piece here. Built from the `spacetimedb` npm package 2.8.2's own source rather than from the docs site, so every name below mirrors something real. The research is on the branch's commits; the two grounding facts that shaped the work: - **The official query builder compiles to a SQL string in the client** (`lib/query.ts`'s `toSql`, consumed by `subscription_builder_impl.ts:122-126`). Queries always leave the client as text. That is why a Lua port is a string templater rather than a query planner. - **The subscription language is small and fully specified**: `SELECT * FROM t [JOIN t2 ON col = col] [WHERE pred]`, ops `= != < <= > >=`, literals `INTEGER FLOAT STRING HEX TRUE FALSE`, `'` escaped by doubling, identifiers quotable with `"`. No projection, no ordering, no limit, at most two tables, both join columns indexed. ## The breaking change Every row and lifecycle callback takes a context first, as the official SDKs do: ```lua db.tables.message:on_insert(function(row) end) -- 2.x db.tables.message:on_insert(function(ctx, row) end) -- 3.0.0 ``` Clean cutover: the conformance harness, every test, the README, the wiki and the new example are all migrated in this branch. No shim, no arity sniffing. `ctx` carries `db`, `reducers`, `procedures`, `event`, and `is_active`/`disconnect`/`subscription_builder`. ## What shipped | | | |---|---| | **Typed query builder** | `stdb/query.lua`. `tables.message:where(function(r) return r.sender:eq("yuris") end)` → `SELECT * FROM "message" WHERE "message"."sender" = 'yuris'`. Operators `eq ne lt lte gt gte`, combinators `and_ or_ not_`, `left_semijoin`/`right_semijoin`. Unknown column, wrong value type, a u64 above 2^53 passed as a number, a non-hex value for an Identity column, an unindexed join column — each raises at the callsite naming what was wrong. `not_` works despite the grammar having no NOT, by inverting comparisons and applying De Morgan. | | **Contexts, builder, accessors** | `ctx` first in every callback; `stdb.builder():with_url(...):with_database(...):with_schema(gen):build()`; `db.reducers.<name>(args, cb)` and `db.procedures.<name>(args, cb)` generated from the schema, raising on a typo with the real names listed; `db:subscription_builder():on_applied(...):on_error(...):subscribe(...)` with `is_active`/`is_ended`/`unsubscribe`/`unsubscribe_then`; `db.db`; `db:disconnect()`. | | **Connection state and latency** | `db:state()` → `connecting`/`connected`/`reconnecting`/`closed`, assigned only at the five points the SDK already learns of a transition. `db:latency()` is an EMA at weight 1/8 (TCP's own SRTT smoothing) over request/response pairs the SDK already exchanges — no new traffic, no heartbeat. `db:ping(cb)` forces one measurement with a one-off query against `st_table` filtered to match nothing. | | **Coroutine mode** | `stdb.await(fn, ...)` and `stdb.await_event(db, type)`, following nakama-defold's opt-in wrapper rather than auto-detecting. A truthy `err` raises inside the coroutine instead of being returned for the caller to forget. The `state` guard is what stops `resume` on a coroutine that never yielded. | | **An openable example** | `dev/example-chat/`, a real `game.project` with a GUI, plus a headless entry point that proves the wiring: it sends a message and only prints PASS when that row comes back through `on_insert`. | | **`.script_api`: answered, not built** | The Defold manual scopes `.script_api` to native extensions and separately says the editor completes Lua modules a script requires. A pure-Lua library needs none. Recorded on the wiki roadmap as a decision. | ## Two real defects found on the way - The README quickstart omitted `reducers`/`procedures`, so following it and calling a reducer failed on the default wire with `no schema entry for reducer send_message`. Found by building the example against the documented path. - Both quickstarts then called a reducer immediately after `connect`, where there is no socket yet: `stdb.connect`'s pre-socket seam raises `send before the socket was created`. Sends now sit in the connected path, where the conformance harness always had them. ## Verification - `lua dev/test/runner.lua` → **12 passed, 0 failed** (a new `test_query.lua`). - `stylua --check stdb dev/test stdb/spacetimedb.editor_script` clean. - Native conformance **PASS on v2 and on v1**; HTML5 conformance **PASS on v2 and on v1**. The harness passing is what proves the ctx migration is complete rather than merely compiling. - Six generated queries POSTed to the live server, all accepted, including a join and `sender = ''' OR 1=1 --'` — the escaped injection attempt returns 200 with no rows instead of executing. - The example built and run headless against a live server: connected, subscribed, sent, and the row returned through `on_insert`. - Every new behaviour carries a test proven failable by the mutation it targets: 8 for the ergonomics layer, 6 for the query builder, 4 for state/latency, 2 for the coroutine wrapper, each reverted to green. ## Not in this branch Frame compression stays out for the reasons #77 records. A CI job for the example's headless proof is worth adding and is not here; the exact verified command is in the example's README. Wiki updated in the same shape: `c68d269`. Refs #77. 
query.tables(gen) returns typed handles for every relation, with
:where(fn) and :join(other, lcol, rcol):where(fn) compiling to the
subscription SQL subset (SELECT * | SELECT t.* FROM ... [JOIN ... ON
col = col] [WHERE predicate]). Column comparisons (eq/ne/lt/le/gt/ge)
validate the Lua value against the column's declared type tag, refuse
u64/i64+ numbers above 2^53 (decimal string required), and require
0x-hex text for Identity/ConnectionId/Uuid/byte columns. Identifiers
are always quoted and string literals always escaped, closing SQL
injection from game-code values. query.all/any/not_ combine and
invert predicates (not_ via De Morgan, since the grammar has no NOT
token). Join columns are checked against the schema's pk/unique/index
metadata and rejected, naming the missing side, before ever reaching
the server.

connection.lua: subscribe_query now accepts a query.lua builder
alongside a plain SQL string, detected by duck-typing on __tostring so
this file gains no dependency on stdb/query.lua.
Minimal project a consumer can File > Open Project and Build: a GUI message
log plus a send button, wired through stdb.schema.fetch -> stdb.connect ->
subscribe_query -> on_insert -> call_reducer against the chat module in
dev/server/. stdb/ is a symlink to the repo's real SDK, not a copy.

Proven headless via headless.project (an overlay analogous to
dev/example/harness.project) driving main/headless.script, which shares its
SDK wiring with the GUI script through main/chat_client.lua. Verified with
bob --variant headless against a live dev server: connect, subscribe, a row
insert from call_reducer's own send_message, PASS.

Along the way: the SDK's own v2 default needs reducers/procedures schema
passed to stdb.connect for BSATN arg encoding (stdb/init.lua documents this),
which the main README's quickstart omits -- call_reducer fails wire-side
without it. Worked around in chat_client.lua; README is out of scope here.
Someone porting a TypeScript callsite should find the operator where they
expect it, so the six comparisons are eq/ne/lt/lte/gt/gte - not le/ge - and the
join is left_semijoin/right_semijoin, which is what the official builder calls
the same operation. left and right name the rows that SURVIVE, not a SQL outer
join: both render one inner JOIN and differ only in the projection.

query.and_/or_ join all/any as the argument-taking spelling the official
module-level combinators use. The underscore is only because and, or and not
are Lua keywords.
db:state() / db:is_connected() / db:latency() / db:ping(cb).

state() is one field (c._state), assigned only at the five places the
SDK already learns of a transition: connection.new (connecting),
credentials() on IdentityToken/InitialConnection (connected, first
connect and every reconnect), schedule_reconnect() (reconnecting when
a retry is scheduled, closed when nothing will replace the socket),
and conn.close() (closed). Never recomputed from the event stream.

latency() is a plain exponential moving average (weight 1/8, matching
TCP's SRTT smoothing per RFC 6298) over every correlated
request/response pair the SDK already exchanges - reducer/procedure
calls, subscribe/unsubscribe, one-off queries - via a sent_at
timestamp on each pending entry and record_rtt() at the ten real
server-reply settle sites. Explicitly excluded from
settle_pending()'s 'connection lost' sweep, which never got a real
reply. Measures application-level round trip including server
processing; not an ICMP ping.

ping(cb) forces a fresh measurement via the cheapest side-effect-free
round trip: a one-off SELECT * FROM st_table WHERE table_name = ''
(a system table on every database, filtered to match nothing). Reuses
one_off_query, so it works identically on both wires with no new wire
support. No timer or background heartbeat added.

connection.lua gained an opts.clock seam (default os.clock, same
pattern as frame_budget's opts.clock/socket.gettime), kept pure and
engine-free; stdb.connect resolves the clock once and shares it with
both frame slicing and connection.new.

Tests added to the existing test_connection.lua and test_init.lua
(state, latency EMA, ping on both wires, full reconnect walk) - no new
test file. lua dev/test/runner.lua: 11 passed, 0 failed. stylua
--check stdb dev/test: clean.
Breaking: every row callback (on_insert/on_update/on_delete) now takes ctx
as its first argument, matching the official SDKs. Migrated every caller
in the repo: smoke.script, test_tables.lua, test_slicing.lua.

- ctx: one shared object per connection (ctx.db/reducers/procedures/event,
  ctx:is_active()/:disconnect()/:subscription_builder()), threaded through
  tables.lua's existing dispatch rather than a parallel path.
- stdb.builder(): thin fluent sugar over stdb.connect(opts).
- db.db (alias of db.tables) and db:disconnect() (alias of db:close()).
- Table:remove_on_insert/update/delete as aliases of off_insert/etc.
- db.reducers.<name>(args, cb) / db.procedures.<name>(args, cb): sugar
  over call_reducer/call_procedure, validated against the schema; unknown
  name raises listing the real names.
- db:subscription_builder():on_applied/on_error(...):subscribe(...) /
  :subscribe_to_all_tables(), returning a handle where unsubscribe/
  unsubscribe_then raise on an already-ended set (unlike subscribe_query's
  frozen no-op handle).
- version 2.0.0 -> 3.0.0 (game.project).

11/11 unit tests green, stylua clean, conformance PASS on v1 and v2.
- Quickstart: add reducers/procedures to stdb.connect, matching stdb/init.lua's
  own documented example. Without them, calling a reducer over the default v2
  wire raises 'no schema entry for reducer ...; arg types are needed to encode
  BSATN' (found by the example-project worker, reproduced and verified fixed).
- Features table: typed query builder is now yes (stdb/query.lua, compiles
  to SQL client-side like the official builders); new rows for coroutine mode
  (stdb.await/await_event), connection state (db:state()/is_connected()) and
  round-trip latency (db:latency()/db:ping()).
Conflicts were additive on both sides and resolved by keeping both: the
connection gains a clock/state/latency trio and a closed flag, close() sets
both the terminal state and that flag, and the two appended test blocks in
test_connection.lua both stay - with a semicolon between them, because Lua
reads two adjacent (function() end)() blocks as a call on the first's result.
The example was written against 2.x's on_insert(row); 3.0.0 passes the context
first. Rebuilt and re-run headless against a live server: connected, subscribed,
and the row it sent came back through on_insert, which is what its PASS waits
for.

Also un-anchor the build-artifact ignores. They were rooted at the repo top, so
a project anywhere else in the tree - the example builds in place - offered up
its bundle, a 4.9 MB engine binary included.
- Add Upgrading from 2.x section: one before/after callback pair for the
  ctx-first signature change.
- Quickstart: on_insert takes ctx first; demonstrate the db.reducers
  accessor alongside the existing string-name call_reducer form.
- Features table: add ctx, connection builder, reducer/procedure
  accessors, subscription builder, db.db, db:disconnect() rows.
README: send inside the connected path, and install v3.0.0
All checks were successful
ci / ci (pull_request) Successful in 22s
Details
conformance / conformance (v1) (pull_request) Successful in 2m10s
Details
conformance / conformance (v2) (pull_request) Successful in 2m30s
Details
adc5c2acb6 
The quickstart called a reducer immediately after connect, where there is no
socket yet and the send raises. It belongs in on_applied, which is also the
point a reader wants: the cache is populated, so the call has context.
yuris merged commit bb4c5f5f2f into main 2026-08-21 19:51:34 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area:defold
Defold engine integration

  
area:docs
Documentation and examples

  
area:infra
Flake, CI, secretspec

  
area:protocol
Wire protocol conformance

  
area:testing
Test harness and fixtures

  
kind:bug
Something broken

  
kind:chore
Maintenance

  
phase:breadth
UnsubscribeMulti, procedures, BSATN

  
phase:core
Core SDK layers (protocol/cache/connection)

  
phase:hardening
Reconnect, token persistence, errors

  
phase:slice
Vertical slice: first playable end-to-end

No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/spacetimedb-defold!102
No description provided.