9 releases 9 tags

RSS feed

• v1.0.0 a2e39ff246

  Compare

  v1.0.0 Stable

  yuris released this 2026-08-20 11:43:15 +00:00 | 50 commits to main since this release

  The 1.0 API freeze: every public name in stdb/ - six pure-Lua modules speaking v1.json.spacetimedb over Defold's own websocket extension, no native extension and no build step - is now permanent. Twenty pull requests since v0.1.0, verified against a live SpacetimeDB 2.8.2 inside a real Defold engine on both the native and HTML5 targets through five conformance phases, with dev/test/fixtures/session.jsonl holding 50 captured frames across four sockets that the suite replays and a generated schema re-derived from the live server on every run and diffed. The release-gate checklist, including what was deliberately deferred, is issue #24.

  Breaking changes

  The tag makes every name permanent, so #55 spent eight renames at the last cheap moment. Six are pure spelling, two change what a call returns. Below that, one value representation changed in #37.

  Event type = "error" → "socket_error"

  The websocket transport's error event. It was the only emitted event type absent from the README's event table - 16 rows for 17 types - and "error" already carried three other meanings beside it: evt.error, sub.state == "error" and call.state == "error". The table is 17/17 now, including the three sites connection.lua builds dynamically (:764, :899, :934) that a literal grep cannot see.

  db.state → db.cache

  The connection's row cache. sub.state and call.state are lifecycle strings, on objects a caller holds at the same moment, so one word named two concepts: if sub.state == "applied" then stdb.cache.get(db.state, ...) end. 55 field lines moved, and cache.lua's own first parameter with them, since leaving it would have kept the word in play.

  undescribed event: rows → dropped

  It is a count, not a payload, and rows sat one character from row on an insert event, which is a row. Renamed in all three places at once: the event, counts.undescribed[i] and sub.undescribed[i].

  schedule_at_to_seconds returns "Interval" / "Time"

  A behaviour change, not a spelling: v0.1.0 returned "interval" / "time". decode_value always produced the capitalised form - those are the server's own declared variant names for the SDK-builtin ScheduleAt - so a consumer who compared against the documented lowercase got false forever with nothing raised. Compare against the tag, kind == v.tag, rather than a literal.

  The helper was fixed rather than the decoder, because 2.8.2 lowercases module enums but not this one: dev/server/index.js declares t.enum("Status", ["Idle", "Busy"]) and the published schema says idle/busy. Lowercasing the decoder would have broken the larger cohort - every cached-row .tag reader, plus reducer parameters and procedure returns.

  stdb.url(host, database) → stdb.url(base_url, database)

  Positional, so no call site changes. The name was wrong and the README believed it, claiming ws:// was prepended: nothing prepends a scheme, so stdb.url("127.0.0.1:3000", "chat") returns a scheme-less string and the connect fails. Pass a scheme-qualified origin.

  protocol.decode_typed(encoded, spec, ...) → (encoded, def, ...)

  Positional, so no call site changes. A schema entry had three names on the frozen surface - def, defs, spec - and spec was the outlier, published in the README twelve lines from t.def. Two user-facing error messages now say "schema def".

  protocol.duration_to_seconds → protocol.time_duration_to_seconds

  The type is TimeDuration, the tag string is "time_duration", the wrapper key is __time_duration_micros__, and the five sibling helpers are named after their types exactly. This was the one abbreviation.

  Event type = "subscription_error" → "subscription_failed"

  One _error against three _failed among the failure events, beside sub.error and sub.state == "error". The wire tag is untouched. evt.type == "subscription_failed" now diverges from sub.state == "error" for the same occurrence, deliberately, matching how reducer_failed already diverges from call.state == "error".

  db.connection_id is canonical hex

  Separate from the renames, in #37. The server sends ConnectionId as a bare 128-bit JSON number, so json.decode rounded it and only 14 of 32 hex digits survived - padding the rest produced a string of exactly the canonical width with a fabricated tail, and two ids differing by 1 on the wire rendered identically. db.connection_id is now 0x plus 32 exact digits, converted from the raw decimal text, so it reads like db.identity. It is still a fresh ConnectionId per socket, so it changes across a reconnect while db.identity does not.

  Features

  Typed table accessors

  Row changes used to arrive as one firehose: every consumer wrote the same loop over evt.counts.events, filtering by table name and branching on event type. #45 gives every table in the schema an accessor under db.tables:

  db.tables.message:on_insert(function(row) end)
  db.tables.message:on_update(function(old, new) end)   -- primary-key tables only
  db.tables.message:on_delete(function(row) end)
  
  db.tables.message:count()
  db.tables.message:iter()
  db.tables.message:find("id", 7)
  db.tables["my-table"]                                 -- a name that is not a Lua identifier
  

  stdb/tables.lua is a new module over the schema and the cache, both passed in, so it is unit-testable without an engine; connection.lua gains one require, one install call and a four-line emit_rows. on_event is untouched and still fires first, per-table callbacks run after it, and callbacks fire per row only once the whole batch is in the cache, so a callback reading count/iter/find sees a complete transaction and never half of one. Deregistration is off_insert(fn) / off_update(fn) / off_delete(fn), spelled the way Subscription:off_applied already is, because a game object registers in init(self) and the closure holds self - if final(self) cannot undo that, the first row after the object is deleted calls into a corpse.

  The namespace is load-bearing rather than decorative: 2.8.2 accepts a table called state, close or cache, publishes it and serves its rows, so a flat db.<name> would collide with the connection's own fields, and any reserved list of those names would itself be frozen API - adding a field to connection.lua would then break anyone whose module has a table by that name. on_update raises on a pk-less table, reading the same def.pk that cache.lua pairs updates off, because such a table's row change arrives as a delete plus an insert that the cache deliberately does not pair, so the registration could never fire. find delegates entirely to cache.find.

  Handles from subscriptions and reducers

  subscribe_query returned a bare query_id and call_reducer a bare request_id, each with one (err, ...) callback. #38 returns a handle from both, and both old forms keep working - unsubscribe accepts a handle or a raw id:

  local sub = db:subscribe_query({ "SELECT * FROM message" })
  sub:on_applied(fn)  sub:on_error(fn)  sub:is_active()  sub:is_ended()  sub:unsubscribe(cb)  sub.query_id
  
  local call = db:call_reducer("send_message", { "yuris", "hello" })
  call:on_ok(fn)  call:on_error(fn)  call:on_out_of_energy(fn)  call.request_id
  

  What each buys: a subscription that errored was silently dropped, discoverable only by having caught the callback at the time; and OutOfEnergy is the account's problem while a failed reducer is the module's, which handlers.TransactionUpdate already separated internally and then flattened into one error string. A late attach fires immediately from the retained result, and on_applied is a list rather than a one-shot, so it fires again on every reconnect replay. A reconnect walks an applied handle back to sent, since resubscribe_all drops the cache, and the flag is set before c.send so a transport that raises mid-replay cannot leave a handle claiming to be active. A query set ends when the drop is issued, not when UnsubscribeMultiApplied lands, because it is already out of query_sets and therefore unreplayable.

  Boundaries: there are no timeouts, so a pending call lives until the connection drops. The legacy subscribe still cannot be undone and still replaces the whole set - calling it twice while one is live inflates the row hold counts, which #51 made loud by emitting a subscription_replaced event rather than doing it silently. Use subscribe_query for anything you may change.

  Unique-column lookups maintained from the schema

  cache.get looked up by primary key only; any other unique column needed a full cache.iter scan. Now that the generator emits unique, #44 maintains a lookup per unique column:

  cache.find(state, "player", "identity", my_identity)
  

  state.indexes[table][column][value] = row is built once from the schema at cache.new, rebuilt by drop_all, and maintained inside the private store / remove, which are only ever reached from apply_database_update - no separate pass. The pk column is deliberately not mirrored, because state.tables[name] already is that index, and find on the pk delegates to get instead, so a caller routes every single-column unique lookup through one function with no pk special case. find on a non-unique column raises rather than returning nil or scanning: a nil would be indistinguishable from a genuine miss, and a scan would be a silent O(n) on the main thread, so hit, miss and unusable stay three distinguishable outcomes. schema.new now rejects a unique name that is not a column, for the same reason it already rejected such a pk.

  The motivating shape is "which row is me?" on a player table keyed by auto-increment id with a unique identity column - a lookup that runs on every connect and every reconnect. It only pays off because #48 also made an Identity column decode to a string instead of a fresh table per row.

  Full AlgebraicType decoding

  The SDK handed row values through untouched, which is correct for scalars and strings and wrong or unusable for everything else. #46 adds protocol.decode_typed(encoded, def, typespace) and protocol.decode_value(v, tag, typespace), written against a new 31-column all_types table captured in both encodings - subscription readback and TransactionUpdate - into session.jsonl. Since #48 the cache routes a table's rows through decode_typed whenever its schema entry carries types, and through decode_row when it does not, so a hand-written schema is unchanged byte for byte.

  Four wire facts the repo had wrong or did not know, captured rather than reasoned: vec_u8 is hex text ("0001feff", and "" when empty), not a JSON array; i256/u256 are hex text of minimal width with a -0x sign prefix, so zero is "0x0"; an Option column is an ordinal-keyed sum [0,v] / [1,{}], never the {"some":v} form protocol.lua's header claimed; and ScheduleAt's variant names are capitalised because they come from the library, while module enum variants are lowercased by the server.

  i128/u128 decode to exact decimal digit strings, always strings so the type never depends on magnitude, with exactness coming from quoting 16-digit-and-longer integer literals in the raw row text before json.decode ever runs. i256/u256 keep the server's hex verbatim. i64/u64 are unchanged - still a Lua double, still rounding past 2^53 - because retyping them would retype message.id, a u64 pk the cache keys on. A 128-bit value that never met that rescue, such as a procedure return through decode_payload, decodes to nil past 2^53 rather than printing exact-looking digits for bits that are already gone.

  Boundary: Lua 5.1 makes both misuses of a 128-bit string silent. row.serial + 1 coerces to a rounded double and raises nothing, and < is lexicographic, so table.sort on a 128-bit column orders 10000000000000000 before 9007199254740993. Compare with ==, which is exact because the representation is canonical, and leave 128-bit arithmetic and ordering to the server.

  A primary-key row change is one update event

  A one-column change on a primary-key table reached the consumer as delete + insert. The cache state was always right, since deletes apply before inserts and a pk change lands as an upsert, but the event stream reported the row leaving and a different row arriving - in Defold that maps to despawn plus respawn: lost tweens, and a go.delete/factory.create pair every tick of movement instead of a go.set_position. #34 makes apply_database_update decode each batch up front, pair deletes with inserts by primary key within one updates entry, suppress the paired delete and emit:

  { type = "update", table = name, row = new_row, old = old_row }
  

  The premise was observed, not inferred: an edit_message reducer driven over a real v1.json.spacetimedb socket packs both halves of the change into one entry (tables.length == 1, updates.length == 1), captured twice independently on separate servers.

  Pairing is by key, never by field similarity, and scoped to one updates entry, because only there is the insert guaranteed to be applied last. A changed pk value therefore declines to pair and still reports delete + insert, pinned by a test whose two rows are identical apart from the key; tables with no pk behave exactly as before. The batch being decoded before any mutation also means a malformed row now leaves the store untouched, where the old code applied the delete and then raised.

  Light mode and confirmed reads on connect

  #33 exposes two connect options the protocol supports and this SDK could not ask for: light (default false) becomes ?light=true, and confirmed_reads (default nil) becomes ?confirmed=true|false. Both reach the URL on native, where auth is a header, and on HTML5, where the token travels in the URL. confirmed_reads is tri-state on purpose - absent, true and false are three distinct requests - because V1 resolves absent and confirmed=false identically while later protocol versions default absent to true, so a boolean option defaulting to false would have frozen V1's meaning into the API. It must be a boolean or nil: the earlier opts.confirmed_reads and "true" or "false" collapsed every truthy value, so a string "false" arriving from a config file or json.decode requested confirmed=true, an exact inversion.

  What light mode does was then measured against a live server rather than read from source. light = true is accepted and validated by 2.8.2 - light=notabool makes it refuse every connection - and is a no-op there: what decides whether you receive a full TransactionUpdate or a TransactionUpdateLight is whether you made the call, not whether you asked for light mode. Your own transaction always arrives full, carrying reducer_call, which matters concretely because the SDK correlates reducer callbacks on reducer_call.request_id and handlers.TransactionUpdateLight never touches c.pending, so a caller's own update arriving light would mean every call_reducer callback silently never fires.

  A TransactionUpdateLight does carry a top-level request_id, and it is the originating client's: a probe sent 4242 from the other socket while the light socket's own last id was 7, and 4242 is what the light frame carried. Correlating on it matches another player's call.

  Bug fixes

  • Overlapping subscriptions no longer lose rows - with two live subscriptions matching a row by distinct queries, a change that took the row out of one of them evicted it from the cache entirely, silently, while the server still held it and the other query still matched. #51 counts how many queries hold each primary key (state.refs), removes a row only at the last hold, and suppresses the delete event exactly when the row survives. The v1 JSON wire carries no per-entry query identifier - QueryUpdate is { deletes, inserts } and TableUpdate.updates is a bare list - so the fix is a hold count rather than an attribution, and both wire orders (bare delete first, pair first) are asserted because nothing on the wire promises one. Cost: +9.4 ms on a 20k-row InitialSubscription that already takes about 114 ms in one frame on Defold's main thread, +1.9% on the steady-state transaction path, and one integer per primary key, about 26 B/key at 20k rows.
  • Identity-keyed tables work - decode_row never converted special-type columns, so an Identity column decoded to a fresh wrapper table per row. A table key only ever matches itself, so the canonical SpacetimeDB game pattern was broken three ways at once: cache.get/cache.find could never resolve a row, store/remove treated every arrival as a new key so the row set grew without bound, and #34's pairing never fired. #48 routes rows through the typed decoder, so an Identity column becomes the server's 0x hex string and is usable as a primary key. It is never cheaper than the untyped path: ×1.22 per row for message (u64, string, string) and ×2.1 for profile (identity, timestamp, option, vec, sum, nested product), which is the realistic worst case to budget against.
  • Procedure and one-off-query callbacks no longer vanish when the socket dies - they carry no handle, so their c.pending entries were dropped when the socket was replaced and the callback was never invoked; a consumer who disabled a button until a procedure returned left it disabled forever, with nothing reported anywhere. #47 adds M.settle_pending(c, replay) and calls it on all three paths a socket can go away on - replaced by set_send, retired by conn.close, and the terminal branch of schedule_reconnect - with subscribe exempt only under a replay that will re-issue it. Each entry settles under pcall and the first error is re-raised once the sweep completes, so one raising callback no longer costs every later caller its answer: 4 of 4 reached, against 1 of 4 before.
  • Two concurrent one-off queries no longer deliver each other's rows - one_off_query keyed c.pending on c.last_one_off, "the query sent last", so with two in flight the first response went to the second caller's callback, the second response reached nobody, and one entry leaked for the life of the connection. #47 keys on the echoed message_id, verified on the wire as lowercase hex::encode of exactly the bytes sent. An OneOffQueryResponse matching no request is now emitted with matched = false instead of consuming a whole frame silently.
  • A relation the schema cannot describe is skipped, not stored as empty rows - table_def returned {}, so decode_row mapped zero columns and an empty row was stored and counted as an insert, with no error, reachable by an ordinary SELECT * FROM my_view. Those fieldless rows were readable too: count() reported them, iter() handed out a table where every access is nil, and because a pk-less delete scans for an equal row and pairs({}) never falsifies a match, a delete could evict the one row the relation did hold. #49 skips the relation whole and reports it in counts.undescribed, as an undescribed event and on sub.undescribed. It is not a raise: a raise aborts the frame's tables loop, so it would drop the rows of relations the consumer did describe, and which ones depends on the server's table order. gen-schema.sh now also emits views, in the view's own column order, with view = true.
  • The HTML5 install instructions did not work - README.md:95 was the only consumer-facing instruction for the only HTML5 path, and unzip -q .internal/lib/*websocket*.zip -d /tmp/ws exits 9 with No zipfiles found, because Defold hash-names its dependency-cache entries and websocket appears nowhere in them. #53 downloads extension-websocket 4.2.3 from upstream and patches it in place, verified by extracting the instructions from the shipped README and running them as a script. The patch's own trailer told you to maintain a fork, contradicting the README and the issue that exists to avoid one; it now describes the vendoring flow, and the patch has a direct raw URL, since [library] include_dirs = stdb means patches/ never appears in a consumer's project tree. game.project is a library manifest at version = 1.0.0 rather than this repo's harness project - it pointed main_collection and game_binding at /dev paths that export-ignore strips from the archive - with the harness keys moved to dev/example/harness.project and layered by bob --settings. .gitignore no longer ships to consumers.

  What's Changed

  • release: -fsS and --fail-with-body are mutually exclusive in #58
  • release: stop shipping the basic-auth claim #56 removed as false in #57
  • docs: correct every behavioural claim measurement contradicted in #56
  • api: eight renames the 1.0 freeze makes permanent in #55
  • archive: fix the broken HTML5 install path and ship a consumer-only game.project in #53
  • cache: count how many queries hold a row, so an overlapping one survives in #51
  • dev-server: gate restart on the database serving, not the port answering in #50
  • cache: skip a relation the schema cannot describe, and emit views in #49
  • cache: route rows through the typed decoder so an Identity key works in #48
  • connection: settle handle-free requests when the socket dies in #47
  • protocol: decode every AlgebraicType from the schema tags in #46
  • tables: a typed accessor per table in #45
  • cache: unique-column lookups maintained from the schema in #44
  • protocol: convert special types explicitly in #37
  • docs: state the real reason column order is load-bearing in #40
  • cache: report a pk row change as one update event in #34
  • subscriptions, reducers: return handles instead of bare ids in #38
  • codegen: keep the module's type information in #36
  • connect: expose light mode and confirmed reads in #33
  • html5: verify identity survives a reload in #17
  • ci: run HTML5 conformance as a step in the conformance job in bfd2b14
  • repo: separate the library from its development infrastructure in 201dbc8
  • chore: untrack secretspec.toml in 5242674
  • ci: drop the bespoke reporting now that the job is trustworthy in 2581e8f
  • ci: stop linking a log the run did not upload in 640ce03
  • conformance: resolve dependencies before building in a9ba07e
  • ci: make the conformance self-report actually readable in 915961d
  • ci: check out with git so the nix image needs no node in a2cdbb2
  • docs: HTML5 traps, an API reference, and the docker-free workflow in fae6e56
  • ci: route conformance to the nix runner and publish its logs in 522d400
  • html5: authenticate with a query token, and prove it in a browser in 0e8745e
  • server: replace docker with pinned SpacetimeDB release binaries in 050e71e

  Full Changelog: v0.1.0...v1.0.0

  Use as a Defold library

  Add to game.project dependencies:

  https://git.nokopia.com/yuris/spacetimedb-defold/archive/v1.0.0.zip
  https://github.com/defold/extension-websocket/archive/refs/tags/4.2.3.zip
  

  extension-websocket is required: Defold does not resolve dependencies
  transitively, so without it connect fails at runtime.

  Targeting HTML5? 4.2.3 drops the subprotocol there, so it needs the
  patch in patches/ AND the remote dependency above removed - keeping
  both puts two websocket extensions in one build and the link fails.
  The README has the four commands.

  Private repo? Append ?token=<access-token> to the URL, the same
  shape Defold documents for GitLab. Basic auth in the URL
  (https://user:token@...) is not a substitute: the archive endpoint
  ignores URL credentials - a real account with a wrong password still
  gets 200 on a public repo - while ?token= is validated and a bad
  one gets a plain 401.

  Downloads

  • Source code (ZIP)
    5 downloads
  • Source code (TAR.GZ)
    1 download