9 releases 9 tags

RSS feed

• v2.0.0 11f8817dd7

  Compare

  v2.0.0 Stable

  yuris released this 2026-08-21 11:20:05 +00:00 | 23 commits to main since this release

  v2.0.0 defaults to the v2 BSATN wire. The v1 JSON wire is still fully supported and
  its behaviour is frozen, but it is now something a caller asks for by name. This
  release also closes the four items v1.1.0 listed as known and not addressed: #63,
  #65, #67 and #69.

  Breaking changes

  connect() negotiates v2.bsatn.spacetimedb when you say nothing

  stdb/init.lua resolves opts.protocol or "v2". Pin the old wire with
  connect({ protocol = "v1" }): stdb/protocol.lua has an empty diff against
  v1.1.0, so v1 is frozen rather than merely retained, and conformance is now a
  matrix over both wires on native and HTML5 instead of v1 alone.

  The flip is what a v2 subscription buys: readback is 46% of v1's row-payload
  bytes and client frames are 31%, and 10000 rows apply in 10.5ms against v1's
  20.0ms (native, median of 3). The big-int quoting bucket is structurally absent -
  that scan exists only because JSON cannot carry a u128 exactly.

  Calls that raise on v2

  Only the first is caught at connect; the others raise at the callsite that
  needs the missing piece, so a v1-era callsite finds out rather than drifting.

  • light = true raises. The v2 schema has no light mode, and dropping the
    request silently would hand the full stream to a caller who asked for a
    fraction of it. v1-only.
  • Calling a reducer or procedure needs reducers = gen.reducers and
    procedures = gen.procedures passed to connect, because BSATN carries
    neither argument names nor widths. Without them the raise lands at the call.
    Pre-encoded BSATN string arguments pass through as before.
  • The legacy db:subscribe raises at the subscribe callsite. Use
    subscribe_query - v2's Subscribe is the query-set form.

  Row decoding

  • A table with no generated types no longer decodes. It is skipped and
    reported through an undescribed event, not raised.
  • An unknown type tag raises instead of passing the value through. BSATN carries
    no widths, so a tag the decoder does not know means it can no longer find the
    next column.
  • One-off query rows arrive as encoded byte strings.
  • Event-table rows arrive decoded as an event and are never cached - the one
    behaviour v1 has no counterpart for.
  • Identity renders as the byte-reverse of the wire, because
    crates/lib/src/identity.rs stores it little-endian while to_hex encodes the
    big-endian array. ReducerResult.timestamp is microseconds, not the
    nanoseconds v2.rs:378 documents; both were settled by hex-dumping real
    frames rather than by reading upstream comments.

  Callback payloads

  • msg tables are the v2 messages. A transaction carries no reducer_call
    and no caller_identity.
  • on_out_of_energy never fires. The v2 outcome has no OutOfEnergy variant
    and there is no substitute signal, so exhaustion arrives as ordinary
    InternalError text.
  • Reducer ret_value and Err payloads are raw BSATN, with a printable summary
    handed to the callback. The (err, msg) callback shape is frozen, which is
    why ret_value is not decoded for you.

  Where the two wires genuinely differ elsewhere, v1's behaviour wins:
  SubscriptionError drops no rows, because only the server knows which of them
  another live query set still matches, and a transaction still emits one updates
  entry per query set per table, because the cache's hold counts depend on that.

  confirmed_reads resolves to the server's default, which differs per wire

  connect sends no confirmed parameter unless asked, exactly as the TS, C#,
  Rust and Unreal SDKs do, so a defaulted caller gets the server's own default -
  and upstream resolves that to false on v1 but true on v2
  (subscribe.rs:103-111). A defaulted caller carried from v1 to v2 therefore has
  updates held until the transaction offset is durable: slightly higher latency for
  stronger consistency. Pass confirmed_reads = false for v1-style immediate
  delivery, or true to ask for it outright. The tri-state resolution is now
  pinned by test so the default cannot be "helpfully" written out as
  confirmed=false.

  The value-surface guarantee now excludes f32

  Rows decode to the same Lua values on either wire for every type tag except
  f32. That is the whole of the promise now. The README and four wiki locations
  previously claimed it held for every type, which was untrue.

  A stored 0.1 reads back as 0.10000000149011612 on v2 and
  0.10000000000000001 on v1. v1 sends decimal text and json.decode returns the
  double nearest 0.1; v2 carries the four bytes the server stored, so it widens
  to that single's exact value. v2 is the more faithful side - it hands back what
  the server actually holds, where v1 reconstructs a double from a decimal
  rendering of a single. Only values that do not fit 24 mantissa bits diverge,
  which is why nothing caught it: every f32 fixture in the repo was exactly
  representable, -1.5 and 0 included.

  No codec changed here - the defect was undocumented, untested silence. It is now
  pinned two ways: the type-tag census asserts an f32 argument comes back
  narrowed, and a cross-protocol check decodes one decimal literal through both
  codecs and asserts they differ for 0.1 while agreeing for -1.5. Each side is
  pinned to its own exact double, so the assertion cannot be repaired by editing
  one expected value.

  Features

  frame_budget, spreading a large apply across engine frames

  connect({ frame_budget = 0.008 }) spends at most that long per engine frame
  decoding rows and resumes on the next, so a large first subscription costs
  several ordinary frames instead of one hitch. It is opt-in and nil by default:
  slicing is visible in timing - callbacks that fired inside the receive frame now
  land one or more frames later - and the 1.0 surface is frozen, so a
  timing-visible change cannot be the default. Unset, the code path is what it was:
  the EVENT_MESSAGE branch calls conn:handle_frame inline and
  cache.slice_hook stays nil.

  stdb/cache.lua gains one module-level hook, cache.slice_hook, called every
  256 decoded rows inside decode_rows' two loops and nowhere else. That is the
  only place in an apply where nothing has been written yet - the cache decodes a
  whole batch before touching the store, so a malformed row cannot half-apply -
  which is why a yield there cannot leave a half-populated table. Store mutation
  and event emission stay atomic, so "callbacks only ever see a complete batch"
  falls out rather than needing bookkeeping. stdb/init.lua holds one FIFO queue
  of raw frame strings and at most one live coroutine, pumped by
  timer.delay(0, true, ...). One coroutine at a time is what keeps ordering
  intact: a TransactionUpdate arriving mid-InitialSubscription is appended
  behind it and applied strictly after. The deadline is per pump invocation and
  shared across every frame drained in it, so N queued frames cannot spend N
  budgets in one engine frame. A dropped socket clears queue and coroutine - the
  cache is cleared and every subscription replayed on reconnect anyway, and
  abandoning is safe precisely because nothing was written. close() cancels the
  pump and clears the hook. A frame_budget of 0, negative, non-numeric, or with
  no clock available raises at connect.

  What it does not bound: two parts of an apply are O(rows) and run whole. On the
  v1 JSON wire, where this was measured, those are decode_server's single
  json.decode of the entire frame text - 12.8-23.3ms at 25000 rows, which makes
  it the worst frame in the apply - and the deliberately atomic store mutation,
  7.3-13.7ms. At an 8ms budget, native, 25000 rows: the row-decoding frames sit at
  8.5ms against an unsliced 55.5ms, and the whole apply spans 53.2ms against
  55.5ms, so this is redistribution rather than overhead. The honest headline is
  that the worst frame drops from ~55ms to ~20ms, not that it drops to the budget.
  On wasm the mechanism works and the arithmetic is the tier's: a browser frame has
  a ~17ms rAF floor, so an 8ms budget is already below the frame quantum - at 10000
  rows the decode frames land at 16-17ms and the first frame at ~75ms. On v2 the
  message decode leaves rows in a BsatnRowList still encoded, so the cache
  decodes them and slicing covers them unchanged.

  The reconnect spread seeds from the persisted identity

  v1.1.0 added connection.backoff_spread(identity, attempt) - a deterministic
  multiplier in [0.5, 1.0) keyed on the identity, so a server restart does not
  bring the whole player base back in one instant - and then fed it
  conn.identity, which is nil until a socket has succeeded. The one case the
  spread exists for, a fleet relaunching into an outage, therefore got no spread at
  all and retried on the flat 1/2/4/8/16/30 ladder in lockstep.

  connect now falls back to persisted.identity, which connection.lua's
  IdentityToken handler was already writing and nothing ever read back:
  backoff_spread(conn.identity or persisted_identity, attempt).

  That saved identity is not server-confirmed at this point - nothing has connected
  yet - and using it anyway is deliberate. The spread asks for a value that differs
  between clients, not for a correct one, so the failure mode of a wrong identity
  is a wrong-but-deterministic slot, which is strictly better than every client in
  the fleet sharing one ladder. The or ordering hands over to the wire identity
  the moment IdentityToken lands, and nothing else in the SDK reads the persisted
  value. A genuinely fresh install, or storage = false, has nothing persisted and
  keeps the exact ladder: one new client is not a herd.

  Bug fixes

  • dev-server.sh overwrote another port's pidfile - point two ports at one
    explicit STDB_STATE_DIR and the second start_server correctly failed
    SpacetimeDB 2.8.2's exclusive database lock, but still wrote $PIDFILE with
    its own doomed child's pid. stop on the first port then read a dead pid,
    reported no server running, and left a live server that nothing could find
    again. start_server now reads /proc/$pid/cmdline for the pid already in the
    pidfile and aborts if it belongs to a live server on a different port, ahead
    of both keypair generation and the spawn, so a refused invocation writes
    nothing at all into the contested state dir - and fails in 2.08s instead of
    waiting out a 31.59s ping timeout. Same trailing-space match as
    server_pid()'s existing check, so port 692 does not match a server on
    6921, and the same degradation: without /proc the guard is inert. A merely
    stale pidfile still proceeds and hits the database lock, which is legible and
    unchanged.
  • The big-int rescue scan ran unguarded - protocol.quote_big_ints ran its
    pattern gsub at both callsites with no precondition: once per frame in
    decode_server, once per row in decode_row. On a table with no 128-bit
    column every one of those scans rewrote nothing and still cost about what the
    parse did - 40% of the applying frame. It now has the same shape as its sibling
    quote_wide_ints: a plain find for the byte run "__, which every key the
    pass can rescue contains, so absent that run the pattern cannot match. A "__
    inside a row's double-encoded string is a false positive that runs the scan for
    nothing, which is exactly the old behaviour; there is no false negative, the
    only direction that could corrupt a u128. Native median at 10000 rows with a
    hand-written schema: frame 29.3ms to 20.2ms, quoting share 40% to 0%. This
    applies to the v1 JSON wire only, and it collected in advance the part a BSATN
    decoder would have deleted - which is why the documented codec-swap bound fell
    from 6-8x to ~3x native.

  What's Changed

  • stdb: default to the v2 BSATN wire, and ship it as 2.0.0 in #94
  • test: bring the v2 smoke to parity with v1's in #92
  • fix: kill chromium's process group before removing its profile in #93
  • ci: run conformance on both wires from the automatic triggers in #91
  • test: pin the confirmed_reads resolution to peer-SDK parity in #90
  • Narrow the value-surface guarantee to exclude f32 in #88
  • test: a census of the type-tag grammar in #87
  • readme: two wire formats, one value surface in #86
  • conformance: run the smoke over v2, and capture the wire in #85
  • stdb: connect({ protocol = "v2" }), opt-in in #84
  • protocol_v2: the v2 message schema over BSATN in #83
  • bsatn: a pure-Lua BSATN primitive codec in #82
  • ci: retarget the HTML5 self-report and close out #16 in #80
  • ci: run conformance on changes that affect it, not on a weekly clock in #79
  • ci: report the HTML5 conformance verdict to #16 in #78
  • docs: move documentation into the wiki, shrink README to a repo description in #75
  • feat: spread a large apply across engine frames behind frame_budget in #76
  • fix: refuse to start over another port's server in a shared state dir in #73
  • feat: seed the reconnect spread from the persisted identity in #72
  • perf: guard the big-int rescue scan on a necessary condition in #71

  Full Changelog: v1.1.0...v2.0.0
  
  Writing release-note head

  Use as a Defold library

  Add to game.project dependencies:

  https://git.nokopia.com/yuris/spacetimedb-defold/archive/v2.0.0.zip
  https://github.com/defold/extension-websocket/archive/refs/tags/4.2.3.zip
  

  extension-websocket is required: Defold does not resolve dependencies
  transitively, so without it connect fails at runtime.

  Targeting HTML5? 4.2.3 drops the subprotocol there, so it needs the
  patch in patches/ AND the remote dependency above removed - keeping
  both puts two websocket extensions in one build and the link fails.
  The README has the four commands.

  Private repo? Append ?token=<access-token> to the URL, the same
  shape Defold documents for GitLab. Basic auth in the URL
  (https://user:token@...) is not a substitute: the archive endpoint
  ignores URL credentials - a real account with a wrong password still
  gets 200 on a public repo - while ?token= is validated and a bad
  one gets a plain 401.

  Downloads

  • Source code (ZIP)
    3 downloads
  • Source code (TAR.GZ)
    1 download