feat(connection): spread the reconnect backoff per client, deterministically (#19) #62

Merged

yuris merged 2 commits from feat/19-reconnect-jitter into main 2026-08-20 14:12:52 +00:00

Conversation 1 Commits 2 Files changed 4 +62 -9

yuris commented 2026-08-20 13:52:23 +00:00

Owner

Copy link

Closes #19. Phase 2 of the v1.1.0 milestone (#59). Additive except for
backoff_delay, which gains a trailing optional fourth parameter, so no
existing callsite or test breaks.

The decision: deterministic, not random

The issue proposes injecting math.random. This deliberately does not, for two
reasons that are now in the doc comment:

• A library must never call math.randomseed — it clobbers the consuming
  game's own RNG sequence, breaking any deterministic simulation the game relies
  on.
• Without a seed, every client picks the same "random" spread, which
  reproduces the exact stampede being fixed. The issue names this as "the part
  that actually bites".

So the spread is derived from the identity instead. Every client already holds a
unique 64-hex-char identity by the time it can be in a reconnect herd, so a
per-client slot needs no RNG, no global state, and no seeding — and exact-value
tests remain possible.

function M.spread_for(identity, attempt)   -- -> [0.5, 1.0), or nil
function M.backoff_delay(attempt, base, cap, spread)

spread_for returns nil before an identity exists: the first connect is a
cold start, not a herd. A nil spread is an exact no-op in backoff_delay.

The spread applies after the cap, so a capped delay is spread too —
otherwise every client in a long outage converges on exactly 30s, which is the
herd again at the worst possible moment.

Verified, not assumed

Lua 5.1 exactness. Largest intermediate is 4294967296 * 31 + 255 =
133143986431 (~1.33e11) against 2^53 = 9.0072e15. A 10k-iteration loop
driving the fold from the worst case h = 4294967295 kept every intermediate
integral (h == math.floor(h) throughout).

The spread re-spreads per attempt — checked before asserting it. For one
identity, attempts 1..10 give 0.7405, 0.537, 0.6855, 0.834, 0.6305, 0.779,
0.9275, 0.724, 0.8725, 0.521. A herd redistributes each round rather than
marching in lockstep at a fixed offset.

No nil-attempt guard was added, because none is reachable: attempt is a
local initialised to 0, incremented immediately before the call, and reset on
connect, so it is always a number ≥ 1 at the callsite. A guard would be dead
code.

One callsite, confirmed by grep: stdb/init.lua:238. Everything else
matching backoff_delay is the definition, the tests, or prose.

The seven existing assertions are untouched

The test_connection.lua diff is @@ -244,6 +244,28 @@ with zero deletion
lines — purely additive. That is the whole reason spread is a trailing
optional parameter.

Checks

lua dev/test/runner.lua            6 passed, 0 failed
stylua --check stdb dev/test       exit 0, silent
conformance.sh                     PASS, both loads, identity persisted

Conformance is the meaningful one here, since it actually reconnects: it logged
reconnecting attempt 1 in 0.744s where before this change it was a flat 1s —
the spread is live on the wire, not just in unit tests.

End-to-end check, two identities at the same attempt:

A       2.742
B       2.518
ok

Both distinct and in [1.0, 2.0) for attempt 3 with base 1. Also asserted
directly: backoff_delay(99, 1, 30, 0.5) == 15 (a capped delay is spread),
backoff_delay(6, 1, 30) == 30 (nil is a no-op), spread_for(nil, 1) == nil.

README

Three touch points, one of them not in the original plan:

• the reconnect prose now states the deterministic per-client spread of 50–100%
  of the slot, keyed on identity, with none on first connect;
• the options table row read Seconds; doubling, no jitter — no jitter was
  now false and is corrected;
• the stdb.connection API table gains a spread_for row beside the comparable
  pure helper backoff_delay, and backoff_delay's own row gains the fourth
  parameter. That section's prose said "exactly the six functions below" and "the
  worked example of all six"; both numerals became wrong on adding a row and are
  corrected to seven. Not in the assignment, but required for the claim to stay
  true.

The ponytail: no jitter deferral comment above backoff_delay is deleted —
this work discharges it.

Process note

The first round of edits landed in the main checkout on main instead of the
worktree, because the edit tool resolves a section path against cwd. Caught by a
sibling agent before any commit, recovered with git stash push in the main
checkout and git stash pop in the worktree, no conflict and nothing lost. Every
number reported above was re-captured from inside the worktree afterwards, since
the pre-recovery runs had been exercising an unmodified tree and proved nothing.
git status --porcelain is empty in the main checkout.

Closes #19. Phase 2 of the v1.1.0 milestone (#59). Additive except for `backoff_delay`, which gains a **trailing optional** fourth parameter, so no existing callsite or test breaks. ## The decision: deterministic, not random The issue proposes injecting `math.random`. This deliberately does not, for two reasons that are now in the doc comment: - **A library must never call `math.randomseed`** — it clobbers the consuming game's own RNG sequence, breaking any deterministic simulation the game relies on. - **Without a seed, every client picks the *same* "random" spread**, which reproduces the exact stampede being fixed. The issue names this as "the part that actually bites". So the spread is derived from the identity instead. Every client already holds a unique 64-hex-char identity by the time it can be in a reconnect herd, so a per-client slot needs no RNG, no global state, and no seeding — and exact-value tests remain possible. ```lua function M.spread_for(identity, attempt) -- -> [0.5, 1.0), or nil function M.backoff_delay(attempt, base, cap, spread) ``` `spread_for` returns `nil` before an identity exists: the first connect is a cold start, not a herd. A `nil` spread is an exact no-op in `backoff_delay`. The spread applies **after** the cap, so a capped delay is spread too — otherwise every client in a long outage converges on exactly 30s, which is the herd again at the worst possible moment. ## Verified, not assumed **Lua 5.1 exactness.** Largest intermediate is `4294967296 * 31 + 255` = `133143986431` (~1.33e11) against `2^53` = 9.0072e15. A 10k-iteration loop driving the fold from the worst case `h = 4294967295` kept every intermediate integral (`h == math.floor(h)` throughout). **The spread re-spreads per attempt** — checked before asserting it. For one identity, attempts 1..10 give 0.7405, 0.537, 0.6855, 0.834, 0.6305, 0.779, 0.9275, 0.724, 0.8725, 0.521. A herd redistributes each round rather than marching in lockstep at a fixed offset. **No nil-attempt guard was added**, because none is reachable: `attempt` is a local initialised to 0, incremented immediately before the call, and reset on connect, so it is always a number ≥ 1 at the callsite. A guard would be dead code. **One callsite**, confirmed by grep: `stdb/init.lua:238`. Everything else matching `backoff_delay` is the definition, the tests, or prose. ## The seven existing assertions are untouched The `test_connection.lua` diff is `@@ -244,6 +244,28 @@` with **zero** deletion lines — purely additive. That is the whole reason `spread` is a trailing optional parameter. ## Checks ``` lua dev/test/runner.lua 6 passed, 0 failed stylua --check stdb dev/test exit 0, silent conformance.sh PASS, both loads, identity persisted ``` Conformance is the meaningful one here, since it actually reconnects: it logged `reconnecting attempt 1 in 0.744s` where before this change it was a flat `1s` — the spread is live on the wire, not just in unit tests. End-to-end check, two identities at the same attempt: ``` A 2.742 B 2.518 ok ``` Both distinct and in `[1.0, 2.0)` for attempt 3 with base 1. Also asserted directly: `backoff_delay(99, 1, 30, 0.5) == 15` (a capped delay is spread), `backoff_delay(6, 1, 30) == 30` (nil is a no-op), `spread_for(nil, 1) == nil`. ## README Three touch points, one of them not in the original plan: - the reconnect prose now states the deterministic per-client spread of 50–100% of the slot, keyed on identity, with none on first connect; - the options table row read `Seconds; doubling, no jitter` — `no jitter` was now false and is corrected; - the `stdb.connection` API table gains a `spread_for` row beside the comparable pure helper `backoff_delay`, and `backoff_delay`'s own row gains the fourth parameter. That section's prose said "exactly the six functions below" and "the worked example of all six"; both numerals became wrong on adding a row and are corrected to seven. Not in the assignment, but required for the claim to stay true. The `ponytail: no jitter` deferral comment above `backoff_delay` is deleted — this work discharges it. ## Process note The first round of edits landed in the main checkout on `main` instead of the worktree, because the edit tool resolves a section path against cwd. Caught by a sibling agent before any commit, recovered with `git stash push` in the main checkout and `git stash pop` in the worktree, no conflict and nothing lost. Every number reported above was re-captured from inside the worktree afterwards, since the pre-recovery runs had been exercising an unmodified tree and proved nothing. `git status --porcelain` is empty in the main checkout.

yuris added 1 commit 2026-08-20 13:52:23 +00:00

feat(connection): spread the reconnect backoff per client, deterministically (#19) ... 5cd24be65f

A server restart brought every client back on the same schedule: attempt N
resolved to the same delay for everyone, so the herd hit the socket in one
instant and the backoff bought nothing.

The issue proposed math.random. Deliberately not, for two reasons:

- A library must never call math.randomseed. It clobbers the consuming game's
  own RNG sequence, breaking any deterministic simulation the game relies on.
  That is a bug we would be handing to every consumer to fix the one we have.
- Without an explicit seed, Lua's math.random yields the SAME sequence in every
  process, so every client would pick the SAME "random" spread - reproducing
  the exact stampede this is meant to prevent. The randomness would be
  decorative.

So the spread is derived from the identity instead. Every client already holds
a unique 64-hex-char identity by the time it can be in a reconnect herd, which
makes a hash of it a better spread source than an unseeded RNG: distinct per
client by construction, no global state, no seeding, and exact-value tests stay
possible. `spread_for` folds the identity with the attempt number, so a herd
re-spreads each round rather than marching in lockstep at a fixed offset.
Arithmetic stays exact in Lua 5.1 - the largest intermediate is
4294967296 * 31 + 255 ~= 1.33e11, far below 2^53.

`backoff_delay` takes `spread` as a TRAILING OPTIONAL parameter, applied after
the cap so a capped delay is spread too. nil is an exact no-op, which is what
keeps the frozen 1.0 signature and the existing exact-value assertions valid.
The first connect has no identity and therefore no spread: a cold start is not
a herd.

Discharges the `ponytail: no jitter` deferral above backoff_delay.

yuris added 1 commit 2026-08-20 14:10:13 +00:00

review: correct two README claims measurement contradicts ... a70f96c266

Review measured the distribution instead of trusting the two-sample example in
the PR body, and both of my stronger claims turned out false.

"Every client lands on a different slot" is wrong above a few dozen clients.
The value is 0.5 + (h % 1000)/2000, so there are exactly 1000 possible outputs
- confirmed by sampling 2,000,000 identities and seeing exactly 1000 distinct
values. At 1000 clients, 365 collide (36.5%), which is precisely the birthday
bound for 1000 slots, and at 100,000 clients the busiest slot holds 128. The
behaviour is fine - a shared slot is two clients 15ms apart at the 30s cap, not
simultaneous - but the sentence overreached, and it overreached about exactly
the whole-player-base case the prose invokes. Now states the slot count and
what sharing one costs.

"Re-spreads each attempt" implied an independent re-draw. Every client's value
does change each attempt (0 of 400 unchanged between attempts 1 and 2), but the
fold is affine in the attempt seed, so the cohort is TRANSLATED by one constant
rather than reshuffled: h(id,2)-h(id,1) mod 2^32 measured as the single value
2807132593 across all 400 real identities, and 57% of pairs sharing a slot at
attempt 1 still share one at attempt 2 against 0.1% for an independent draw.
The population stays uniform at every attempt, so the feature works, but a
reader expecting re-randomisation would be wrong about the failure mode.
Reworded to the accurate, weaker claim.

Also makes the spread range exact. The attainable set is {0.5 + k/2000 : k in
0..999}, so the maximum is 99.95% and 100% is never produced; "50-100%" now
reads 50-99.95% in both places, matching the [0.5, 1.0) already in the API
table.

Not changed, deliberately: spread_for(id, nil) raises a raw arithmetic error.
main's backoff_delay(nil, 1, 30) raises identically, the SDK's own caller can
never pass nil, and matching the file's existing rigor beats adding a guard for
an unreachable case. On the record rather than silently accepted.

yuris commented 2026-08-20 14:10:41 +00:00

Author

Owner

Copy link

Review actioned. Verdict: READY TO MERGE — matching yours. Two README claims
fixed, one filed as follow-up, one recorded and deliberately not changed.

Thank you for measuring the distribution instead of accepting the two-sample
example in my PR body. Both of my stronger claims were false, and the PR body
repeated one of them.

D1 (every client lands on a different slot) — fixed

1000 possible outputs, 365 of 1000 clients colliding, busiest slot at 100k
holding 128. My sentence overreached about exactly the whole-player-base case
the surrounding prose invokes. The row now states the slot count and what
sharing one actually costs (15ms apart at the cap, not simultaneous), so a
reader gets the real shape.

D2 (re-spreads each attempt) — fixed, and the measurement is the interesting part

h(id,2) - h(id,1) mod 2^32 being the single constant 2807132593 across all 400
real identities is a much sharper statement than "it changes each attempt", and
57% of colliding pairs staying collided against 0.1% for an independent draw is
the number that makes it matter. The fold is affine in the attempt seed, so the
cohort translates rather than reshuffles. Reworded to the accurate weaker claim.
My PR body's "rather than marching in lockstep at a fixed offset" was wrong in
the same way — a fixed offset is precisely what it is.

D4 (50–100% reads inclusive) — fixed

The attainable set is {0.5 + k/2000 : k in 0..999}, so 100% is never produced.
Both prose sites now read 50–99.95%, matching the [0.5, 1.0) already in the API
table.

D5 (mass cold start has no spread) — filed as a follow-up

This is the most valuable thing in the review and I want it on the record
properly rather than buried in a merged PR, so it is now its own issue. You are
right that "server is down, the whole player base relaunches" is a real
synchronised herd that #19 does not cover, and right that the data needed is
already loaded three lines above the callsite — connect reads persisted.token
but never persisted.identity, and connection.new starts identity = nil.

I did not fold it into this PR because it is a genuine judgement call, not an
oversight: the on-disk identity is not server-confirmed at that point, so feeding
it to spread_for means spreading on an unverified value. That deserves deciding
on its own terms rather than as a rider.

D3 (raw error on a nil attempt) — recorded, not changed

Agreed with your own reasoning: main's backoff_delay(nil, 1, 30) raises
identically, the SDK's caller can never pass nil, and matching the file's
existing rigor beats a guard for an unreachable case. Noted in the commit so the
choice is on the record rather than silently accepted. Same for the 1/0 and
0/0 observation.

Independently re-verified after the edits

lua dev/test/runner.lua 6 passed 0 failed; the diff since your review is
README-only (+3/-3).

Your end-to-end closure — reconnecting attempt 1 in 0.9555s and then
spread_for reproducing 0.9555 exactly from the run's own logged identity — is a
better proof than anything in my original PR body.

Review actioned. Verdict: **READY TO MERGE** — matching yours. Two README claims fixed, one filed as follow-up, one recorded and deliberately not changed. Thank you for measuring the distribution instead of accepting the two-sample example in my PR body. Both of my stronger claims were false, and the PR body repeated one of them. ## D1 (every client lands on a different slot) — fixed 1000 possible outputs, 365 of 1000 clients colliding, busiest slot at 100k holding 128. My sentence overreached about exactly the whole-player-base case the surrounding prose invokes. The row now states the slot count and what sharing one actually costs (15ms apart at the cap, not simultaneous), so a reader gets the real shape. ## D2 (re-spreads each attempt) — fixed, and the measurement is the interesting part `h(id,2) - h(id,1) mod 2^32` being the single constant 2807132593 across all 400 real identities is a much sharper statement than "it changes each attempt", and 57% of colliding pairs staying collided against 0.1% for an independent draw is the number that makes it matter. The fold is affine in the attempt seed, so the cohort translates rather than reshuffles. Reworded to the accurate weaker claim. My PR body's "rather than marching in lockstep at a fixed offset" was wrong in the same way — a fixed offset is precisely what it is. ## D4 (50–100% reads inclusive) — fixed The attainable set is `{0.5 + k/2000 : k in 0..999}`, so 100% is never produced. Both prose sites now read 50–99.95%, matching the `[0.5, 1.0)` already in the API table. ## D5 (mass cold start has no spread) — filed as a follow-up This is the most valuable thing in the review and I want it on the record properly rather than buried in a merged PR, so it is now its own issue. You are right that "server is down, the whole player base relaunches" is a real synchronised herd that #19 does not cover, and right that the data needed is already loaded three lines above the callsite — `connect` reads `persisted.token` but never `persisted.identity`, and `connection.new` starts `identity = nil`. I did not fold it into this PR because it is a genuine judgement call, not an oversight: the on-disk identity is not server-confirmed at that point, so feeding it to `spread_for` means spreading on an unverified value. That deserves deciding on its own terms rather than as a rider. ## D3 (raw error on a nil attempt) — recorded, not changed Agreed with your own reasoning: `main`'s `backoff_delay(nil, 1, 30)` raises identically, the SDK's caller can never pass nil, and matching the file's existing rigor beats a guard for an unreachable case. Noted in the commit so the choice is on the record rather than silently accepted. Same for the `1/0` and `0/0` observation. ## Independently re-verified after the edits `lua dev/test/runner.lua` 6 passed 0 failed; the diff since your review is README-only (+3/-3). Your end-to-end closure — `reconnecting attempt 1 in 0.9555s` and then `spread_for` reproducing 0.9555 exactly from the run's own logged identity — is a better proof than anything in my original PR body.

yuris referenced this pull request 2026-08-20 14:10:41 +00:00

reconnect: a mass cold start still retries in lockstep #65

yuris merged commit cf8b430dc1 into main 2026-08-20 14:12:52 +00:00

yuris referenced this pull request from a commit 2026-08-20 14:12:54 +00:00

feat(connection): spread the reconnect backoff per client, deterministically (#19) (#62)

yuris referenced this pull request 2026-08-20 14:15:21 +00:00

v1.1.0 release checklist #59

yuris referenced this pull request 2026-08-20 16:01:40 +00:00

v1.1.0 release checklist #59

yuris referenced this pull request 2026-08-20 16:32:45 +00:00

v1.1.0 release checklist #59

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area:defold

Defold engine integration

  

area:docs

Documentation and examples

  

area:infra

Flake, CI, secretspec

  

area:protocol

Wire protocol conformance

  

area:testing

Test harness and fixtures

  

kind:bug

Something broken

  

kind:chore

Maintenance

  

phase:breadth

UnsubscribeMulti, procedures, BSATN

  

phase:core

Core SDK layers (protocol/cache/connection)

  

phase:hardening

Reconnect, token persistence, errors

  

phase:slice

Vertical slice: first playable end-to-end

No labels

area:defold

area:docs

area:infra

area:protocol

area:testing

kind:bug

kind:chore

phase:breadth

phase:core

phase:hardening

phase:slice

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/spacetimedb-defold!62

No description provided.