Reconnect backoff has no jitter (recorded, not yet due) #19

New issue

Closed

opened 2026-08-19 14:58:39 +00:00 by yuris · 0 comments

yuris commented

2026-08-19 14:58:39 +00:00

Owner

stdb/connection.lua:203 carries a deliberate deferral:

---ponytail: no jitter. Fine for a handful of clients; if a server restart ever
---stampedes a whole player base, add a random spread here.
function M.backoff_delay(attempt, base, cap)

The trigger named there has not fired — nothing is stampeding anything — so this issue is a record, not a request to go implement it. Filing it because CI just produced the first real evidence that the reconnect path runs longer than a workstation suggests, which makes the ceiling worth writing down before someone ships a game on it.

The evidence

backoff_delay is base * 2^(attempt-1) capped at cap, with no random component: every client that loses the same server reconnects on the same schedule, to the millisecond, forever.

Locally a server restart is absorbed in one attempt. On the CI runner, run #38 took four:

[html5] SMOKE: server dropped us (code 1006)
[html5] SMOKE: reconnecting attempt 1 in 1s
[html5] SMOKE: reconnecting attempt 2 in 2s
[html5] SMOKE: reconnecting attempt 3 in 4s
[html5] SMOKE: reconnecting attempt 4 in 5s
[handshake] <- 101 Switching Protocols

That is the backoff working correctly. The point is the shape of it: a ~13s outage window is entirely normal for a real server restart, and during that window every connected client is retrying in lockstep at 1s, 2s, 4s, 8s… Each attempt is a fresh websocket upgrade plus a fresh IdentityToken round trip plus a resubscribe of every query the client had, so the herd arrives all at once precisely when the server is least able to absorb it — and a rejected retry pushes the whole herd to the next slot together.

Single-player-scale usage will never notice. A few hundred clients restarting against one module is where synchronized retries stop being theoretical.

Fix, when the trigger fires

Full jitter in backoff_delay, which is already a pure function with tests, so this is a contained change:

local delay = math.min(base * 2 ^ (attempt - 1), cap)
return delay * (0.5 + math.random() * 0.5)   -- 50-100% of the slot

Two consequences to handle rather than discover:

math.random makes the function impure, and the existing tests assert exact delays. Either inject the RNG as an argument defaulting to math.random (matching how storage and is_html5 are already injectable seams), or assert on the range instead of the value. The seam is more consistent with the rest of the SDK.
math.randomseed is per-process; Defold seeds it lazily. Without an explicit seed every client picks the same "random" spread, which reproduces the exact problem being fixed. This is the part that actually bites, and the reason not to treat jitter as a one-liner.

Not doing now, deliberately

The current behaviour is correct, tested, and adequate for every use this SDK has. Adding an RNG seam plus seeding plus range-based tests to a working backoff, on the strength of a load nobody is applying, buys nothing today and costs a permanently more complicated function.

Reasons to reopen the decision: a deployment with more than a few dozen concurrent clients, an observed retry storm after a module publish, or any report of reconnects failing in bursts rather than individually.

`stdb/connection.lua:203` carries a deliberate deferral: ```lua ---ponytail: no jitter. Fine for a handful of clients; if a server restart ever ---stampedes a whole player base, add a random spread here. function M.backoff_delay(attempt, base, cap) ``` The trigger named there has not fired — nothing is stampeding anything — so this issue is a record, not a request to go implement it. Filing it because CI just produced the first real evidence that the reconnect path runs longer than a workstation suggests, which makes the ceiling worth writing down before someone ships a game on it. ## The evidence `backoff_delay` is `base * 2^(attempt-1)` capped at `cap`, with no random component: every client that loses the same server reconnects on the *same* schedule, to the millisecond, forever. Locally a server restart is absorbed in one attempt. On the CI runner, [run #38](https://git.nokopia.com/yuris/spacetimedb-defold/actions/runs/38) took four: ``` [html5] SMOKE: server dropped us (code 1006) [html5] SMOKE: reconnecting attempt 1 in 1s [html5] SMOKE: reconnecting attempt 2 in 2s [html5] SMOKE: reconnecting attempt 3 in 4s [html5] SMOKE: reconnecting attempt 4 in 5s [handshake] <- 101 Switching Protocols ``` That is the backoff working correctly. The point is the *shape* of it: a ~13s outage window is entirely normal for a real server restart, and during that window every connected client is retrying in lockstep at 1s, 2s, 4s, 8s… Each attempt is a fresh websocket upgrade plus a fresh `IdentityToken` round trip plus a resubscribe of every query the client had, so the herd arrives all at once precisely when the server is least able to absorb it — and a rejected retry pushes the whole herd to the next slot together. Single-player-scale usage will never notice. A few hundred clients restarting against one module is where synchronized retries stop being theoretical. ## Fix, when the trigger fires Full jitter in `backoff_delay`, which is already a pure function with tests, so this is a contained change: ```lua local delay = math.min(base * 2 ^ (attempt - 1), cap) return delay * (0.5 + math.random() * 0.5) -- 50-100% of the slot ``` Two consequences to handle rather than discover: - **`math.random` makes the function impure**, and the existing tests assert exact delays. Either inject the RNG as an argument defaulting to `math.random` (matching how `storage` and `is_html5` are already injectable seams), or assert on the range instead of the value. The seam is more consistent with the rest of the SDK. - **`math.randomseed`** is per-process; Defold seeds it lazily. Without an explicit seed every client picks the *same* "random" spread, which reproduces the exact problem being fixed. This is the part that actually bites, and the reason not to treat jitter as a one-liner. ## Not doing now, deliberately The current behaviour is correct, tested, and adequate for every use this SDK has. Adding an RNG seam plus seeding plus range-based tests to a working backoff, on the strength of a load nobody is applying, buys nothing today and costs a permanently more complicated function. Reasons to reopen the decision: a deployment with more than a few dozen concurrent clients, an observed retry storm after a module publish, or any report of reconnects failing in bursts rather than individually.

yuris referenced this issue

2026-08-19 15:20:43 +00:00

v1.0.0 release checklist #24

yuris referenced this issue

2026-08-19 18:26:34 +00:00

v1.0.0 release checklist #24

yuris referenced this issue

2026-08-20 01:45:51 +00:00

v1.0.0 release checklist #24

yuris referenced this issue

2026-08-20 13:26:59 +00:00

v1.1.0 release checklist #59