Connection state machine: reconnect + resubscribe #5

New issue

Closed

opened 2026-08-19 06:49:20 +00:00 by yuris · 2 comments

yuris commented

2026-08-19 06:49:20 +00:00

Owner

On disconnect the SDK should reconnect with exponential backoff and replay subscriptions (new query_set_ids), dropping stale cache.

Reconnect with backoff (websocket extension fires EVENT_DISCONNECTED)
Replay Subscribe for each live query set after InitialConnection
Clear cache on reconnect to avoid stale rows
on_event gets reconnecting / reconnected types

On disconnect the SDK should reconnect with exponential backoff and replay subscriptions (new query_set_ids), dropping stale cache. - [ ] Reconnect with backoff (websocket extension fires EVENT_DISCONNECTED) - [ ] Replay Subscribe for each live query set after InitialConnection - [ ] Clear cache on reconnect to avoid stale rows - [ ] `on_event` gets `reconnecting` / `reconnected` types

yuris added the

area:defold

phase:hardening

labels

2026-08-19 06:49:20 +00:00

forgejo-actions referenced this issue

2026-08-19 07:58:46 +00:00

Conformance in CI: works locally, blocked on runner environment #7

yuris referenced this issue from a commit

2026-08-19 08:01:24 +00:00

ci: probe a reachable server address instead of assuming localhost

yuris commented

2026-08-19 08:34:36 +00:00

Author

Owner

Partially delivered in ee90548: connection.resubscribe_all() tracks live subscriptions, clears the cache (so stale rows cannot survive the gap), and replays every query. Covered by a unit test with a fake transport.

What is still missing is the part that needs the engine: nothing currently calls it. stdb/init.lua surfaces disconnected as an event but does not reconnect the websocket or back off.

Reconnect the socket on EVENT_DISCONNECTED with exponential backoff (double, cap ~30s)
Call resubscribe_all() once the new connection reports connected
Emit reconnecting / reconnected events
Extend main/smoke.script to kill and restore the server mid-session, proving recovery in-engine

Partially delivered in ee90548: `connection.resubscribe_all()` tracks live subscriptions, clears the cache (so stale rows cannot survive the gap), and replays every query. Covered by a unit test with a fake transport. What is still missing is the part that needs the engine: nothing currently *calls* it. `stdb/init.lua` surfaces `disconnected` as an event but does not reconnect the websocket or back off. - [ ] Reconnect the socket on `EVENT_DISCONNECTED` with exponential backoff (double, cap ~30s) - [ ] Call `resubscribe_all()` once the new connection reports `connected` - [ ] Emit `reconnecting` / `reconnected` events - [ ] Extend `main/smoke.script` to kill and restore the server mid-session, proving recovery in-engine

yuris commented

2026-08-19 08:42:23 +00:00

Author

Owner

Closed in `4c0682a` — proven against a real server restart

The binding now reconnects on its own, and the conformance test kills the server to prove it:

SMOKE: PHASE1 OK cached row id=1
SMOKE: reconnecting attempt 1 in 1s
SMOKE: server dropped us (code 0)
SMOKE: reconnected, replayed 1 subscription(s)
SMOKE: PASS survived restart; row id=1 restored

tools/conformance.sh runs the engine in the background, waits for PHASE1 OK, runs docker restart underneath it, and the test only passes if the client comes back and sees the row it inserted before the drop.

What landed

Backoff: connection.backoff_delay(attempt, base, cap) — pure, doubling, clamped. Unit tested at the boundaries (attempt 0, clamp, custom base). Defaults 1s → 30s, configurable via backoff_base / backoff_cap.
Identity survives: reconnect reuses the server-issued token from the first IdentityToken, so the client keeps its identity across the gap.
Cache correctness: on the new socket's IdentityToken, the cache is dropped and every live subscription replayed. Stale rows cannot outlive a disconnect — that was the risk worth protecting.
set_send: swapping transport also clears pending, so a callback owed by the dead socket can never fire against the new one. Unit tested.
db:close() stops reconnecting; reconnect = false opts out entirely.
Events documented in the README: disconnected, reconnecting (attempt, delay), reconnected (subscriptions replayed).

Known cosmetic wrinkle

extension-websocket fires ERROR before DISCONNECTED, so the log can show "reconnecting attempt 1" just before "server dropped us". The scheduler is idempotent (reconnect_pending guard), so only one attempt is queued — the ordering is just noisy, not wrong.

Deliberately not done

No jitter on the backoff. Fine for a handful of clients; if a server restart ever stampedes a whole player base, spread the retries. Marked with a ponytail: comment at the call site.

## Closed in 4c0682a — proven against a real server restart The binding now reconnects on its own, and the conformance test kills the server to prove it: ``` SMOKE: PHASE1 OK cached row id=1 SMOKE: reconnecting attempt 1 in 1s SMOKE: server dropped us (code 0) SMOKE: reconnected, replayed 1 subscription(s) SMOKE: PASS survived restart; row id=1 restored ``` `tools/conformance.sh` runs the engine in the background, waits for `PHASE1 OK`, runs `docker restart` underneath it, and the test only passes if the client comes back and sees the row it inserted before the drop. ### What landed - **Backoff**: `connection.backoff_delay(attempt, base, cap)` — pure, doubling, clamped. Unit tested at the boundaries (attempt 0, clamp, custom base). Defaults 1s → 30s, configurable via `backoff_base` / `backoff_cap`. - **Identity survives**: reconnect reuses the server-issued token from the first `IdentityToken`, so the client keeps its identity across the gap. - **Cache correctness**: on the new socket's `IdentityToken`, the cache is dropped and every live subscription replayed. Stale rows cannot outlive a disconnect — that was the risk worth protecting. - **`set_send`**: swapping transport also clears `pending`, so a callback owed by the dead socket can never fire against the new one. Unit tested. - **`db:close()`** stops reconnecting; `reconnect = false` opts out entirely. - Events documented in the README: `disconnected`, `reconnecting` (attempt, delay), `reconnected` (subscriptions replayed). ### Known cosmetic wrinkle `extension-websocket` fires ERROR before DISCONNECTED, so the log can show "reconnecting attempt 1" just before "server dropped us". The scheduler is idempotent (`reconnect_pending` guard), so only one attempt is queued — the ordering is just noisy, not wrong. ### Deliberately not done No jitter on the backoff. Fine for a handful of clients; if a server restart ever stampedes a whole player base, spread the retries. Marked with a `ponytail:` comment at the call site.

yuris closed this issue

2026-08-19 08:42:23 +00:00

yuris referenced this issue

2026-08-19 08:49:46 +00:00

Token persistence via sys.save #6

yuris referenced this issue

2026-08-19 16:11:43 +00:00

cache: report a pk row change as one update event #34