yuris/spacetimedb-defold
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 9 Packages Wiki Activity Actions

dev-server restart returned before the module database had begun opening #39

New issue
Closed
opened 2026-08-19 17:03:10 +00:00 by yuris · 0 comments
yuris commented 2026-08-19 17:03:10 +00:00
Owner
Copy link

Found while verifying PR #36. Not caused by that PR — reproduced against main and against the branch without its changes, and the same tree then passed six consecutive runs plus HTML5.

Symptom

Native conformance failed once at phase 2:

reconnected but the row never came back

The failing run shows a second disconnect (code 1001) and a second reconnect attempt that passing runs do not.

Mechanism

dev/tools/dev-server.sh's wait_ready polls /v1/ping for a 200. That proves the HTTP port is answering. It does not prove the module database has finished reopening after a restart.

So a client that reconnects the instant ping goes green can be served a subscription against a database still coming up. The SDK does the right thing and retries, but phase 2 asserts the row is restored within its budget, and the extra round trip can push it past that.

This is the same class of flakiness as the CI reconnect timing in #16, and it has the same shape: nothing is broken, the readiness probe is just weaker than what the test depends on.

Work

  • Make wait_ready prove the database is ready, not just the HTTP port. A spacetime sql <db> "SELECT * FROM message LIMIT 0" or an equivalent cheap query against the actual module is the honest probe — it fails until the database is genuinely serving
  • Confirm the flake disappears by running native conformance repeatedly (the failing run was 1 in ~8, so a handful of runs is not evidence — aim for 20)
  • Consider whether the HTML5 harness needs the same treatment; it restarts the server the same way via dev-server.sh restart

Why bother

A 1-in-8 flake in the harness that gates a release is worse than a slow harness. It also erodes the thing that makes this repo's conformance useful: when it goes red, the current default assumption is that something real broke. Every unexplained flake weakens that.

Related: #16 keeps continue-on-error on the HTML5 CI step partly because reconnect timing on the runner is slower than local. If this readiness race is the underlying cause of both, fixing it here may be what lets that flag come off.

Found while verifying PR #36. Not caused by that PR — reproduced against `main` and against the branch without its changes, and the same tree then passed six consecutive runs plus HTML5. ## Symptom Native conformance failed once at phase 2: ``` reconnected but the row never came back ``` The failing run shows a **second** disconnect (code 1001) and a second reconnect attempt that passing runs do not. ## Mechanism `dev/tools/dev-server.sh`'s `wait_ready` polls `/v1/ping` for a 200. That proves the HTTP port is answering. It does **not** prove the module database has finished reopening after a `restart`. So a client that reconnects the instant ping goes green can be served a subscription against a database still coming up. The SDK does the right thing and retries, but phase 2 asserts the row is restored within its budget, and the extra round trip can push it past that. This is the same class of flakiness as the CI reconnect timing in #16, and it has the same shape: nothing is broken, the readiness probe is just weaker than what the test depends on. ## Work - [ ] Make `wait_ready` prove the *database* is ready, not just the HTTP port. A `spacetime sql <db> "SELECT * FROM message LIMIT 0"` or an equivalent cheap query against the actual module is the honest probe — it fails until the database is genuinely serving - [ ] Confirm the flake disappears by running native conformance repeatedly (the failing run was 1 in ~8, so a handful of runs is not evidence — aim for 20) - [ ] Consider whether the HTML5 harness needs the same treatment; it restarts the server the same way via `dev-server.sh restart` ## Why bother A 1-in-8 flake in the harness that gates a release is worse than a slow harness. It also erodes the thing that makes this repo's conformance useful: when it goes red, the current default assumption is that something real broke. Every unexplained flake weakens that. Related: #16 keeps `continue-on-error` on the HTML5 CI step partly because reconnect timing on the runner is slower than local. If this readiness race is the underlying cause of both, fixing it here may be what lets that flag come off.
yuris closed this issue 2026-08-20 06:37:09 +00:00
yuris changed title from conformance: dev-server readiness probe races the database reopening to dev-server restart returned before the module database had begun opening 2026-08-20 06:38:25 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
release/v4.0.1
fix/hold-pre-open-frames
v4.0.2
v4.0.1
v4.0.0
v3.1.0
v3.0.0
v2.0.0
v1.1.0
v1.0.0
v0.1.0
Labels
Clear labels   
area:defold
Defold engine integration

  
area:docs
Documentation and examples

  
area:infra
Flake, CI, secretspec

  
area:protocol
Wire protocol conformance

  
area:testing
Test harness and fixtures

  
kind:bug
Something broken

  
kind:chore
Maintenance

  
phase:breadth
UnsubscribeMulti, procedures, BSATN

  
phase:core
Core SDK layers (protocol/cache/connection)

  
phase:hardening
Reconnect, token persistence, errors

  
phase:slice
Vertical slice: first playable end-to-end

No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/spacetimedb-defold#39
No description provided.