yuris/spacetimedb-defold
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 9 Packages Wiki Activity Actions

dev-server: two ports sharing one explicit STDB_STATE_DIR leak a server #63

New issue
Closed
opened 2026-08-20 13:52:23 +00:00 by yuris · 2 comments
yuris commented 2026-08-20 13:52:23 +00:00
Owner
Copy link

Found while reviewing #60 (which closes #54). Not fixed there: #54 removes the default collision, and thiscase needs an explicit shared state dir.

Reproduced on the #54 branch:

  1. STDB_PORT=6913 STDB_STATE_DIR=/tmp/shared dev-server.sh -> pidfile holds 3848953, serving.
  2. STDB_PORT=6919 STDB_STATE_DIR=/tmp/shared dev-server.sh -> cannot take the database lock (server.log: spacetime.pid already exists, owned by 3848953), spends wait_port fully, and OVERWRITES the pidfile with its own dead child 3849062.
  3. STDB_PORT=6913 ... dev-server.sh stop -> [dev-server] no server running.
  4. curl 127.0.0.1:6913/v1/ping -> 200. The original server is still alive and now has no record anywhere.

server_pid() cannot fix this: by step 3 the pid it would need is gone from the pidfile. The fix has to be earlier - refuse to start when the pidfile names a live server for a different port, or key the pidfile within the state dir by port.

Low priority. It requires deliberately pointing two ports at one STDB_STATE_DIR, which is exactly the configuration #54 stopped anyone reaching by accident.

Found while reviewing #60 (which closes #54). Not fixed there: #54 removes the *default* collision, and thiscase needs an explicit shared state dir. Reproduced on the #54 branch: 1. STDB_PORT=6913 STDB_STATE_DIR=/tmp/shared dev-server.sh -> pidfile holds 3848953, serving. 2. STDB_PORT=6919 STDB_STATE_DIR=/tmp/shared dev-server.sh -> cannot take the database lock (server.log: spacetime.pid already exists, owned by 3848953), spends wait_port fully, and OVERWRITES the pidfile with its own dead child 3849062. 3. STDB_PORT=6913 ... dev-server.sh stop -> [dev-server] no server running. 4. curl 127.0.0.1:6913/v1/ping -> 200. The original server is still alive and now has no record anywhere. server_pid() cannot fix this: by step 3 the pid it would need is gone from the pidfile. The fix has to be earlier - refuse to start when the pidfile names a live server for a different port, or key the pidfile within the state dir by port. Low priority. It requires deliberately pointing two ports at one STDB_STATE_DIR, which is exactly the configuration #54 stopped anyone reaching by accident.
yuris commented 2026-08-20 14:12:13 +00:00
Author
Owner
Copy link

Correction to this issue as filed, from the round-two review of #60.

I wrote that server_pid() cannot address this because "the pid it needs is
gone from the pidfile" and so "the fix has to be earlier". The first half is
true, the conclusion is not: the pid is still in /proc, which is the source of
truth #60 just introduced.

Review demonstrated a ~6-line fallback inside server_pid that scans /proc
for a cmdline containing both --listen-addr 127.0.0.1:$PORT and
--data-dir $DATA . It recovered the leaked server with no false positive on
the other port and none on a foreign state dir. No pidfile, no new dependency,
no pipeline.

So the option space is larger than this issue first suggested, and the cheapest
fix may well be inside server_pid after all. Leaving the priority as-is - it
still needs a deliberately shared state dir - but the next implementer should not
be sent past that option.

Correction to this issue as filed, from the round-two review of #60. I wrote that `server_pid()` cannot address this because "the pid it needs is gone from the pidfile" and so "the fix has to be earlier". The first half is true, the conclusion is not: the pid is still in `/proc`, which is the source of truth #60 just introduced. Review demonstrated a ~6-line fallback inside `server_pid` that scans `/proc` for a cmdline containing both `--listen-addr 127.0.0.1:$PORT ` and `--data-dir $DATA `. It recovered the leaked server with no false positive on the other port and none on a foreign state dir. No pidfile, no new dependency, no pipeline. So the option space is larger than this issue first suggested, and the cheapest fix may well be inside `server_pid` after all. Leaving the priority as-is - it still needs a deliberately shared state dir - but the next implementer should not be sent past that option.
yuris closed this issue 2026-08-20 17:30:28 +00:00
yuris commented 2026-08-20 18:48:50 +00:00
Author
Owner
Copy link

Closed by #73.

dev-server.sh now refuses to start when $PIDFILE holds a live server on a different port, instead of overwriting it with its own doomed child. server_pid() could not repair this after the fact — the first pid was already gone from the file — so the fix is refusing the overwrite.

Leak reproduced on main first, two ports sharing /tmp/63-test:

$ cat /tmp/63-test/server.pid
4163738
$ STDB_PORT=6922 ... dev-server.sh
Error: cannot take lock on database; spacetime.pid already exists (owned by pid Some(4163738))
=== pidfile now: 4164257        <- 6922's dead child
$ STDB_PORT=6921 ... dev-server.sh stop
[dev-server] no server running
$ pgrep -af 'listen-addr 127.0.0.1:692'
4163738 spacetimedb-standalone start --listen-addr 127.0.0.1:6921 ...   <- live, unreachable

After, the same sequence exits 1 in 2.1s (vs 31.6s waiting out a ping timeout) with the pidfile intact, and stop then works — ping returns 000, nothing left in pgrep.

Review caught that the first fix was incomplete in the path the project's own harnesses use. stop_server ends in an unconditional rm -f "$PIDFILE", and restart) called it before start_server — so restart on the second port deleted the record the guard reads and reproduced #63 verbatim on the branch. Since restart is what both conformance harnesses drive, that was the common path rather than a corner case. The guard is now a function, refuse_foreign_state_dir(), called from the top of start_server and from restart) before stop_server.

Untouched paths re-verified, because moving the check into restart put the harnesses' own path at risk: same-port restart still succeeds and the server comes back; same-port re-invocation still reports already running; two ports with default state dirs still coexist (the #54 behaviour, both answering 200); a stale pidfile still lets a start proceed. Guard is Linux-only via /proc, degrading to previous behaviour elsewhere — the same trade server_pid() already made.

shellcheck exit 0. Verified on main @ 52dd72d alongside the rest of the milestone: native and HTML5 conformance pass, 8 passed, 0 failed.

Closed by #73. `dev-server.sh` now refuses to start when `$PIDFILE` holds a live server on a *different* port, instead of overwriting it with its own doomed child. `server_pid()` could not repair this after the fact — the first pid was already gone from the file — so the fix is refusing the overwrite. Leak reproduced on `main` first, two ports sharing `/tmp/63-test`: ``` $ cat /tmp/63-test/server.pid 4163738 $ STDB_PORT=6922 ... dev-server.sh Error: cannot take lock on database; spacetime.pid already exists (owned by pid Some(4163738)) === pidfile now: 4164257 <- 6922's dead child $ STDB_PORT=6921 ... dev-server.sh stop [dev-server] no server running $ pgrep -af 'listen-addr 127.0.0.1:692' 4163738 spacetimedb-standalone start --listen-addr 127.0.0.1:6921 ... <- live, unreachable ``` After, the same sequence exits 1 in 2.1s (vs 31.6s waiting out a ping timeout) with the pidfile intact, and `stop` then works — ping returns `000`, nothing left in `pgrep`. **Review caught that the first fix was incomplete in the path the project's own harnesses use.** `stop_server` ends in an unconditional `rm -f "$PIDFILE"`, and `restart)` called it *before* `start_server` — so `restart` on the second port deleted the record the guard reads and reproduced #63 verbatim on the branch. Since `restart` is what both conformance harnesses drive, that was the common path rather than a corner case. The guard is now a function, `refuse_foreign_state_dir()`, called from the top of `start_server` **and** from `restart)` before `stop_server`. Untouched paths re-verified, because moving the check into `restart` put the harnesses' own path at risk: same-port `restart` still succeeds and the server comes back; same-port re-invocation still reports `already running`; two ports with default state dirs still coexist (the #54 behaviour, both answering 200); a stale pidfile still lets a start proceed. Guard is Linux-only via `/proc`, degrading to previous behaviour elsewhere — the same trade `server_pid()` already made. `shellcheck` exit 0. Verified on `main` @ 52dd72d alongside the rest of the milestone: native and HTML5 conformance pass, `8 passed, 0 failed`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
release/v4.0.1
fix/hold-pre-open-frames
v4.0.2
v4.0.1
v4.0.0
v3.1.0
v3.0.0
v2.0.0
v1.1.0
v1.0.0
v0.1.0
Labels
Clear labels   
area:defold
Defold engine integration

  
area:docs
Documentation and examples

  
area:infra
Flake, CI, secretspec

  
area:protocol
Wire protocol conformance

  
area:testing
Test harness and fixtures

  
kind:bug
Something broken

  
kind:chore
Maintenance

  
phase:breadth
UnsubscribeMulti, procedures, BSATN

  
phase:core
Core SDK layers (protocol/cache/connection)

  
phase:hardening
Reconnect, token persistence, errors

  
phase:slice
Vertical slice: first playable end-to-end

No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/spacetimedb-defold#63
No description provided.