fix: refuse to start over another port's server in a shared state dir #73

Merged
yuris merged 2 commits from dev/63-shared-state-guard into main 2026-08-20 17:30:28 +00:00
Owner

Closes #63.

Point two ports at one explicit STDB_STATE_DIR and the second start_server fails SpacetimeDB's database lock — correctly, the lock is exclusive — but it still wrote $PIDFILE with its own doomed child's pid. After that, stop on the first port read a dead pid, reported no server running, and left a live server nothing could find again.

server_pid() cannot repair this after the fact: the first pid is already gone from the file by the time anyone asks. So the fix refuses the overwrite, before anything is spawned.

The guard reads /proc/$pid/cmdline for the pid already in the pidfile — the same read server_pid() already does — and aborts if it belongs to a live server on a different port. Same trailing-space match as the existing check (it stops port 692 matching a server on 6921), and the same degradation: without /proc, the guard is inert and behaviour is exactly what it was.

It sits at the top of start_server, before the keypair generation as well as the spawn, so a refused invocation writes nothing at all into the contested state dir.

Reproduced first, then fixed

On main @ a606406, two ports sharing /tmp/63-test:

$ cat /tmp/63-test/server.pid
4163738
$ STDB_PORT=6922 STDB_STATE_DIR=/tmp/63-test ... dev-server.sh
[dev-server] starting spacetimedb-standalone 2.8.2 at http://127.0.0.1:6922
Error: cannot take lock on database; spacetime.pid already exists (owned by pid Some(4163738))
[dev-server] server never answered http://127.0.0.1:6922/v1/ping
=== pidfile now: 4164257        <- 6922's dead child

The leak, exactly as the issue describes:

$ STDB_PORT=6921 STDB_STATE_DIR=/tmp/63-test ... dev-server.sh stop
[dev-server] no server running
$ pgrep -af 'listen-addr 127.0.0.1:692'
4163738 spacetimedb-standalone start --listen-addr 127.0.0.1:6921 ...   <- still live, unreachable via stop

On this branch, same sequence:

[dev-server] state dir /tmp/63-test already hosts a server for another port (pid 4168108).
[dev-server] Two ports cannot share one STDB_STATE_DIR - the database lock is exclusive.
[dev-server] Unset STDB_STATE_DIR (the default derives from the port) or stop that server first.
=== exit code: 1
=== pidfile: 4168108            <- unchanged, still 6921's

It also fails in 2.08s instead of 31.59s, because it no longer waits out a ping timeout on a server that was never going to start. And stop now works:

$ STDB_PORT=6921 STDB_STATE_DIR=/tmp/63-test ... dev-server.sh stop
[dev-server] stopping server (pid 4168108)
=== ping 6921: 000
=== pgrep: (empty)

Untouched paths confirmed

  • Same-port re-invocation with a shared state dir still reports server already running at http://127.0.0.1:6921 (pid …) and proceeds to publish.
  • Two ports with default state dirs still coexist (the #54 behaviour) — both answered 200, both stopped cleanly.

nix run nixpkgs#shellcheck -- dev/tools/dev-server.sh exits 0 with no output. Lua suite and stylua --check unaffected (7 passed, 0 failed), as expected for a shell-only change.

One scope note: the guard keys on the pid recorded in the pidfile, so if the first server is already stopped and the pidfile is merely stale, the second port proceeds and hits SpacetimeDB's own database lock. That is unchanged and correct — that error is legible, and it no longer destroys a live server's record on the way.

Closes #63. Point two ports at one explicit `STDB_STATE_DIR` and the second `start_server` fails SpacetimeDB's database lock — correctly, the lock is exclusive — but it *still* wrote `$PIDFILE` with its own doomed child's pid. After that, `stop` on the first port read a dead pid, reported `no server running`, and left a live server nothing could find again. `server_pid()` cannot repair this after the fact: the first pid is already gone from the file by the time anyone asks. So the fix refuses the overwrite, before anything is spawned. The guard reads `/proc/$pid/cmdline` for the pid already in the pidfile — the same read `server_pid()` already does — and aborts if it belongs to a live server on a *different* port. Same trailing-space match as the existing check (it stops port `692` matching a server on `6921`), and the same degradation: without `/proc`, the guard is inert and behaviour is exactly what it was. It sits at the top of `start_server`, before the keypair generation as well as the spawn, so a refused invocation writes nothing at all into the contested state dir. ## Reproduced first, then fixed On `main` @ a606406, two ports sharing `/tmp/63-test`: ``` $ cat /tmp/63-test/server.pid 4163738 $ STDB_PORT=6922 STDB_STATE_DIR=/tmp/63-test ... dev-server.sh [dev-server] starting spacetimedb-standalone 2.8.2 at http://127.0.0.1:6922 Error: cannot take lock on database; spacetime.pid already exists (owned by pid Some(4163738)) [dev-server] server never answered http://127.0.0.1:6922/v1/ping === pidfile now: 4164257 <- 6922's dead child ``` The leak, exactly as the issue describes: ``` $ STDB_PORT=6921 STDB_STATE_DIR=/tmp/63-test ... dev-server.sh stop [dev-server] no server running $ pgrep -af 'listen-addr 127.0.0.1:692' 4163738 spacetimedb-standalone start --listen-addr 127.0.0.1:6921 ... <- still live, unreachable via stop ``` On this branch, same sequence: ``` [dev-server] state dir /tmp/63-test already hosts a server for another port (pid 4168108). [dev-server] Two ports cannot share one STDB_STATE_DIR - the database lock is exclusive. [dev-server] Unset STDB_STATE_DIR (the default derives from the port) or stop that server first. === exit code: 1 === pidfile: 4168108 <- unchanged, still 6921's ``` It also fails in 2.08s instead of 31.59s, because it no longer waits out a ping timeout on a server that was never going to start. And `stop` now works: ``` $ STDB_PORT=6921 STDB_STATE_DIR=/tmp/63-test ... dev-server.sh stop [dev-server] stopping server (pid 4168108) === ping 6921: 000 === pgrep: (empty) ``` ## Untouched paths confirmed - Same-port re-invocation with a shared state dir still reports `server already running at http://127.0.0.1:6921 (pid …)` and proceeds to publish. - Two ports with **default** state dirs still coexist (the #54 behaviour) — both answered `200`, both stopped cleanly. `nix run nixpkgs#shellcheck -- dev/tools/dev-server.sh` exits 0 with no output. Lua suite and `stylua --check` unaffected (`7 passed, 0 failed`), as expected for a shell-only change. One scope note: the guard keys on the pid recorded in the pidfile, so if the first server is already stopped and the pidfile is merely stale, the second port proceeds and hits SpacetimeDB's own database lock. That is unchanged and correct — that error is legible, and it no longer destroys a live server's record on the way.
fix: refuse to start over another port's server in a shared state dir (#63)
All checks were successful
ci / ci (pull_request) Successful in 20s
e7cee3d694
Author
Owner

Both findings actioned. The first was a real hole and the review is right that it mattered more than a corner case.

restart bypassed the guard. stop_server ends in an unconditional rm -f "$PIDFILE", and restart) called it before start_server — so the first port's record was deleted before the guard could read it, and #63 reproduced verbatim. restart is also what both conformance harnesses drive, so that was the common path, not the rare one.

The guard is now a function, refuse_foreign_state_dir(), sitting beside server_pid() and called from two places: the first statement of start_server, and the first statement of restart) before stop_server. stop_server's rm -f is left alone so stale-pidfile cleanup keeps working, and stop) needs nothing since it only calls stop_server when server_pid already matched this port.

Reproduced the bypass and then its absence, ports 6951/6952 sharing /tmp/63-restart:

pidfile after start: 558
=== RESTART on 6952 ===
[dev-server] state dir /tmp/63-restart already hosts a server for another port (pid 558).
[dev-server] Two ports cannot share one STDB_STATE_DIR - the database lock is exclusive.
[dev-server] Unset STDB_STATE_DIR (the default derives from the port) or stop that server first.
pidfile after restart attempt: 558
=== stop 6951 ===
[dev-server] stopping server (pid 558)
ping 6951: 000
leftovers: 0

Second finding: the mkdir -p claim was false. Correct, and now true rather than reworded — the guard runs before mkdir -p, as the first statement of start_server, since it only reads $PIDFILE.

Same-port paths re-verified, because moving the check into restart put the harnesses' own path at risk:

=== same-port RESTART ===
[dev-server] restarting server (data preserved)
[dev-server] server back up at http://127.0.0.1:6951
pidfile now: 1275 (was 1110)     ping 6951: 200
=== same-port re-invocation ===
[dev-server] server already running at http://127.0.0.1:6951 (pid 1275)
=== two DEFAULT state dirs (#54) ===
6951: 200  6952: 200
leftovers: 0

shellcheck still exits 0. Native conformance passes on the branch (all six phases), which exercises the same-port restart path twice per run.

Both findings actioned. The first was a real hole and the review is right that it mattered more than a corner case. **`restart` bypassed the guard.** `stop_server` ends in an unconditional `rm -f "$PIDFILE"`, and `restart)` called it before `start_server` — so the first port's record was deleted before the guard could read it, and #63 reproduced verbatim. `restart` is also what both conformance harnesses drive, so that was the common path, not the rare one. The guard is now a function, `refuse_foreign_state_dir()`, sitting beside `server_pid()` and called from two places: the first statement of `start_server`, and the first statement of `restart)` *before* `stop_server`. `stop_server`'s `rm -f` is left alone so stale-pidfile cleanup keeps working, and `stop)` needs nothing since it only calls `stop_server` when `server_pid` already matched this port. Reproduced the bypass and then its absence, ports 6951/6952 sharing `/tmp/63-restart`: ``` pidfile after start: 558 === RESTART on 6952 === [dev-server] state dir /tmp/63-restart already hosts a server for another port (pid 558). [dev-server] Two ports cannot share one STDB_STATE_DIR - the database lock is exclusive. [dev-server] Unset STDB_STATE_DIR (the default derives from the port) or stop that server first. pidfile after restart attempt: 558 === stop 6951 === [dev-server] stopping server (pid 558) ping 6951: 000 leftovers: 0 ``` **Second finding: the `mkdir -p` claim was false.** Correct, and now true rather than reworded — the guard runs before `mkdir -p`, as the first statement of `start_server`, since it only reads `$PIDFILE`. **Same-port paths re-verified**, because moving the check into `restart` put the harnesses' own path at risk: ``` === same-port RESTART === [dev-server] restarting server (data preserved) [dev-server] server back up at http://127.0.0.1:6951 pidfile now: 1275 (was 1110) ping 6951: 200 === same-port re-invocation === [dev-server] server already running at http://127.0.0.1:6951 (pid 1275) === two DEFAULT state dirs (#54) === 6951: 200 6952: 200 leftovers: 0 ``` `shellcheck` still exits 0. Native conformance passes on the branch (all six phases), which exercises the same-port `restart` path twice per run.
fix: check the shared-state guard before restart wipes the pidfile (#63)
All checks were successful
ci / ci (pull_request) Successful in 1m28s
2d761131d8
yuris merged commit 81602be420 into main 2026-08-20 17:30:28 +00:00
Sign in to join this conversation.
No description provided.