Conformance in CI: works locally, blocked on runner environment #7
Labels
No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
yuris/spacetimedb-defold#7
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Tier-3 testing: GitHub-Actions-style pipeline (Forgejo Actions) that:
spacetime start+ publishes test moduleUnit CI landed in
0a6602c—.forgejo/workflows/ci.ymlruns stylua, the lua5.1 unit suite, andnix flake checkon every push and PR (run #2 green).Two things that bit during setup, worth knowing before the conformance job is written:
test/.deps/json.luais gitignored, so a fresh checkout has no JSON codec — the dev shells now symlink it from the store viavendorJson. Any CI step that runs the tests must go through a dev shell (or the check derivation), never barelua..#cishell (lua5_1 + stylua only). Do not point conformance at the default shell — it drags in jdk25, bob, and lua-language-server. The bob toolchain is needed for the headless-dmengine job, so that job should get its own shell attr rather than reusingdefault.This issue now covers only the live-server tier:
spacetime start+ published module + two headless clients asserting cache convergence.Conformance CI: run example against spacetime startto Conformance CI: live-server job (spacetime start + 2 headless clients)Conformance job failed on run #5 (commit
3be6593db1):copying path '/nix/store/kq6bqbxkvblwxx4g89l63103jq72lxcd-roc-toolkit-0.4.0' from 'https://cache.nixos.org'... copying path '/nix/store/ahpzhy9kw4w3wnva40n8bi6qlhn9frsy-icu4c-78.3-dev' from 'https://cache.nixos.org'... copying path '/nix/store/28pl6r2nw05ai4fn9bvmdfaq2jb96l0g-libical-3.0.20' from 'https://cache.nixos.org'... copying path '/nix/store/w0p3046zxcqlybnhpz71xzivnxp30wy0-tinysparql-3.11.1' from 'https://cache.nixos.org'... copying path '/nix/store/kx9qcgaqkpchayz6a604zwq2cbj5d24i-ffmpeg-headless-9.0-lib' from 'https://cache.nixos.org'... copying path '/nix/store/wbv6jwxq9ay99jwx1sagxcgwrd10wk9f-bluez-5.87' from 'https://cache.nixos.org'... copying path '/nix/store/bfqsxlviikq7vlp36kasy5hhamlxlkd2-nodejs-slim-24.19.0' from 'https://cache.nixos.org'... copying path '/nix/store/mijaaxa13nv1h754xdic87f31z3mnbc6-gtk+3-3.24.52' from 'https://cache.nixos.org'... copying path '/nix/store/0a9l8lf8394msppsna27y58f8ljqyifn-openjdk-25.0.4+7' from 'https://cache.nixos.org'... copying path '/nix/store/hgcqpw6f8yg3cf0yii0p72kdp8334shf-nodejs-slim-24.19.0-corepack' from 'https://cache.nixos.org'... copying path '/nix/store/6x3bfz00nsp3qhcgg6wzvgfv67dhgy9j-nodejs-slim-24.19.0-npm' from 'https://cache.nixos.org'... building '/nix/store/px8hj7awj6ffpcamsaf95ywrmnn8n6aq-nodejs-24.19.0.drv'... copying path '/nix/store/3d1c302vw7kc8a5vknhmn34c0pd7zm6m-gcc-wrapper-15.3.0' from 'https://cache.nixos.org'... copying path '/nix/store/b4rc51gg4wxzia8xg834aphj83p5ss33-gstreamer-1.28.5' from 'https://cache.nixos.org'... copying path '/nix/store/4fgkln5dql87s2dlbahyh2cia78ivs2n-python3.14-packaging-26.2' from 'https://cache.nixos.org'... copying path '/nix/store/w60q4s3anfrlnn72841kw60fsjc0qmhk-stdenv-linux' from 'https://cache.nixos.org'... copying path '/nix/store/lfrjb7zz2g8mkm1g1jjg2hxwd1crryf6-glib-2.88.3-dev' from 'https://cache.nixos.org'... copying path '/nix/store/fdjdj3h57zi1g4x4pd8qplrflyz78883-gst-plugins-base-1.28.5' from 'https://cache.nixos.org'... copying path '/nix/store/n1cr847pvw2zppnsbl7s7cgpg4ibpypg-libcanberra-0.30' from 'https://cache.nixos.org'... building '/nix/store/4pdb4ygw35iiiik9sajriz7zr0cb8v4z-bob-run.drv'... copying path '/nix/store/ax8jrqcq50digb2xx9hlbi3xw70h9iqb-libcamera-0.7.0' from 'https://cache.nixos.org'... building '/nix/store/086v3sk8xdmcdqsyhq64xzvv2bxibbf7-fhsenv-ensure-gsettings-schemas-directory.drv'... copying path '/nix/store/h06qaqbc0llyr2vpqrj9ia2dpwgw120m-pipewire-1.6.8' from 'https://cache.nixos.org'... building '/nix/store/9c6fxh4pc12sp0ndk6bidn4g39si9n8b-bob-init.drv'... building '/nix/store/bpvg0sz2ibjll478jidnc7rvnx357pmn-defold-run.drv'... building '/nix/store/lvxy0iqd6am5zfqs9lp896rksjcb4dig-json.lua.drv'... copying path '/nix/store/bs3khmbhwlam0n9bj7n8nyj76kn0ki9a-openal-soft-1.24.3' from 'https://cache.nixos.org'... building '/nix/store/jzk32z8w4s92v5a8v8b3s6ixlxx8f3y2-defold-run-init.drv'... building '/nix/store/fbmh45jb2x4fhvi9612yxng6m30df85w-defold-run-fhsenv-rootfs.drv'... building '/nix/store/dbjc7m9hif2dd2kyxydydydxiafrq0l3-defold-run-bwrap.drv'... building '/nix/store/gs5qmm4s8igp91xj6l1rhpbn38z1za0j-defold-run.drv'... building '/nix/store/xxp6q54qv9h4s0rj01aqdsza8yifk6by-bob-fhsenv-rootfs.drv'... building '/nix/store/m3a2ml31jlz5jn92dvbaa7mkyxmnwp14-bob-bwrap.drv'... building '/nix/store/c2h8gdmvjzxkw8grlkxnnmyaxi1bxqvh-bob.drv'... building '/nix/store/g26kxq0yirdkwy5ams2kv5m1l4sxgmif-nix-shell-env.drv'... this path will be fetched (107.5 KiB download, 107.4 KiB unpacked): /nix/store/mrn1i6hs3jaxj5h0ab5bi0jdqaa004nn-bash-interactive-5.3p15-man copying path '/nix/store/mrn1i6hs3jaxj5h0ab5bi0jdqaa004nn-bash-interactive-5.3p15-man' from 'https://cache.nixos.org'... curl: (7) Failed to connect to 127.0.0.1:3000 after 0 ms: Could not connect to server [dev-server] server never became readyConformance failed on run #7 (
8989aa8836).Runner topology:
Tail:
Conformance failed on run #8 (
b30a36aa30).Runner topology:
Tail:
Status: works locally, unresolved on this runner — now manual
The engine conformance test is done and passing. From cold, in ~15s:
It starts a real
clockworklabs/spacetime:v2.8.2, publishes the chat module, builds the Defold project withbob --variant headless, runs the engine, and asserts the row round-tripped into the client cache. This is the only coverage forstdb/init.lua, and it immediately earned its keep by catching two engine-only bugs (binary-vs-text frames, andrequire("websocket")failing at build time).What happened in CI
Five runs, each failing while unit gates stayed green. Findings, in order:
in-container: yes, client 29.5.x / server 29.4.3). Sibling containers publish ports on the host, so127.0.0.1:3000from the job is a dead end. Fixed by probing the container bridge IP, then by attaching the server to the job's own netns via--network container:$(hostname)(the job's hostname is its container id).issues: writewas missing, so the self-report curl was rejected and swallowed by its|| true. Fixed.defold-runwas abuildFHSEnv, which needs user namespaces that unprivileged job containers usually deny. Replaced withdefold-patch(patchelf the interpreter + rpath — no namespaces). Still failed.Run #10 spent ~5 minutes before failing, so it got well past setup into the nix/bob work, then died without the reporter landing a comment.
Why it is now manual
This Forgejo build serves no job logs over the API (
/actions/runs/{id}/jobs,/logsall 404), and the issue-comment channel only lands intermittently. Iterating blind at ~5 min per attempt against an environment I cannot inspect is a bad trade, and a permanently red job on every push is noise.Conformance moved to
.forgejo/workflows/conformance.yml:workflow_dispatch+ weekly schedule,continue-on-error, and it reports outcome to this issue on every run (if: always()), including a topology block withusernscount, whetherunshare --userworks, and free disk.To close this issue
Trigger the workflow manually once and read the comment it posts. The topology block should say which of these it is:
unshare works: no→ the runner denies user namespaces; nix builds needing sandboxing will struggle. Run the runner with--privileged, or give it a pre-built nix store.free disk→ the run pulls a JDK, bob FHS env, node and the Defold engine; a small runner volume will fill.Alternatively, accept it as a local pre-release gate. It is one command and it is documented in the README.
Conformance CI: live-server job (spacetime start + 2 headless clients)to Conformance in CI: works locally, blocked on runner environmentRunner config obtained — one hypothesis is dead, and the blocker is still observability
The runner is a
code.forgejo.org/forgejo/runner:12container on the orchestrator with the host docker socket mounted, spawning job containers as siblings on the host daemon:Finding 4 was wrong: user namespaces are not the problem
Those three options are exactly what bubblewrap needs. Reproduced locally with the same flags:
So the
defold-run→defold-patchswitch was not the fix it looked like (harmless, and still the better default), andunshare works: nois unlikely to be what the topology block reports. Caveat:max_user_namespacesis a host sysctl, so a the orchestrator worker with it set to 0 would still fail — the topology step prints it, which is why the log matters.What is actually expensive: no persisted /nix
catthehacker/ubuntu:act-22.04ships no nix, so every run installs it and then downloads the whole conformance closure — JDK 25, bob's FHS env, the Defold engine, node, esbuild, and now chromium. Roughly 2 GB, per run, discarded at the end. That matches run #10 burning ~5 minutes before dying, and makes disk exhaustion a live suspect on a small the orchestrator volume.Nix is not the missing piece here: this pipeline already uses it, and that determinism is why the build got as far as it did. What broke is host and topology properties — capabilities, kernel sysctls, docker socket semantics, disk — and no flake pins those.
Fixed now: logs land somewhere readable
The reason five runs taught us so little is that this Forgejo serves no job logs over the API. The generic package registry is a different code path and does work — verified against this instance:
conformance.ymlnow publishestopology.logand the fullconformance.logthere onif: always(), versioned by run number and attempt, and the issue comment links to it. Also fixed: the comment body was escaped with hand-rolledsed, which mangles any log containing a quote or backslash — that is most of them. It usesjq -Rsnow, falling back to python3.Two smaller things:
Install nixprobedcommand -v nix, which is the wrong test once /nix is a persisted volume: the store is warm but the per-user profile is not, so it would reinstall on top of a perfectly good store. It now also checks/nix/var/nix/profiles/default/bin/nix./nix mounted:and free space on/nixseparately, so a warm-store change can be confirmed rather than assumed.tools/conformance.shcallsdocker, but the flake never provided it — it worked locally only because the workstation has a CLI on PATH.pkgs.docker-clientis in the.#conformanceshell now, so the job no longer depends on the image shipping one. Verified: native conformance passes end to end through the new shell, both runs, identity persisted.To actually close this
Runner-side changes are needed and they are not in this repo; filed as #18. The next manual run should now produce a readable log either way.
Docker is gone from conformance, which removes most of this issue's surface
Asked whether the test needs docker at all. It did not. It used it for three things — running the server, running the
spacetimeCLI to publish, anddocker restartto force the reconnect phase — and all three collapse into a pinned binary.nix/spacetime.nixpackages the upstream 2.8.2 release tarball: two files needing only glibc, libstdc++ and zlib, so autoPatchelf is the entire derivation. The tests now need nothing but nix.What that deletes from this investigation
Findings 1 and 4 are simply no longer reachable:
--network container:$(hostname)namespace sharing are deleted.127.0.0.1always means this machine.What is left that can fail on the runner: nix setup cost and disk. That is #18.
Why not nixpkgs' spacetimedb
Checked both channels, since it would have been the lazier option:
nixos-26.05nixpkgs-unstableBoth trail the version the fixtures were captured from, and
meta.license.free = false(BSL 1.1) means Hydra never builds it, so it is structurally absent fromcache.nixos.orgrather than incidentally missing. Using it would mean compiling SpacetimeDB and librusty_v8 on every cold runner in order to arrive at the wrong server version.Measured
Native tier, cold: 31s → 10s. HTML5 tier: 5s. Both pass:
Bugs found while doing it
yes y | spacetime publishreported a successful publish as a failure. Underset -o pipefailthe pipeline returnsyes's SIGPIPE (141). It only ever worked because the pipeline used to run insidedocker run sh -c, where pipefail was not set. Uses the CLI's own--yesnow.start_servernow checks its own child is alive and says so plainly.Side effect worth knowing
JWT signing keys are generated once and kept, so an identity token now survives a server restart. Every fresh docker container signed with new keys, which is what made cached tokens come back as mystery 401s.
tools/dev-server.shgrewstopfor clean teardown; the server is still deliberately left running between runs, the way the container was.Conformance run #25: failure (
a2cdbb23c5)Full log: https://git.nokopia.com/api/packages/yuris/generic/conformance-logs/run-25-1/conformance.log
Topology:
Tail:
Conformance run #28: success (
a9ba07efbb)Full log: https://git.nokopia.com/api/packages/yuris/generic/conformance-logs/run-28-1/conformance.log
Topology:
Tail:
Conformance run #29: success (
640ce03dd0)Logs:
Topology:
Tail:
Green. Closing.
Runs #28 and #29 both pass on the
nix-latestrunner, #29 in ~100s with a warm store. The full four-phase round trip runs in CI: connect, subscribe, reducer, server restart, subscription replay, query sets, procedures, and identity persistence across a process restart.What was actually wrong
Ten runs of failure, and none of it was the original hypothesis. In order of discovery:
127.0.0.1:3000was a dead end. Fixed by bridge-IP probing, then netns sharing — then deleted entirely, because docker itself turned out to be unnecessary (the server and CLI are pinned release binaries now).issues: writewas missing, so the self-report was swallowed.--cap-add SYS_ADMINwith unconfined seccomp and apparmor is exactly what bubblewrap needs; reproducing those flags locally givesunshare --user→ yes. Thedefold-run→defold-patchswitch was not the fix it appeared to be./actions/runs/*/jobs,/logs,/artifactsall 404. The self-report comment now carries a 120-line tail plus a topology block, and that is what finally made the next three failures diagnosable in one attempt each. The package-registry upload is rejected under the job token (generic packages are owner-scoped, the token is repo-scoped) and the comment now says so instead of linking a 404; wireCONFORMANCE_LOG_TOKENif the full log is ever wanted.defold-patchomitted libstdc++ from the rpath. Invisible locally because the workstation'sLD_LIBRARY_PATHhappened to carry a gcc-lib path;env -i lddsaidnot found. Surfaced by running the suite in a minimal nix container.ghcr.io/joschi/forgejo-nixsetsENTRYPOINT ["/bin/bash"], so the keep-alive command became/bin/bash tail -f /dev/nulland the container died before any step ran — 88s, no logs. Officialnixos/nixleavesEntrypointnull. It ships no node, soactions/checkout@v4was replaced with a plaingit clone; the workflow now needs only git and nix.valid_volumesdefaults to[], which silently discards every bind mount (hostConfig.Binds = []string{}insanitizeConfig). The--volume forgejo_nix_store:/nixwas dropped without failing anything — it would have cost the entire speedup while looking like it worked.bobneedsresolvebeforebuild..internal/libis gitignored, so a workstation always has it from an earlier build and a fresh checkout never does. Verified by deleting.internalanddistlocally and running from that state.Three of those nine (6, 8, 9) were only observable outside a developer machine. That is the actual value of this tier.
Runner configuration that works
On runner-node-b only, keeping the ubuntu labels so general capacity is unchanged:
The label is confined to one runner deliberately: docker volumes are node-local, so the warm store only exists on the node that built it.
Remaining work is tracked in #18 (runner notes), #16 (HTML5 tier in CI) and #15 (upstream extension fix).