yuris/spacetimedb-defold

Fork 0

Runner: persist /nix and stop depending on the job image #18

New issue

Closed

opened 2026-08-19 12:16:41 +00:00 by yuris · 6 comments

yuris commented

2026-08-19 12:16:41 +00:00

Owner

Conformance in CI (#7) needs two changes to the Forgejo runner's the orchestrator service definition. Neither lives in this repo, so this issue is the record of what to change and why.

Current definition, the relevant part:

labels:
  - ubuntu-latest:docker://catthehacker/ubuntu:act-22.04
  - ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04
container:
  docker_host: unix:///var/run/docker.sock
  options: --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN

1. Persist /nix across runs

This is the expensive one. The job image has no nix, so every run installs it and re-downloads ~2 GB of closure (JDK 25, bob's FHS env, the Defold engine, node, esbuild, chromium) only to throw it away. That is most of the ~5 minutes before run #10 died, and it makes disk exhaustion a live suspect.

container:
  options: >-
    --security-opt seccomp=unconfined --security-opt apparmor=unconfined
    --cap-add SYS_ADMIN --volume forgejo_nix:/nix

First run populates the volume; later runs find a warm store. conformance.yml already detects this case — it looks for /nix/var/nix/profiles/default/bin/nix, not just nix on PATH, because the store persists while the per-user profile does not.

Watch for: the volume grows without bound. A periodic nix-collect-garbage -d, or just deleting the volume when it gets large, is the whole maintenance story.

2. Consider a nix-native label

An alternative to installing nix on every cold run:

labels:
  - nix:docker://nixos/nix:latest

Then runs-on: nix and the install step becomes a no-op. Note two consequences before doing it:

nixos/nix ships no docker CLI, which tools/conformance.sh needs to start the server. Already handled: pkgs.docker-client is in the .#conformance shell, so the CLI comes from the flake rather than the image.
It needs --extra-experimental-features "nix-command flakes" and sandbox = false (a nested nix sandbox inside a job container is unreliable), which is what the current nix.conf writing already sets up.

This is optional if change 1 lands — a warm store makes the installer cheap. Worth it mainly to stop depending on catthehacker's contents at all.

Not the problem

For the record, so it is not re-litigated: user namespaces are fine. --cap-add SYS_ADMIN plus unconfined seccomp and apparmor is what bubblewrap needs, confirmed by reproducing the flags locally (unshare --user → YES, max_user_namespaces → 111389). The only remaining userns risk is the host sysctl on the the orchestrator worker itself, which the topology block now prints.

Acceptance

One manual conformance run after change 1, whose published log (/api/packages/yuris/generic/conformance-logs/run-N-1/conformance.log) shows either a pass or a specific, named failure — and a second run that reports /nix mounted: <fs> and completes setup noticeably faster.

Conformance in CI (#7) needs two changes to the Forgejo runner's the orchestrator service definition. Neither lives in this repo, so this issue is the record of what to change and why. Current definition, the relevant part: ```yaml labels: - ubuntu-latest:docker://catthehacker/ubuntu:act-22.04 - ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04 container: docker_host: unix:///var/run/docker.sock options: --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN ``` ## 1. Persist /nix across runs This is the expensive one. The job image has no nix, so every run installs it and re-downloads ~2 GB of closure (JDK 25, bob's FHS env, the Defold engine, node, esbuild, chromium) only to throw it away. That is most of the ~5 minutes before run #10 died, and it makes disk exhaustion a live suspect. ```yaml container: options: >- --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN --volume forgejo_nix:/nix ``` First run populates the volume; later runs find a warm store. `conformance.yml` already detects this case — it looks for `/nix/var/nix/profiles/default/bin/nix`, not just `nix` on PATH, because the store persists while the per-user profile does not. Watch for: the volume grows without bound. A periodic `nix-collect-garbage -d`, or just deleting the volume when it gets large, is the whole maintenance story. ## 2. Consider a nix-native label An alternative to installing nix on every cold run: ```yaml labels: - nix:docker://nixos/nix:latest ``` Then `runs-on: nix` and the install step becomes a no-op. Note two consequences before doing it: - `nixos/nix` ships **no docker CLI**, which `tools/conformance.sh` needs to start the server. Already handled: `pkgs.docker-client` is in the `.#conformance` shell, so the CLI comes from the flake rather than the image. - It needs `--extra-experimental-features "nix-command flakes"` and `sandbox = false` (a nested nix sandbox inside a job container is unreliable), which is what the current `nix.conf` writing already sets up. This is optional if change 1 lands — a warm store makes the installer cheap. Worth it mainly to stop depending on `catthehacker`'s contents at all. ## Not the problem For the record, so it is not re-litigated: **user namespaces are fine**. `--cap-add SYS_ADMIN` plus unconfined seccomp and apparmor is what bubblewrap needs, confirmed by reproducing the flags locally (`unshare --user` → YES, `max_user_namespaces` → 111389). The only remaining userns risk is the host sysctl on the the orchestrator worker itself, which the topology block now prints. ## Acceptance One manual `conformance` run after change 1, whose published log (`/api/packages/yuris/generic/conformance-logs/run-N-1/conformance.log`) shows either a pass or a specific, named failure — and a second run that reports `/nix mounted: <fs>` and completes setup noticeably faster.

yuris referenced this issue

2026-08-19 12:36:25 +00:00

Conformance in CI: works locally, blocked on runner environment #7

yuris commented

2026-08-19 12:36:25 +00:00

Author

Owner

Docker is no longer used by conformance (see #7), which changes this issue: the job no longer needs the docker socket at all, so the image is free to be whatever is cheapest for nix.

That removes the objection to a nix-native label. Revised recommendation:

labels:
  - nix:docker://nixos/nix:latest
container:
  options: >-
    --security-opt seccomp=unconfined --security-opt apparmor=unconfined
    --cap-add SYS_ADMIN --volume forgejo_nix:/nix

Then runs-on: nix and the install step is a no-op. The two earlier caveats are both resolved:

The missing docker CLI in nixos/nix no longer matters — nothing shells out to docker.
The socket mount is not needed for this job either.

The --volume forgejo_nix:/nix half is still the one that matters most: without it every cold run re-downloads the whole closure (JDK, bob, the Defold engine, chromium, and now the 56 MB SpacetimeDB tarball). conformance.yml already detects a warm store via /nix/var/nix/profiles/default/bin/nix and the topology block reports /nix mounted: so the change is verifiable from the published log.

Watch for: the volume grows without bound. Periodic nix-collect-garbage -d, or deleting the volume when it gets large, is the whole maintenance story.

Docker is no longer used by conformance (see #7), which changes this issue: **the job no longer needs the docker socket at all**, so the image is free to be whatever is cheapest for nix. That removes the objection to a nix-native label. Revised recommendation: ```yaml labels: - nix:docker://nixos/nix:latest container: options: >- --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN --volume forgejo_nix:/nix ``` Then `runs-on: nix` and the install step is a no-op. The two earlier caveats are both resolved: - The missing docker CLI in `nixos/nix` no longer matters — nothing shells out to docker. - The socket mount is not needed for this job either. The `--volume forgejo_nix:/nix` half is still the one that matters most: without it every cold run re-downloads the whole closure (JDK, bob, the Defold engine, chromium, and now the 56 MB SpacetimeDB tarball). `conformance.yml` already detects a warm store via `/nix/var/nix/profiles/default/bin/nix` and the topology block reports `/nix mounted:` so the change is verifiable from the published log. Watch for: the volume grows without bound. Periodic `nix-collect-garbage -d`, or deleting the volume when it gets large, is the whole maintenance story.

yuris commented

2026-08-19 12:48:48 +00:00

Author

Owner

Tested it rather than guessed: a nix-native label works, and it already found a bug

Ran the full native conformance suite inside a bare nixos/nix container, using this runner's exact security options:

docker run --rm --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
  --cap-add SYS_ADMIN -v forgejo_nix:/nix -v repo:/src:ro nixos/nix:latest \
  bash -c 'nix develop .#conformance --command bash tools/conformance.sh'

[conformance] PASS (identity 0xc200f147…2dd4 persisted across a process restart)

So the answer is yes — and the numbers are the argument:

Run	Wall time
Cold (empty `/nix` volume)	335s
Warm (same volume reused)	30s

The warm store is worth ~11x. Store size after one full run: 4.7 GB, which is the number to watch on the the orchestrator volume.

It caught a latent bug the workstation was hiding

The first container run failed at engine exec:

./spacetimedbdefold.x86_64: error while loading shared libraries:
libstdc++.so.6: cannot open shared object file

defold-patch set an rpath covering libgcc, zlib, openal, GL and X11 — but not libstdc++, which is in the engine's direct DT_NEEDED set. It passed locally for a stupid reason:

$ env -i ldd spacetimedbdefold.x86_64 | grep libstdc
        libstdc++.so.6 => not found

It only resolved because the workstation's ambient LD_LIBRARY_PATH happened to contain a gcc-lib path leaked from an unrelated shell. Exactly the "works on my machine" class this was supposed to prevent, and nix did not catch it, because the leak came from the environment rather than the closure. Fixed by adding stdenv.cc.cc.lib to the rpath; env -i ldd now reports zero missing libraries.

That is a concrete argument for the nix-native image beyond speed: a minimal container has no ambient library path to accidentally rescue a broken closure.

Which image

nixos/nix has nix 2.35.2, git, bash, curl and tar — but no node, and actions/checkout@v4 is a JavaScript action, so it cannot run there. Two ways out.

Recommended — a purpose-built Forgejo runner image that bundles nix (flakes on) and node, so nothing else changes:

runner:
  labels:
    - ubuntu-latest:docker://catthehacker/ubuntu:act-22.04
    - nix-latest:docker://ghcr.io/joschi/forgejo-nix:latest
container:
  docker_host: unix:///var/run/docker.sock
  options: >-
    --security-opt seccomp=unconfined --security-opt apparmor=unconfined
    --cap-add SYS_ADMIN --volume forgejo_nix:/nix

It ships bash, busybox, curl, git + LFS, node, cachix, attic and the Forgejo CLI, and the images are cosign-signed (joschi/forgejo-runner-nix-containers, derived from codeberg.org/nix-actions/container). Then runs-on: nix-latest in both workflows and the install-nix step self-skips.

Alternative — bare nixos/nix (proven above) requires dropping JS actions, i.e. replacing actions/checkout@v4 with a plain git clone. Not done here: it only becomes necessary if a nodeless image is chosen, and the current step works on both catthehacker and the joschi image.

Worth noting the mainstream pattern is the opposite trade — keep an act image and install nix per run via cachix/install-nix-action (example). That is effectively what the hand-rolled installer step does today, and it pays the download cost every cold run, which is the thing the /nix volume fixes.

Also note

docker_host and the socket mount are now only needed by other repos' jobs — conformance no longer touches docker at all (#7).

Revised acceptance

Add the label and the /nix volume.
Flip both workflows to runs-on: nix-latest.
One manual conformance run; the published log should show /nix mounted: <fs> and complete in well under a minute on the second run.

## Tested it rather than guessed: a nix-native label works, and it already found a bug Ran the full native conformance suite inside a bare `nixos/nix` container, using this runner's exact security options: ``` docker run --rm --security-opt seccomp=unconfined --security-opt apparmor=unconfined \ --cap-add SYS_ADMIN -v forgejo_nix:/nix -v repo:/src:ro nixos/nix:latest \ bash -c 'nix develop .#conformance --command bash tools/conformance.sh' [conformance] PASS (identity 0xc200f147…2dd4 persisted across a process restart) ``` So the answer is yes — and the numbers are the argument: | Run | Wall time | |---|---| | Cold (empty `/nix` volume) | **335s** | | Warm (same volume reused) | **30s** | The warm store is worth ~11x. Store size after one full run: **4.7 GB**, which is the number to watch on the the orchestrator volume. ### It caught a latent bug the workstation was hiding The first container run failed at engine exec: ``` ./spacetimedbdefold.x86_64: error while loading shared libraries: libstdc++.so.6: cannot open shared object file ``` `defold-patch` set an rpath covering libgcc, zlib, openal, GL and X11 — but not libstdc++, which is in the engine's direct `DT_NEEDED` set. It passed locally for a stupid reason: ``` $ env -i ldd spacetimedbdefold.x86_64 | grep libstdc libstdc++.so.6 => not found ``` It only resolved because the workstation's ambient `LD_LIBRARY_PATH` happened to contain a `gcc-lib` path leaked from an unrelated shell. Exactly the "works on my machine" class this was supposed to prevent, and nix did *not* catch it, because the leak came from the environment rather than the closure. Fixed by adding `stdenv.cc.cc.lib` to the rpath; `env -i ldd` now reports zero missing libraries. That is a concrete argument for the nix-native image beyond speed: a minimal container has no ambient library path to accidentally rescue a broken closure. ### Which image `nixos/nix` has nix 2.35.2, git, bash, curl and tar — but **no node**, and `actions/checkout@v4` is a JavaScript action, so it cannot run there. Two ways out. **Recommended** — a purpose-built Forgejo runner image that bundles nix (flakes on) *and* node, so nothing else changes: ```yaml runner: labels: - ubuntu-latest:docker://catthehacker/ubuntu:act-22.04 - nix-latest:docker://ghcr.io/joschi/forgejo-nix:latest container: docker_host: unix:///var/run/docker.sock options: >- --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN --volume forgejo_nix:/nix ``` It ships bash, busybox, curl, git + LFS, node, cachix, attic and the Forgejo CLI, and the images are cosign-signed ([joschi/forgejo-runner-nix-containers](https://github.com/joschi/forgejo-runner-nix-containers), derived from [codeberg.org/nix-actions/container](https://codeberg.org/nix-actions/container)). Then `runs-on: nix-latest` in both workflows and the install-nix step self-skips. **Alternative** — bare `nixos/nix` (proven above) requires dropping JS actions, i.e. replacing `actions/checkout@v4` with a plain `git clone`. Not done here: it only becomes necessary if a nodeless image is chosen, and the current step works on both `catthehacker` and the joschi image. Worth noting the mainstream pattern is the opposite trade — keep an act image and install nix per run via `cachix/install-nix-action` ([example](https://philippeloctaux.com/blog/run-nix-on-forgejo-actions-on-nixos/)). That is effectively what the hand-rolled installer step does today, and it pays the download cost every cold run, which is the thing the `/nix` volume fixes. ### Also note `docker_host` and the socket mount are now only needed by *other* repos' jobs — conformance no longer touches docker at all (#7). ### Revised acceptance 1. Add the label and the `/nix` volume. 2. Flip both workflows to `runs-on: nix-latest`. 3. One manual conformance run; the published log should show `/nix mounted: <fs>` and complete in well under a minute on the second run.

yuris commented

2026-08-19 12:52:55 +00:00

Author

Owner

Two runners: dedicate the nix label to one of them

There are two runners, each pinned to a different swarm node (node.hostname == runner-node-a / runner-node-b). That makes dedicating one to nix the right call, for a reason that is specific to this setup:

Docker volumes are node-local. The warm /nix store — measured at 4.7 GB after one full conformance run — exists only on the node whose daemon created it. If both runners advertised a nix label, jobs would alternate between nodes, each node would need its own 4.7 GB copy, and every job landing on the "other" node would be a cold 335s run instead of a warm 30s one. Confining the label to one runner keeps the store hot and the disk cost on one node.

Change runner-2 only

Keep the ubuntu labels so general capacity is not halved — a nix-latest job can only go to runner-2 because it is the only runner advertising that label, while everything else still schedules on either.

echo 'runner:' > /config.yaml &&
echo '  labels:' >> /config.yaml &&
echo '    - ubuntu-latest:docker://catthehacker/ubuntu:act-22.04' >> /config.yaml &&
echo '    - ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04' >> /config.yaml &&
echo '    - nix-latest:docker://ghcr.io/joschi/forgejo-nix:latest' >> /config.yaml &&
echo 'container:' >> /config.yaml &&
echo '  docker_host: unix:///var/run/docker.sock' >> /config.yaml &&
echo '  options: --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN --volume forgejo_nix_store:/nix' >> /config.yaml &&

Two lines changed: one added label, one added --volume. runner-1 stays exactly as it is.

The volume needs no volumes: entry in the compose file — docker run -v forgejo_nix_store:/nix through the mounted socket creates it on that node on first use. Declaring it in compose would subject it to the orchestrator's project-name prefixing, which is a different name than the one in container.options.

Then in both workflows:

-    runs-on: ubuntu-latest
+    runs-on: nix-latest

Mechanics worth knowing before making the change

Labels come from the config file, not registration. The reference says: "If it's empty when executing the daemon, it will use labels in the .runner file." This config declares labels explicitly, so a service restart is sufficient — no re-registration. If the new label does not appear at /admin/actions/runners after the restart, delete /data/.runner from forgejo_runner_data_2 to force re-registration; the if [ ! -f /data/.runner ] guard will then re-run it.
Images are never re-pulled. The docs are explicit: starting a container is the equivalent of docker run <image>, so "container images, once downloaded, are never updated". Pin by digest for reproducibility, or set container.force_pull: true if you would rather track :latest.
container.options applies to every job on that runner, so ubuntu jobs on runner-2 will also see /nix mounted. Harmless, and it has a useful side effect: conformance.yml's install step probes /nix/var/nix/profiles/default/bin/nix, so even an ubuntu-latest job landing on runner-2 finds the warm store instead of running the installer.
The alternative — letting the workflow request the mount via jobs.<id>.container.volumes — needs container.valid_volumes allowlisted on the runner (it defaults to [], meaning no workflow-specified volumes are permitted). Runner-side options is the smaller change.
runner.capacity defaults to 1, so conformance will queue behind any other job on runner-2 rather than run beside it.

One inconsistency to check

The two services point at different instance addresses — runner-1 at http://<forgejo-host>:5858, runner-2 at http://<forgejo-host>:5858. If those are the same Forgejo reached over different node-local addresses, nothing to do. If they are genuinely two instances, label routing will not behave as described, because the job would never be offered to runner-2 at all.

## Two runners: dedicate the nix label to one of them There are two runners, each pinned to a different swarm node (`node.hostname == runner-node-a` / `runner-node-b`). That makes dedicating one to nix the right call, for a reason that is specific to this setup: **Docker volumes are node-local.** The warm `/nix` store — measured at 4.7 GB after one full conformance run — exists only on the node whose daemon created it. If both runners advertised a nix label, jobs would alternate between nodes, each node would need its own 4.7 GB copy, and every job landing on the "other" node would be a cold 335s run instead of a warm 30s one. Confining the label to one runner keeps the store hot and the disk cost on one node. ### Change runner-2 only Keep the ubuntu labels so general capacity is not halved — a `nix-latest` job can only go to runner-2 because it is the only runner advertising that label, while everything else still schedules on either. ```sh echo 'runner:' > /config.yaml && echo ' labels:' >> /config.yaml && echo ' - ubuntu-latest:docker://catthehacker/ubuntu:act-22.04' >> /config.yaml && echo ' - ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04' >> /config.yaml && echo ' - nix-latest:docker://ghcr.io/joschi/forgejo-nix:latest' >> /config.yaml && echo 'container:' >> /config.yaml && echo ' docker_host: unix:///var/run/docker.sock' >> /config.yaml && echo ' options: --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add SYS_ADMIN --volume forgejo_nix_store:/nix' >> /config.yaml && ``` Two lines changed: one added label, one added `--volume`. runner-1 stays exactly as it is. The volume needs no `volumes:` entry in the compose file — `docker run -v forgejo_nix_store:/nix` through the mounted socket creates it on that node on first use. Declaring it in compose would subject it to the orchestrator's project-name prefixing, which is a different name than the one in `container.options`. Then in both workflows: ```diff - runs-on: ubuntu-latest + runs-on: nix-latest ``` ### Mechanics worth knowing before making the change - **Labels come from the config file, not registration.** The reference says: *"If it's empty when executing the `daemon`, it will use labels in the `.runner` file."* This config declares labels explicitly, so a service restart is sufficient — no re-registration. If the new label does not appear at `/admin/actions/runners` after the restart, delete `/data/.runner` from `forgejo_runner_data_2` to force re-registration; the `if [ ! -f /data/.runner ]` guard will then re-run it. - **Images are never re-pulled.** The docs are explicit: starting a container is the equivalent of `docker run <image>`, so *"container images, once downloaded, are never updated"*. Pin by digest for reproducibility, or set `container.force_pull: true` if you would rather track `:latest`. - **`container.options` applies to every job on that runner**, so ubuntu jobs on runner-2 will also see `/nix` mounted. Harmless, and it has a useful side effect: `conformance.yml`'s install step probes `/nix/var/nix/profiles/default/bin/nix`, so even an `ubuntu-latest` job landing on runner-2 finds the warm store instead of running the installer. - The alternative — letting the *workflow* request the mount via `jobs.<id>.container.volumes` — needs `container.valid_volumes` allowlisted on the runner (it defaults to `[]`, meaning no workflow-specified volumes are permitted). Runner-side `options` is the smaller change. - `runner.capacity` defaults to 1, so conformance will queue behind any other job on runner-2 rather than run beside it. ### One inconsistency to check The two services point at different instance addresses — runner-1 at `http://<forgejo-host>:5858`, runner-2 at `http://<forgejo-host>:5858`. If those are the same Forgejo reached over different node-local addresses, nothing to do. If they are genuinely two instances, label routing will not behave as described, because the job would never be offered to runner-2 at all.

yuris commented

2026-08-19 13:13:47 +00:00

Author

Owner

First run on the nix runner failed at container start — it is the image, not the workflow

Run #23: 88 seconds, no logs, no published artifact, no self-report comment. Both final steps are if: always(), so their silence proves no step ran at all — the job container never came up.

Cause, reproduced locally:

$ docker image inspect ghcr.io/joschi/forgejo-nix:latest --format '{{.Config.Entrypoint}}'
[/bin/bash]

$ docker run --rm ghcr.io/joschi/forgejo-nix:latest tail -f /dev/null
/sbin/tail: /sbin/tail: cannot execute binary file

The runner keeps a job container alive with a command like tail -f /dev/null. With ENTRYPOINT ["/bin/bash"] that becomes /bin/bash tail -f /dev/null, so bash tries to interpret the tail binary as a script and the container exits immediately. Images that work with act leave Entrypoint null and put the shell in Cmd:

Image	Entrypoint	Cmd	node	Works
`catthehacker/ubuntu:act-22.04`	null	`/bin/bash`	yes	yes
`nixos/nix:latest`	null	bash	no	yes
`ghcr.io/joschi/forgejo-nix`	`/bin/bash`	—	yes	no
`codeberg.org/nix-actions/container`	—	—	—	only an aarch64 tag is published

Switch to official nixos/nix

It leaves Entrypoint null and the full suite already passes inside it. What it lacks is node, so actions/checkout@v4 — a JavaScript action — cannot run there. Fixed in a2cdbb2: checkout is a plain git clone, and the report step borrows jq from the store since the image ships neither jq nor python3 (escaping re-verified against quotes and backslashes).

The workflow now depends on nothing but git and nix, which keeps the image choice free and removes a third-party image from the CI path.

Label change on runner-2:

echo '    - nix-latest:docker://nixos/nix@sha256:7a007c766426c1877758ddc5cb87a965ac131fc78c582ce0083d922d51ae945c' >> /config.yaml &&

Pinned by digest because the runner never re-pulls an image once downloaded.

Delete the volume first

The failed container was created before it died, so forgejo_nix_store on runner-node-b may already have been seeded with the joschi image's /nix. Mounting that into nixos/nix shadows its own store with a foreign one whose default profile sits elsewhere — a confusing half-broken state. On runner-node-b, before redeploying:

docker volume rm forgejo_nix_store

It is repopulated from the image on the next run.

## First run on the nix runner failed at container start — it is the image, not the workflow Run #23: **88 seconds, no logs, no published artifact, no self-report comment**. Both final steps are `if: always()`, so their silence proves no step ran at all — the job container never came up. Cause, reproduced locally: ``` $ docker image inspect ghcr.io/joschi/forgejo-nix:latest --format '{{.Config.Entrypoint}}' [/bin/bash] $ docker run --rm ghcr.io/joschi/forgejo-nix:latest tail -f /dev/null /sbin/tail: /sbin/tail: cannot execute binary file ``` The runner keeps a job container alive with a command like `tail -f /dev/null`. With `ENTRYPOINT ["/bin/bash"]` that becomes `/bin/bash tail -f /dev/null`, so bash tries to *interpret the tail binary as a script* and the container exits immediately. Images that work with act leave `Entrypoint` null and put the shell in `Cmd`: | Image | Entrypoint | Cmd | node | Works | |---|---|---|---|---| | `catthehacker/ubuntu:act-22.04` | null | `/bin/bash` | yes | yes | | `nixos/nix:latest` | null | bash | **no** | yes | | `ghcr.io/joschi/forgejo-nix` | **`/bin/bash`** | — | yes | **no** | | `codeberg.org/nix-actions/container` | — | — | — | only an aarch64 tag is published | ### Switch to official nixos/nix It leaves `Entrypoint` null and the full suite already passes inside it. What it lacks is node, so `actions/checkout@v4` — a JavaScript action — cannot run there. Fixed in a2cdbb2: checkout is a plain `git clone`, and the report step borrows `jq` from the store since the image ships neither jq nor python3 (escaping re-verified against quotes and backslashes). The workflow now depends on nothing but git and nix, which keeps the image choice free and removes a third-party image from the CI path. Label change on runner-2: ```sh echo ' - nix-latest:docker://nixos/nix@sha256:7a007c766426c1877758ddc5cb87a965ac131fc78c582ce0083d922d51ae945c' >> /config.yaml && ``` Pinned by digest because the runner never re-pulls an image once downloaded. ### Delete the volume first The failed container was created before it died, so `forgejo_nix_store` on runner-node-b may already have been seeded with the *joschi* image's `/nix`. Mounting that into `nixos/nix` shadows its own store with a foreign one whose default profile sits elsewhere — a confusing half-broken state. On runner-node-b, before redeploying: ```sh docker volume rm forgejo_nix_store ``` It is repopulated from the image on the next run.

yuris referenced this issue

2026-08-19 13:37:18 +00:00

Conformance in CI: works locally, blocked on runner environment #7

yuris commented

2026-08-19 13:37:34 +00:00

Author

Owner

Acceptance met — conformance is green in CI (#7 closed)

Runs #28 and #29 both pass on the nix-latest runner. The topology block confirms the warm store, which was the whole point of this issue:

/nix mounted: /dev/sda1 /nix ext4 rw,relatime,discard,errors=remount-ro
nix store paths: 5498
free disk /nix: 74G  36G  35G  51%

Run	Wall time
#25 (cold, before the volume worked)	2m20s to failure
#28 (first green)	~2m
#29 (warm)	~100s

Two corrections to what this issue previously recommended, both now settled:

The image: not ghcr.io/joschi/forgejo-nix — its ENTRYPOINT ["/bin/bash"] kills the job container before any step runs. Official nixos/nix pinned by digest.
valid_volumes is mandatory, not optional. With it unset, sanitizeConfig sets hostConfig.Binds = []string{} and the --volume in container.options is discarded silently — no error, no warm store, just permanently cold runs. This was the reason docker volume ls showed nothing after run #23.

The working configuration is recorded in the closing comment on #7.

Left open here

Disk: the store is 4.7 GB and grows. nix-collect-garbage -d inside a job, or deleting forgejo_nix_store on runner-node-b when it gets large, is the whole maintenance story. Nothing is scheduled to do it.

Optional: CONFORMANCE_LOG_TOKEN as a repo secret would let the full log upload to the package registry. The job token cannot — generic packages are owner-scoped, the token is repo-scoped — and the 120-line tail in the self-report comment has been sufficient for every failure so far.

## Acceptance met — conformance is green in CI (#7 closed) Runs #28 and #29 both pass on the `nix-latest` runner. The topology block confirms the warm store, which was the whole point of this issue: ``` /nix mounted: /dev/sda1 /nix ext4 rw,relatime,discard,errors=remount-ro nix store paths: 5498 free disk /nix: 74G 36G 35G 51% ``` | Run | Wall time | |---|---| | #25 (cold, before the volume worked) | 2m20s to failure | | #28 (first green) | ~2m | | #29 (warm) | **~100s** | Two corrections to what this issue previously recommended, both now settled: - **The image**: not `ghcr.io/joschi/forgejo-nix` — its `ENTRYPOINT ["/bin/bash"]` kills the job container before any step runs. Official `nixos/nix` pinned by digest. - **`valid_volumes` is mandatory, not optional.** With it unset, `sanitizeConfig` sets `hostConfig.Binds = []string{}` and the `--volume` in `container.options` is discarded silently — no error, no warm store, just permanently cold runs. This was the reason `docker volume ls` showed nothing after run #23. The working configuration is recorded in the closing comment on #7. ### Left open here Disk: the store is 4.7 GB and grows. `nix-collect-garbage -d` inside a job, or deleting `forgejo_nix_store` on runner-node-b when it gets large, is the whole maintenance story. Nothing is scheduled to do it. Optional: `CONFORMANCE_LOG_TOKEN` as a repo secret would let the full log upload to the package registry. The job token cannot — generic packages are owner-scoped, the token is repo-scoped — and the 120-line tail in the self-report comment has been sufficient for every failure so far.

yuris referenced this issue

2026-08-19 13:56:54 +00:00

Conformance in CI: works locally, blocked on runner environment #7

yuris commented

2026-08-19 14:51:29 +00:00

Author

Owner

Both changes landed and are proven in production. Run #38's topology step:

in-container: yes
cpus: 4
kernel: 6.12.95+deb13-amd64
/nix mounted: /dev/sda1 /nix ext4 rw,relatime,discard,errors=remount-ro 0 0
free disk /: overlay 74G 36G 35G 51% /
nix store paths: 5512

Change 1 (persist /nix) — /nix mounted: reports a real ext4 device rather than the no (image layer) fallback the probe prints otherwise, and the store holds 5512 paths that no run in this job downloaded. That is the acceptance criterion.
Change 2 (nix-native label) — also done, and it turned out to be more than optional. conformance.yml now uses runs-on: nix-latest, and the install step short-circuits on /nix/var/nix/profiles/default/bin/nix exactly as the description anticipated: the store persists while the per-user profile does not.

Setup is now ~2s of the run. The 100s total is conformance actually doing work — 51s native, 47s HTML5 — not environment assembly.

Amendments worth recording

Two things the description did not predict, both settled in #7 and now encoded in conformance.yml:

actions/checkout@v4 had to go. It is a JavaScript action and needs node in the job image; nixos/nix ships none. The one nix+node image available sets ENTRYPOINT ["/bin/bash"], which mangles the runner's keep-alive command so the container dies before any step runs — an 88s failure with no logs whatsoever. Plain git clone with github.token depends on nothing but git and keeps the image choice free.
Docker left the picture entirely. The description budgeted for pkgs.docker-client because the harness needed a container for the server. It does not any more: the server and CLI are pinned release binaries from the flake, so the docker socket, the sibling-container port topology and user namespaces are all off the critical path. The --cap-add SYS_ADMIN and unconfined seccomp/apparmor options are still in the runner definition and still fine, but conformance no longer relies on them.

The "not the problem" section holds: user namespaces were never the fault.

Maintenance, as predicted

The forgejo_nix volume grows without bound. 36G of 74G used at run #38. nix-collect-garbage -d inside a job, or deleting the volume when it gets uncomfortable, remains the whole story — worth a look if a run ever fails on disk rather than on tests.

Closing.

Both changes landed and are proven in production. Run [#38](https://git.nokopia.com/yuris/spacetimedb-defold/actions/runs/38)'s topology step: ``` in-container: yes cpus: 4 kernel: 6.12.95+deb13-amd64 /nix mounted: /dev/sda1 /nix ext4 rw,relatime,discard,errors=remount-ro 0 0 free disk /: overlay 74G 36G 35G 51% / nix store paths: 5512 ``` - **Change 1 (persist /nix)** — `/nix mounted:` reports a real ext4 device rather than the `no (image layer)` fallback the probe prints otherwise, and the store holds 5512 paths that no run in this job downloaded. That is the acceptance criterion. - **Change 2 (nix-native label)** — also done, and it turned out to be more than optional. `conformance.yml` now uses `runs-on: nix-latest`, and the install step short-circuits on `/nix/var/nix/profiles/default/bin/nix` exactly as the description anticipated: the store persists while the per-user profile does not. Setup is now ~2s of the run. The 100s total is conformance actually doing work — 51s native, 47s HTML5 — not environment assembly. ## Amendments worth recording Two things the description did not predict, both settled in #7 and now encoded in `conformance.yml`: - **`actions/checkout@v4` had to go.** It is a JavaScript action and needs node in the job image; `nixos/nix` ships none. The one nix+node image available sets `ENTRYPOINT ["/bin/bash"]`, which mangles the runner's keep-alive command so the container dies before any step runs — an 88s failure with no logs whatsoever. Plain `git clone` with `github.token` depends on nothing but git and keeps the image choice free. - **Docker left the picture entirely.** The description budgeted for `pkgs.docker-client` because the harness needed a container for the server. It does not any more: the server and CLI are pinned release binaries from the flake, so the docker socket, the sibling-container port topology and user namespaces are all off the critical path. The `--cap-add SYS_ADMIN` and unconfined seccomp/apparmor options are still in the runner definition and still fine, but conformance no longer relies on them. The "not the problem" section holds: user namespaces were never the fault. ## Maintenance, as predicted The `forgejo_nix` volume grows without bound. 36G of 74G used at run #38. `nix-collect-garbage -d` inside a job, or deleting the volume when it gets uncomfortable, remains the whole story — worth a look if a run ever fails on disk rather than on tests. Closing.

yuris closed this issue

2026-08-19 14:51:29 +00:00