HTML5 conformance in CI: prove Chromium runs in the job container #16
Labels
No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
yuris/spacetimedb-defold#16
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Native conformance is stuck on runner environment (#7): it needs to exec Defold's generic Linux binary, which means either bubblewrap (user namespaces, commonly denied in an unprivileged job container) or patchelf, and five runs died without a readable log because this Forgejo serves none over the API.
The HTML5 harness added in #8 avoids most of that, so it is a plausible route to some engine-level coverage in CI where the native tier cannot get any:
defold-runanddefold-patchare not involved at all.--no-sandbox; nothing needsunshare.dev/tools/html5-conformance.mjshas zero npm dependencies — node's own globalWebSocketdrives devtools.What it still shares with #7:
bob(so the JDK), network access tobuild.defold.comfor the native extension, and a docker socket for the server plus the phase-2 restart.Answered: Chromium runs here
Run #163 is green in the job container. Full six-phase smoke, both loads, identity persisted across the reload,
subprotocol=v1.json.spacetimedband101on every handshake.Container: 4 cpus, kernel 6.12.95, 35G free on a persisted
/nix(5747 store paths warm). The wasm build plus Chromium is not close to a disk problem. Whole job, both tiers, ~2.5 minutes.Run #162 was the same code passing every phase and then exiting 1 on
cdp socket error. Not an environment failure:finish()kills chromium, the devtools socket dies, theerrorlistener re-entersfinish()while the first call is still awaitingrm(), and the loser's exit code is the one that escapes. A warm page cache hides it locally. Fixed by a re-entry flag in #78.Bulk apply is ~3x slower in the container than locally: 294ms per 10k rows against 117ms, same shape (60% frame overhead, 22% decode). Expected on a shared 4-cpu runner; noting it so nobody reads a future regression into it.
Work
Blocked on #15— not blocked.dev/tools/html5-conformance.shunpacks the extension bob resolved and appliespatches/itself, so the run carries the subprotocol fix with no fork and no upstream release. #15 tracks getting the fix upstream so consumers stop needing the patch; it does not gate this job.if: always()— added in #78, scoped to the HTML5 step rather than the whole job, and marked for deletion when this issue closes. The step already existed inconformance.yml; what was missing was any way to see its result, since the API serves no logs.continue-on-error— split out to #81. Runs 716 and 717 disagree on the same commit, so steadiness is a separate question from whether the container works, and #81 owns it along with the self-report.Acceptance
Met by run #163. Met by run #163 and every run since. Closed; the flake that remains is #81.
Tooling blocker is gone — the HTML5 tier now runs as one command
This issue assumed HTML5 conformance needed a maintained fork of extension-websocket. It does not.
dev/tools/html5-conformance.shnow vendors the extension itself:bob resolvefetches the archive, the script unpacks it, appliespatches/extension-websocket-html5-subprotocol.patch, and builds a scratch project carrying the extension locally with the remote dependency stripped (keeping it puts twowebsocketextensions in one build).[handshake] -> subprotocol=v1.json.spacetimedbconfirms the patch is in effect on the wire.Before this, the script's header instructed you to set
WEBSOCKET_DEPENDENCYand nothing in the script read that variable, so every run built against unpatched 4.2.3 and timed out at the handshake after 90s. The tier had only ever passed by hand-building a tree in/tmp, which is not a test anyone else can run. Two bugs surfaced fixing it:unzipandpatchresolved from an ambient user profile rather than the dev shell — the same leak class that hid the missing libstdc++ from the engine's rpath. Both are declared in the conformance shell now.unzip -l | grep -qalways reported false:grep -qexits on the first match,unziptakes SIGPIPE, andpipefailfails the pipeline. Second time this SIGPIPE class has bitten in this repo.What is left for this issue
Only the CI decision, not the tooling. The wasm build calls out to Defold's extender service, so this job is slower and network-dependent in a way the native tier is not — worth deciding whether it belongs in the weekly conformance run or a separate manual workflow. Note this does not close #15: consumers still need a patched extension for their own HTML5 builds; only our test path is self-sufficient.
Unblocked, and the work is done — this issue was written against a runner situation that no longer exists.
What changed since it was filed
Every premise in the description is stale:
conformance.ymlnow runs on thenix-latestrunner and passes in ~100s with a warm/nixvolume. There is nocontinue-on-erroron it and no self-reporting, because the job is reliable. So HTML5 is no longer "the tier native conformance cannot reach"; it is simply a second tier worth having.flake.nix:112-113), anddev-server.sh restartbounces the process directly (html5-conformance.mjs:134-140).pkgs.chromiumin theconformanceshell withexport CHROMIUM=…(flake.nix:206,215). The same warm-store mechanism that fixed the native tier covers it.html5-conformance.sh:44-61unzips the extension bob fetched, appliespatches/extension-websocket-html5-subprotocol.patch, and builds a scratch project carrying it locally. No fork, no upstream release, no manual setup. See #15 for why that also removes the fork-hosting question.What landed
The HTML5 tier is a step in the existing conformance job, not a second job. It needs a byte-identical checkout / nix-install / topology preamble, and Forgejo cannot share steps without a composite action — so a second job would have meant duplicating ~50 lines of the hardest-won configuration in this repo.
continue-on-errorstays until it has been green a few weeks. The tier is unproven in this container and nowhere else, and a red X on it should not retract the native verdict in the same job.Local run, exit 0 in 36.7s — the handshake lines this issue asked for:
Note
subprotocol=v1.json.spacetimedbrather than<ABSENT>— the vendored patch working, with no fork anywhere in the picture.Deliberately not done
if: always(). #7 settled on that shape because this Forgejo serves no logs over the API and a blank failure was unreadable. That was a workaround for an unreliable job; the web UI shows the log fine. Adding a reporting harness to a job that is now trustworthy would re-import the complexity #7 spent ten runs removing.workflow_dispatchfor HTML5 alone. The existing dispatch runs both tiers. Splitting them costs the duplicated preamble for the ability to skip 25s of wasm build.What is left
One manual
workflow_dispatchrun to confirm Chromium starts in the job container, then dropcontinue-on-error. If it cannot start there, the honest outcome is to delete the step and say so here — the acceptance criterion as written.HTML5 conformance in CI: the tier native conformance cannot reachto HTML5 conformance in CI: prove Chromium runs in the job containerAcceptance met: run #38 on
bfd2b14,workflow_dispatch, PASS. Chromium starts in the job container.Both tiers in one job, 100s total:
The handshake evidence this issue asked for, from the CI log:
subprotocol=v1.json.spacetimedbrather than<ABSENT>, against an unpatched upstream — the harness's own vendor-and-patch step, no fork involved. The open unknowns from the description are all closed: no Defold binary exec, no user namespaces, no docker, and the wasm build plus browser fit in the runner's disk without complaint.One thing worth watching, and why the flag stays
Reconnect needed four attempts in CI versus one locally:
The server takes materially longer to come back on the runner than on a workstation, so phase 2 spent ~13s of its backoff budget where locally it spends ~1s. It passed, and the backoff is doing precisely what it exists for — but the margin is thinner than the local run suggests, and it is the one part of this tier that could go red on a slow day without anything actually being broken.
So
continue-on-error: truestays for now. The unknown it was written to guard (does Chromium run here at all) is resolved, but a single green run does not establish that the reconnect phase is stable under runner load. Revisit after a few weekly runs: consistently green → drop the flag; intermittently red at attempt 5+ → the fix is the retry budget in the harness, not the workflow.Leaving this open until the flag comes off.
yuris referenced this issue2026-08-19 15:20:43 +00:00
CI history checked, and the API cannot settle this — here is why
This issue turns on "once a few weekly runs have been green". I went to count
them. The answer is more awkward than a count.
conformance.ymlhas 8 runs total: 6success, 2failure(Forgejo'sactions API serves no timestamps or per-step results on this endpoint, so that is
run-level status only). The workflow is
workflow_dispatchplus a weekly Mondaycron, so 8 runs is the whole history.
The problem: run-level
successdoes not mean the HTML5 step passed. That isthe entire effect of
continue-on-error: trueatconformance.yml:116— a redHTML5 tier is recorded as a footnote and the job still reports green. So the 6
successes are consistent with HTML5 having passed 6 times and with it having
failed all 6 and been swallowed. Nothing in the API distinguishes them, which is
the same limitation
conformance.yml:30-32already notes for logs: "Read the joblog in the web UI — the API serves none."
So the evidence this issue needs is only reachable from the web UI, by a human
opening those 8 job logs and reading whether the HTML5 step's verdict line says
[html5] PASS. That is not agent-reachable, which upgrades this from "a judgementon accumulated history" to "a judgement that first needs someone to go and read
6-8 job logs".
One new data point, and it argues for waiting
During the v1.1.0 release gate,
html5-conformance.shfailed once locally andthen passed four times in a row (28/29/28s against a 120s timeout). Recorded on
#52 with the caveat that I did not capture the frames, so I cannot attribute it.
I did rule out one thing worth ruling out explicitly: it is not #66's new
PHASE6 pushing the per-load budget. The passing runs sit at roughly 4x headroom,
and a timeout emits its own distinct message rather than a bare
FAIL.But a 1-in-5 local failure is exactly the kind of thing
continue-on-errorexiststo absorb, and it happened on the tier that is better understood locally than in
the job container. Dropping the flag now would convert an unexplained intermittent
into a red X on a workflow whose native half is described as reliable.
Suggested disposition
Keep the flag. Two concrete steps that would actually close this, in order:
line to
$GITHUB_STEP_SUMMARY, or fail a separate non-blocking step on theabsence of
[html5] PASS, so future runs leave a signal the API can seewithout anyone opening a log. Without this, the same manual log-reading is
required every time this question is revisited.
intermittent has either recurred with a frame capture or gone quiet across a
meaningful number of runs.
Step 1 is agent-doable and would make step 2 cheap. Say the word and I will open
it as a PR.
HTML5 tier, run #162: failure (
b28ac1e3af)Topology:
Tail:
HTML5 tier, run #163: success (
4ca7c9f1fa)Topology:
Tail:
HTML5 tier, run #167: success (
62b3e7784a)Topology:
Tail:
HTML5 tier, run #169: failure (
38f14c758b)Topology:
Tail: