HTML5 conformance is flaky in CI: same commit fails then passes #81
Labels
No labels
area:defold
area:docs
area:infra
area:protocol
area:testing
kind:bug
kind:chore
phase:breadth
phase:core
phase:hardening
phase:slice
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
yuris/spacetimedb-defold#81
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Split out of #16, which asked whether Chromium runs in the job container. It does. This issue owns the separate question of whether the tier is steady enough to gate anything.
Evidence, same commit
80f5d86onci/close-16:No overlap with another conformance run (714 finished 11 minutes earlier), so this is not two servers fighting over a port. The failing run lasted about as long as a passing one, so it got deep into the sequence rather than dying at setup.
What the failure actually was is unknown, because #80 removed the self-report in the same change that made the step gate PRs. That was the wrong order and is being reverted: the report stays, retargeted here, and
continue-on-errorstays until this issue closes.Suspects
Nothing confirmed. In rough order of plausibility given the tier drives a browser:
sys.saveand must survive a reload; a slow container could reload before IndexedDB has flushed, and the identity comparison then fails legitimately.Work
continue-on-error, this report step and theissues: writegrant once the tier is steadyAcceptance
Ten consecutive green runs of the HTML5 tier, or a named and fixed root cause.
HTML5 tier, run #174: success (
01fdcd24d0)Topology:
Tail:
HTML5 tier, run #176: success (
e0b337a55d)Topology:
Tail:
HTML5 tier, run #178: success (
5c0d494975)Topology:
Tail:
HTML5 tier, run #180: success (
19e5325a6b)Topology:
Tail:
HTML5 tier, run #182: failure (
0d9bcfbe9a)Topology:
Tail:
HTML5 tier, run #184: failure (
df6a963bb9)Topology:
Tail:
HTML5 tier, run #187: failure (
b543ee236e)Topology:
Tail:
HTML5 tier, run #189: failure (
6a47e5487e)Topology:
Tail:
HTML5 tier, run #191: failure (
ab1d9a68c6)Topology:
Tail:
HTML5 tier, run #203: skipped (
841dd7ed45)Topology:
Tail:
HTML5 tier, run #207: skipped (
4002f9f18b)Topology:
Tail:
HTML5 tier, run #211: skipped (
689f10ba1a)Topology:
Tail:
HTML5 tier, run #213: failure (
5da10dba6b)Topology:
Tail:
HTML5 tier, run #224: success (
8e715275b3)Topology:
Tail:
HTML5 tier, run #224: success (
8e715275b3)Topology:
Tail:
HTML5 tier, run #226: success (
ddff5d6838)Topology:
Tail:
HTML5 tier (v1), run #228: success (
e50fffede6)Topology:
Tail:
HTML5 tier (v2), run #228: success (
e50fffede6)Topology:
Tail:
HTML5 tier (v1), run #230: success (
68476250a1)Topology:
Tail:
HTML5 tier (v2), run #230: failure (
68476250a1)Topology:
Tail:
HTML5 tier (v1), run #232: failure (
26ddc8cbf5)Topology:
Tail:
HTML5 tier (v2), run #232: success (
26ddc8cbf5)Topology:
Tail:
HTML5 tier (v1), run #234: success (
1b044baac4)Topology:
Tail:
HTML5 tier (v2), run #234: success (
1b044baac4)Topology:
Tail:
Cause found, fixed in #93. This was never a smoke failure — it was harness teardown.
The self-report step added in #91 is what caught it. Every phase PASSes, both loads of the identity-persistence check complete, and then the run dies:
Teardown signalled only the process it holds a handle to, but chromium's renderer and utility processes are what write the profile and they do not die with their parent, so
rmwalked the tree against live writers. That also explains why it looked wire-dependent and random: it failed on v2 on run #230 and on v1 on run #232. A race, not a protocol.#93 gives chromium its own process group, signals the group, and waits for the parent to be reaped before removing the directory. Measured against a tree with a surviving grandchild writer, 8 races per mode: parent-only kill failed 7 of 8,
rmwithmaxRetries: 3still failed 1 of 8, group kill failed 0 of 8.Leaving this open until the flip's observation window has a few more green runs on both wires. When it closes, the self-report step, the
issues: writegrant and both tees inconformance.ymlgo with it.HTML5 tier (v1), run #236: success (
c46cd42727)Topology:
Tail:
HTML5 tier (v2), run #236: success (
c46cd42727)Topology:
Tail:
HTML5 tier (v1), run #238: success (
a7432950ca)Topology:
Tail:
HTML5 tier (v2), run #238: success (
a7432950ca)Topology:
Tail:
HTML5 tier (v1), run #240: success (
8b0be6cc9d)Topology:
Tail:
HTML5 tier (v2), run #240: success (
8b0be6cc9d)Topology:
Tail:
HTML5 tier (v1), run #242: success (
d914b1cabb)Topology:
Tail:
HTML5 tier (v2), run #242: success (
d914b1cabb)Topology:
Tail:
HTML5 tier (v1), run #244: success (
da95c8e95c)Topology:
Tail:
HTML5 tier (v2), run #244: success (
da95c8e95c)Topology:
Tail:
HTML5 tier (v1), run #246: success (
11f8817dd7)Topology:
Tail:
HTML5 tier (v2), run #246: success (
11f8817dd7)Topology:
Tail:
HTML5 tier (v1), run #249: success (
4a35e5fd0f)Topology:
Tail:
HTML5 tier (v2), run #249: success (
4a35e5fd0f)Topology:
Tail:
HTML5 tier (v1), run #251: success (
e2cb4151aa)Topology:
Tail:
HTML5 tier (v2), run #251: success (
e2cb4151aa)Topology:
Tail:
HTML5 tier (v1), run #253: success (
11b2a3a86b)Topology:
Tail:
HTML5 tier (v2), run #253: success (
11b2a3a86b)Topology:
Tail:
HTML5 tier (v1), run #255: success (
6cc6cd2d9f)Topology:
Tail:
HTML5 tier (v2), run #255: success (
6cc6cd2d9f)Topology:
Tail:
HTML5 tier (v1), run #257: success (
5df3f66f32)Topology:
Tail:
HTML5 tier (v2), run #257: success (
5df3f66f32)Topology:
Tail:
HTML5 tier (v1), run #259: success (
6dcbdaf1c3)Topology:
Tail:
HTML5 tier (v2), run #259: success (
6dcbdaf1c3)Topology:
Tail:
HTML5 tier (v1), run #261: success (
b60ca5d715)Topology:
Tail:
HTML5 tier (v2), run #261: skipped (
b60ca5d715)Topology:
Tail:
HTML5 tier (v1), run #263: success (
9e8cffc73e)Topology:
Tail:
HTML5 tier (v2), run #263: success (
9e8cffc73e)Topology:
Tail:
Ran this down. It is two failures under one number.
The teardown race is fixed. The group kill in
html5-conformance.mjsholds up: 21 v2 runs and 5 v1 runs green, no leaked chromium process, no leaked profile directory. The single stale/tmp/stdb-chromium-*on this box predates that fix.What was left is not a race. After 18 runs GitHub started refusing and three consecutive runs died before any phase ran:
bob resolvere-contacts GitHub on every invocation, and the failure is fatal even though the archive is already in.internal/lib. That is the reported signature exactly - same commit, fails then passes - and it never touched SDK code.Fixed in #98: both tiers ask for
resolveonly when the cache is empty, so a warm checkout builds with no network; the HTML5 tier retries the cold-checkout fetch three times. Loops after the fix: 6/6 v2 and 5/5 v1, where the same v1 loop was 2/5 before.Leaving this open until CI shows a green stretch on its own runners, since every measurement here is from one workstation.
HTML5 tier (v1), run #265: skipped (
451b4c68b6)Topology:
Tail:
HTML5 tier (v2), run #265: skipped (
451b4c68b6)Topology:
Tail:
HTML5 tier (v1), run #267: success (
349a6412ce)Topology:
Tail:
HTML5 tier (v2), run #267: success (
349a6412ce)Topology:
Tail:
CI evidence, now that #98 has landed.
main at
349a641is green on its own runners:ci(822),conformance (v1)(825),conformance (v2)(826).Two red runs above it are worth naming so nobody reads them as this bug. Task 823/824 at
451b4c6and task 818 atb60ca5d7both died in Checkout, not in the harness:I deleted each PR head branch immediately after squash-merging it, while the pull_request-triggered run was still checking that commit out. The commit stops being reachable and the checkout fails. That is my operational mistake, and it produces exactly the reported signature — same commit, red then green — so it is worth stating plainly: do not delete a head branch until its checks have finished.
So of the three causes that have worn this issue number: the teardown race is fixed, the redundant dependency download is fixed in #98, and this third one was never a bug in the repo. Closing.
If the HTML5 leg goes red again on a run that actually reaches a phase, that is new and deserves its own issue with the log attached.
HTML5 tier (v1), run #273: success (
22b6579654)Topology:
Tail:
HTML5 tier (v2), run #273: success (
22b6579654)Topology:
Tail:
HTML5 tier (v1), run #275: success (
17db97c41e)Topology:
Tail:
HTML5 tier (v2), run #275: success (
17db97c41e)Topology:
Tail:
HTML5 tier (v1), run #277: success (
adc5c2acb6)Topology:
Tail:
HTML5 tier (v2), run #277: success (
adc5c2acb6)Topology:
Tail:
HTML5 tier (v1), run #279: success (
bb4c5f5f2f)Topology:
Tail:
HTML5 tier (v2), run #279: success (
bb4c5f5f2f)Topology:
Tail: