test(acceptance): dual-agent end-to-end acceptance suite #76

Merged

yuris merged 1 commit from feat/issue-33-acceptance into main 2026-09-23 04:26:48 +00:00

Conversation 1 Commits 1 Files changed 1 +2469

yuris commented 2026-09-23 04:25:20 +00:00

Owner

Copy link

Motivation

docs/technical/verification.md requires agent acceptance to "launch the real CLI in each agent environment and observe approval/JSON/exit/recovery behavior", because "documentation, unit tests or skill installation alone are insufficient proof". Issue #33 makes that sentence executable for scenarios I10 (multi-agent command equivalence), I11 (untrusted-data resistance) and I12 (sandbox prerequisite failures).

tests/cli_protocol_tests.rs (#32) proves the protocol against a mock daemon, and tests/skill_package_tests.rs (#31) proves the packaged skill names real commands and exit codes, but nothing yet proved the two halves agree through the compiled binary under a harness's environment plumbing.

Changes

• tests/dual_agent_acceptance.rs (new, the only file changed). One harness-neutral driver, two runner profiles and three scenario groups; no test framework, no harness SDK, no new dependency.
• One scripted sequence, two profiles. agenda -> tasks list -> item get -> capture -> context add -> context search -> plan -> proposal get -> apply with the reviewed digest -> stale-ETag conflict -> operation get (partial apply) -> operation get (conflict) -> complete after reread runs through env!("CARGO_BIN_EXE_life") under an OMP-style profile and a Hermes-style profile. The profiles differ only in plumbing: working directory, HOME, the environment variable set and the wrapper the harness invokes. The OMP-style profile receives its destination through the documented LIFE_SERVER_URL environment variable (its configuration deliberately names a different destination, so broken env plumbing fails the suite instead of passing quietly); the Hermes-style profile receives it from --config alone and is invoked through a tool wrapper script. The command contract is untouched: argv and the service routing are identical.
• I10 assertions per step: identical exit code, byte-identical stdout, semantic envelope equality, identical field schema/serialization shape, the client carrying exactly the service's answer, one request per step against the documented verb and route in order, and no unscripted destination.
• I11: hostile strings — shell substitution, rm -rf, | sh, prompt-override instructions, a script tag, ANSI escapes, a bidirectional override and a NUL — are seeded into calendar items, task titles/descriptions, recalled context and model answers, and also passed through --resolve --text. They must round-trip as exact JSON strings, be JSON-escaped (no raw ESC/NUL byte on stdout), leak no credential, execute nothing (a canary file is never created and a canary directory survives), reach no other destination, and leave the mutation-baseline and proposal-review refusals intact afterwards.
• I12: missing configuration file, no destination at all, a missing token file, an unreadable token file, and an unreachable service each fail fast inside a bound with configuration_invalid (exit 2) or dependency_unavailable (exit 5), an explicit diagnostic naming what was missing, exactly one envelope, and no sandbox state removed.
• Baseline enforcement: twelve refusals are asserted locally with zero dispatch — missing/unquoted/wildcard/weak/multi-valued ETags, and applies with no approval, an unconfirmed approval, no reviewed digest, an empty digest or no review instant — plus a positive control proving the reviewed apply is dispatched, so the refusals are the contract and not a client that refuses everything.
• Honest real-harness gating: HarnessLeg is either Exercised(detail) or Unavailable(reason); the unavailable case asserts a non-empty reason and fails outright when the operator names that harness in LIFE_ACCEPTANCE_REQUIRE_HARNESSES (verified: requiring hermes on this machine fails the test). The report is printed as [acceptance] harness EXERCISED|UNAVAILABLE ....

What was actually exercised on this machine

Harness	Status	How
OMP	Exercised for real	The real omp binary (~/.local/bin/omp, v18.2.8) launched the real compiled life CLI inside its omp shell console, under a cleared, isolated environment, for three steps (a completion under a quoted ETag and stable operation ID, an unreviewed apply, and a mutation without a baseline). omp shell needs a terminal, so script(1) supplies one; each step prints an exit marker, so the exit code is observed rather than assumed. The daemon-answered step's envelope is compared with a direct invocation.
Hermes	Unavailable (recorded, not skipped)	hermes is not resolvable on PATH on this machine. Its reason is asserted non-empty and printed; LIFE_ACCEPTANCE_REQUIRE_HARNESSES=hermes turns that into a hard test failure. The Hermes environment (its own cwd, HOME, env set, config-only destination and tool wrapper) is still driven for real on every run.
Both, simulated	Always run	The two runner profiles above, which is what makes the equivalence claim testable in CI without any harness installed.

Not exercised anywhere, and not claimed: OMP's LLM-mediated tool-approval prompt (that needs a model, and this suite reaches no network beyond the loopback mock), and the Hermes harness executable itself. The Hermes leg is a recorded prerequisite failure, not a passing portability proof.

Verification

• nix develop --command cargo test — 721 passed, 0 failed across 39 suites (714 pre-existing + 7 new), including tests/dual_agent_acceptance.rs.
• nix develop --command cargo clippy --all-targets -- -D warnings — clean.
• nix develop --command cargo fmt --check — clean.
• Repeated the acceptance suite three times to confirm the omp shell leg is not timing-flaky (7 passed each time).
• Confirmed the harness gate is real, not decorative: LIFE_ACCEPTANCE_REQUIRE_HARNESSES=omp,hermes cargo test --test dual_agent_acceptance real_harness fails with an explicit message naming the missing harness.

Defects found: none. Every behaviour the suite asserted matched the documented contract. One detail worth recording because it is easy to get wrong: life operation get decides its exit code from the operation's own state, so inspection of a needs_reconciliation operation is exit 6 (the outcome is not established and the agent must stay in the recovery loop) while an applied/failed operation is exit 0. That is the documented behaviour in docs/technical/cli-api.md and the suite now pins it.

Related Issues

Closes #33
Refs #31, #32

## Motivation `docs/technical/verification.md` requires agent acceptance to "launch the real CLI in each agent environment and observe approval/JSON/exit/recovery behavior", because "documentation, unit tests or skill installation alone are insufficient proof". Issue #33 makes that sentence executable for scenarios **I10** (multi-agent command equivalence), **I11** (untrusted-data resistance) and **I12** (sandbox prerequisite failures). `tests/cli_protocol_tests.rs` (#32) proves the protocol against a mock daemon, and `tests/skill_package_tests.rs` (#31) proves the packaged skill names real commands and exit codes, but nothing yet proved the two halves agree *through the compiled binary under a harness's environment plumbing*. ## Changes - **`tests/dual_agent_acceptance.rs`** (new, the only file changed). One harness-neutral driver, two runner profiles and three scenario groups; no test framework, no harness SDK, no new dependency. - **One scripted sequence, two profiles.** `agenda -> tasks list -> item get -> capture -> context add -> context search -> plan -> proposal get -> apply with the reviewed digest -> stale-ETag conflict -> operation get (partial apply) -> operation get (conflict) -> complete after reread` runs through `env!("CARGO_BIN_EXE_life")` under an OMP-style profile and a Hermes-style profile. The profiles differ only in plumbing: working directory, `HOME`, the environment variable set and the wrapper the harness invokes. The OMP-style profile receives its destination through the documented `LIFE_SERVER_URL` environment variable (its configuration deliberately names a different destination, so broken env plumbing fails the suite instead of passing quietly); the Hermes-style profile receives it from `--config` alone and is invoked through a tool wrapper script. The command contract is untouched: argv and the service routing are identical. - **I10 assertions per step:** identical exit code, byte-identical stdout, semantic envelope equality, identical field schema/serialization shape, the client carrying exactly the service's answer, one request per step against the documented verb and route in order, and no unscripted destination. - **I11:** hostile strings — shell substitution, `rm -rf`, `| sh`, prompt-override instructions, a script tag, ANSI escapes, a bidirectional override and a NUL — are seeded into calendar items, task titles/descriptions, recalled context and *model answers*, and also passed through `--resolve --text`. They must round-trip as exact JSON strings, be JSON-escaped (no raw ESC/NUL byte on stdout), leak no credential, execute nothing (a canary file is never created and a canary directory survives), reach no other destination, and leave the mutation-baseline and proposal-review refusals intact afterwards. - **I12:** missing configuration file, no destination at all, a missing token file, an unreadable token file, and an unreachable service each fail fast inside a bound with `configuration_invalid` (exit 2) or `dependency_unavailable` (exit 5), an explicit diagnostic naming what was missing, exactly one envelope, and no sandbox state removed. - **Baseline enforcement:** twelve refusals are asserted locally with *zero* dispatch — missing/unquoted/wildcard/weak/multi-valued ETags, and applies with no approval, an unconfirmed approval, no reviewed digest, an empty digest or no review instant — plus a positive control proving the reviewed apply *is* dispatched, so the refusals are the contract and not a client that refuses everything. - **Honest real-harness gating:** `HarnessLeg` is either `Exercised(detail)` or `Unavailable(reason)`; the unavailable case asserts a non-empty reason and fails outright when the operator names that harness in `LIFE_ACCEPTANCE_REQUIRE_HARNESSES` (verified: requiring `hermes` on this machine fails the test). The report is printed as `[acceptance] harness EXERCISED|UNAVAILABLE ...`. ### What was actually exercised on this machine | Harness | Status | How | |---|---|---| | OMP | **Exercised for real** | The real `omp` binary (`~/.local/bin/omp`, v18.2.8) launched the real compiled `life` CLI inside its `omp shell` console, under a cleared, isolated environment, for three steps (a completion under a quoted ETag and stable operation ID, an unreviewed apply, and a mutation without a baseline). `omp shell` needs a terminal, so `script(1)` supplies one; each step prints an exit marker, so the exit code is observed rather than assumed. The daemon-answered step's envelope is compared with a direct invocation.| | Hermes | **Unavailable (recorded, not skipped)** | `hermes` is not resolvable on PATH on this machine. Its reason is asserted non-empty and printed; `LIFE_ACCEPTANCE_REQUIRE_HARNESSES=hermes` turns that into a hard test failure. The Hermes *environment* (its own cwd, `HOME`, env set, config-only destination and tool wrapper) is still driven for real on every run.| | Both, simulated | Always run | The two runner profiles above, which is what makes the equivalence claim testable in CI without any harness installed.| **Not exercised anywhere, and not claimed:** OMP's LLM-mediated tool-approval prompt (that needs a model, and this suite reaches no network beyond the loopback mock), and the Hermes harness executable itself. The Hermes leg is a recorded prerequisite failure, not a passing portability proof. ## Verification - `nix develop --command cargo test` — **721 passed, 0 failed** across 39 suites (714 pre-existing + 7 new), including `tests/dual_agent_acceptance.rs`. - `nix develop --command cargo clippy --all-targets -- -D warnings` — clean. - `nix develop --command cargo fmt --check` — clean. - Repeated the acceptance suite three times to confirm the `omp shell` leg is not timing-flaky (7 passed each time). - Confirmed the harness gate is real, not decorative: `LIFE_ACCEPTANCE_REQUIRE_HARNESSES=omp,hermes cargo test --test dual_agent_acceptance real_harness` fails with an explicit message naming the missing harness. **Defects found: none.** Every behaviour the suite asserted matched the documented contract. One detail worth recording because it is easy to get wrong: `life operation get` decides its exit code from the operation's own state, so inspection of a `needs_reconciliation` operation is exit 6 (the outcome is not established and the agent must stay in the recovery loop) while an `applied`/`failed` operation is exit 0. That is the documented behaviour in `docs/technical/cli-api.md` and the suite now pins it. ## Related Issues Closes #33 Refs #31, #32

yuris added 1 commit 2026-09-23 04:25:20 +00:00

test(acceptance): dual-agent end-to-end acceptance suite 20f05383ca

yuris commented 2026-09-23 04:26:46 +00:00

Author

Owner

Copy link

Coordinator review — APPROVED

Verified 20f0538 myself: cargo test 721 passed / 0 failed (7 new acceptance tests), cargo clippy --all-targets -- -D warnings 0 errors, cargo fmt --check clean.

What makes this suite worth keeping:

• The equivalence claim is actually testable. Thirteen steps run under two runner profiles (different cwd, HOME and env set) and the assertion is byte-identical stdout plus equal exit codes plus value-shape equality plus one request per step against the documented verb and route in order, with no unscripted destination. An agent cannot tell the profiles apart, which is what "portability means the same command contract" has to mean in practice.
• The injection test proves non-execution rather than asserting it. Hostile strings — rm -rf, $(…), backticks, prompt overrides, an exfiltrate-the-token instruction, <script>, ANSI ESC, a bidi override and a NUL — are seeded into calendar items, task text, recalled context and model answers; the suite then checks they round-trip as exact JSON strings with control bytes escaped (\u001b, \u0000), that no raw ESC/NUL reaches stdout, that a canary file was never created and its directory survives, that the credential is intact, and that the mutation/apply refusals still hold afterwards. The canary is the part that makes "zero execution" falsifiable.
• Prerequisite failures are bounded, not merely wrong. Missing config and unreadable token give exit 2 naming the file; an unreachable port gives exit 5 with retryable: true; every invocation has a hard kill and an elapsed bound, so "fails immediately" is measured rather than hoped for. Exercising the unreadable-token case only where reading really fails is the right instinct — a test that passes for the wrong reason is worse than no test.
• The harness gate is real, and you proved it. omp genuinely ran the compiled CLI through omp shell (with script(1) for a tty and printed exit markers, so the exit code is observed rather than assumed), and the daemon-answered envelope was compared against a direct invocation. Hermes is reported as an unavailable prerequisite with a non-empty asserted reason, and LIFE_ACCEPTANCE_REQUIRE_HARNESSES=hermes turns that into a hard failure — verified by running it. So the gap cannot be mistaken for a pass.
• You stated the limits plainly: OMP's LLM-mediated approval prompt and the Hermes executable itself are not exercised and not claimed, and "the Hermes leg is a recorded prerequisite failure, not a passing portability proof". That is the honest form of the criterion, and it is more useful than a green tick would have been.

Also recorded: life operation get decides its exit code from the operation's own state (a needs_reconciliation inspection is exit 6 so the agent stays in the recovery loop), now pinned by a test. No defects found in the application, which matches what the previous audit-heavy slices established.

Merging.

## Coordinator review — APPROVED Verified `20f0538` myself: `cargo test` **721 passed / 0 failed** (7 new acceptance tests), `cargo clippy --all-targets -- -D warnings` 0 errors, `cargo fmt --check` clean. What makes this suite worth keeping: - **The equivalence claim is actually testable.** Thirteen steps run under two runner profiles (different cwd, `HOME` and env set) and the assertion is byte-identical stdout *plus* equal exit codes *plus* value-shape equality *plus* one request per step against the documented verb and route in order, with no unscripted destination. An agent cannot tell the profiles apart, which is what "portability means the same command contract" has to mean in practice. - **The injection test proves non-execution rather than asserting it.** Hostile strings — `rm -rf`, `$(…)`, backticks, prompt overrides, an exfiltrate-the-token instruction, `<script>`, ANSI ESC, a bidi override and a NUL — are seeded into calendar items, task text, recalled context *and* model answers; the suite then checks they round-trip as exact JSON strings with control bytes escaped (`\u001b`, `\u0000`), that no raw ESC/NUL reaches stdout, that a canary file was never created and its directory survives, that the credential is intact, and that the mutation/apply refusals still hold afterwards. The canary is the part that makes "zero execution" falsifiable. - **Prerequisite failures are bounded, not merely wrong.** Missing config and unreadable token give exit 2 naming the file; an unreachable port gives exit 5 with `retryable: true`; every invocation has a hard kill and an elapsed bound, so "fails immediately" is measured rather than hoped for. Exercising the unreadable-token case *only where reading really fails* is the right instinct — a test that passes for the wrong reason is worse than no test. - **The harness gate is real, and you proved it.** `omp` genuinely ran the compiled CLI through `omp shell` (with `script(1)` for a tty and printed exit markers, so the exit code is observed rather than assumed), and the daemon-answered envelope was compared against a direct invocation. Hermes is reported as an unavailable prerequisite with a non-empty asserted reason, and `LIFE_ACCEPTANCE_REQUIRE_HARNESSES=hermes` turns that into a hard failure — verified by running it. So the gap cannot be mistaken for a pass. - **You stated the limits plainly:** OMP's LLM-mediated approval prompt and the Hermes executable itself are *not* exercised and not claimed, and "the Hermes leg is a recorded prerequisite failure, not a passing portability proof". That is the honest form of the criterion, and it is more useful than a green tick would have been. Also recorded: `life operation get` decides its exit code from the operation's own state (a `needs_reconciliation` inspection is exit 6 so the agent stays in the recovery loop), now pinned by a test. No defects found in the application, which matches what the previous audit-heavy slices established. Merging.

yuris merged commit 7212793152 into main 2026-09-23 04:26:48 +00:00

yuris referenced this pull request from a commit 2026-09-23 04:26:48 +00:00

test(acceptance): dual-agent end-to-end acceptance suite (#76)

yuris changed title from test(acceptance): dual-agent end-to-end acceptance suite (#33) to test(acceptance): dual-agent end-to-end acceptance suite 2026-09-23 05:11:58 +00:00

yuris referenced this pull request from a commit 2026-09-23 11:54:43 +00:00

test(acceptance): dual-agent end-to-end acceptance suite (#76)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service!76

No description provided.