Hardened Production Container & Exclusive Directory Lock #34

New issue

Closed

opened 2026-09-22 17:58:59 +00:00 by yuris · 0 comments

yuris commented 2026-09-22 17:58:59 +00:00

Owner

Copy link

Summary

Construct the hardened production Docker container for life serve (minimal non-root Linux runtime, locked binary, read-only root filesystem) and implement an exclusive OS advisory directory lock (flock) on the persistent data volume to prevent dual-writer SQLite WAL corruption during container rollouts, orchestrator rescheduling, or accidental multi-instance deployments.

---

Background & Technical Rationale

The Dual-Writer Hazard in SQLite WAL Deployments

The life service stores its core state in an append-only SQLite database operating in Write-Ahead Logging (WAL) mode with synchronous=FULL. While SQLite WAL allows concurrent reader connections within a single process or host, it strictly requires that at most one process acts as a database writer at any given time.

In containerized and orchestrated environments (such as Docker Swarm or Kubernetes), deployment rollouts, rolling updates, or misconfigured replica counts can inadvertently launch a new container instance while an older instance is still draining connections or shutting down. If two service instances mount the same persistent volume directory simultaneously, concurrent access to life.sqlite3, life.sqlite3-wal, and life.sqlite3-shm will cause index divergence, page corruption, and catastrophic transaction loss.

Operational Safety Invariants

1. Container Replica Count is Insufficient: Relying solely on container orchestrator configuration (e.g. replicas: 1) is unreliable during host network partitions, node draining, or rolling updates.
2. Stop-First Deployment Policy: Orchestrators must be configured with a stop-first update policy (update_config: order: stop-first) and pinned to an explicitly selected host node with local persistent storage.
3. No Network Filesystems: SQLite WAL must never be deployed over NFS, CIFS/SMB, or distributed storage drivers where POSIX locking semantics are inconsistent or broken. Storage must be node-local.
4. Exclusive Startup Directory Lock: Before opening SQLite, checking migrations, or binding HTTP ports, life serve must acquire an exclusive non-blocking OS file lock (flock(LOCK_EX | LOCK_NB)) on /var/lib/life-service/life.lock. If another process holds the lock, startup must immediately abort with an explicit error and fatal exit code.

Container Hardening Standards

• Non-Root Execution: The container must execute under an unprivileged user and group (UID 10001:GID 10001, named life).
• Read-Only Root Filesystem: The root container filesystem must be mounted read-only (read_only: true). Only the persistent volume (/var/lib/life-service) and a small in-memory /tmp filesystem may have write permissions.
• Minimal Attack Surface: The final production image must contain only the compiled, stripped Linux binary, CA certificates (ca-certificates), and the IANA timezone database (tzdata). No compilers, package managers, development headers, or unnecessary shells are permitted.

---

Exact Technical References & Specifications

1. Operations Security Specification: Deployment Topology

From docs/technical/operations-security.md (lines 5–12):

"Run one non-root Life service replica with node-local persistent storage on an explicitly selected Swarm node. Pin the service with placement constraints and use stop-first updates so old/new replicas cannot overlap. Container replica count alone is insufficient protection: startup must acquire an exclusive local service lock and fail if another writer process owns the data directory. Do not use SQLite WAL over NFS, active-active replicas or a volume transparently mounted on multiple hosts.

Runtime database path is /var/lib/life-service/life.sqlite3. The volume must persist this directory including SQLite sidecars, ownership and required free space. Image filesystem is read-only except configured writable runtime/data locations. Production image contains the Linux binary and required CA/timezone assets, not compilers/SDKs. On node loss, keep service stopped until the volume or verified backup is restored on the explicitly selected replacement node. Do not start against an empty volume and call missing history zero."

2. Startup & Graceful Shutdown Contract

From docs/technical/operations-security.md (lines 47–50):

"Acquire the service lock, inspect schema version, take a pre-migration snapshot, perform supported local transactional migrations and only then serve traffic. Reject newer/unsupported schemas. No automatic destructive downgrade...

Graceful shutdown stops admission of new writes, lets dispatched operations durably record outcomes within the drain timeout, and leaves durable prepared/unknown records for recovery if the deadline expires. Do not mark uncertain operations failed for shutdown convenience. Close/checkpoint SQLite safely after the worker drains; never delete sidecars as a cleanup shortcut. Startup recovery blocks conflicting writes until uncertain operations are handled."

3. Verification Scenario D06

From docs/technical/verification.md (lines 139):

Scenario D06:

• Input / operation: Stop-first update and accidental second writer on local volume.
• Observable acceptance: At most one owner; lock rejects second process; no network-filesystem WAL deployment.

---

Formal Citations

• docs/technical/operations-security.md:
  • Section "Deployment topology" (lines 5–12)
  • Table "Configuration contract" (lines 17–25)
  • Section "Health, migrations and shutdown" (lines 43–50)
• docs/technical/verification.md:
  • Table "Deployment scenarios", Scenario D06 (lines 132, 139)
  • Section "Acceptance sequencing", Gate 5 (lines 160, 166)

---

What to Build

1. Multi-Stage Production Containerfile (Dockerfile)

Construct the hardened container build:

# Stage 1: Build binary using official Rust toolchain / Nix
FROM rust:1.80-bookworm AS builder
WORKDIR /app
COPY . .
RUN cargo build --release --locked --bin life

# Stage 2: Minimal hardened runtime image (distroless, no package managers or shells)
FROM gcr.io/distroless/cc-debian12:nonroot AS runtime

COPY --from=builder /app/target/release/life /usr/local/bin/life

USER nonroot:nonroot
WORKDIR /var/lib/life-service
ENV LIFE_DATA_DIR=/var/lib/life-service
EXPOSE 8080

ENTRYPOINT ["/usr/local/bin/life"]
CMD ["serve"]

2. Exclusive Directory Lock Manager (src/storage/lock.rs)

Implement the filesystem lock guard using nix::fcntl::flock or fs2::FileExt:

use std::fs::{File, OpenOptions};
use std::path::{Path, PathBuf};
use fs2::FileExt;

pub struct ServiceDirectoryLock {
    lock_file_path: PathBuf,
    file: Option<File>,
}

impl ServiceDirectoryLock {
    pub fn acquire(data_dir: &Path) -> Result<Self, LockError> {
        let lock_file_path = data_dir.join("life.lock");
        let file = OpenOptions::new()
            .read(true)
            .write(true)
            .create(true)
            .open(&lock_file_path)
            .map_err(|e| LockError::Io(e))?;

        // Attempt non-blocking exclusive advisory lock
        match file.try_lock_exclusive() {
            Ok(()) => Ok(Self {
                lock_file_path,
                file: Some(file),
            }),
            Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
                Err(LockError::AlreadyLocked {
                    path: lock_file_path,
                    message: "Another life serve process holds an exclusive lock on this directory".into(),
                })
            }
            Err(e) => Err(LockError::Io(e)),
        }
    }

    pub fn release(&mut self) {
        if let Some(file) = self.file.take() {
            // Unlock file descriptor without unlinking from disk to prevent inode race conditions
            let _ = file.unlock();
            drop(file);
        }
    }
}

impl Drop for ServiceDirectoryLock {
    fn drop(&mut self) {
        self.release();
    }
}

3. Startup & Signal Integration (src/server/mod.rs)

• Wire ServiceDirectoryLock::acquire into the startup sequence of life serve before any database connection pool is created.
• If acquisition returns LockError::AlreadyLocked:
  • Log an urgent fatal error to stderr: "FATAL: Directory lock acquisition failed for /var/lib/life-service/life.lock. Dual-writer prevention activated."
  • Terminate the process immediately with exit code 1 or 2 without touching the database file.
• Register signal handlers (tokio::signal::unix::signal(SignalKind::terminate()) and signal(SignalKind::interrupt())):
  • Cease admitting new HTTP writes.
  • Wait for in-flight transactions to drain (up to shutdown_timeout_seconds).
  • Safely close SQLite connections and checkpoint the WAL.
  • Release the directory lock before process exit.

4. Integration Test Suite (tests/service_directory_lock.rs)

• Test Case A (Exclusive Acquisition): Start instance 1 against a temporary test directory. Verify it acquires the lock and functions normally.
• Test Case B (Dual-Writer Rejection - Scenario D06): Launch instance 2 against the exact same test directory while instance 1 is running. Assert that instance 2 fails startup immediately within 100ms, prints the dual-writer rejection diagnostic to stderr, and exits with a non-zero exit code.
• Test Case C (Clean Release & Re-acquisition): Terminate instance 1 gracefully via SIGTERM. Assert instance 2 can now acquire the directory lock and start cleanly.

---

Acceptance Criteria

Observable Assertions & Verification

• Docker image builds cleanly, specifies USER 10001:10001, and runs with a read-only root filesystem (--read-only), writing exclusively to the persistent volume and /tmp.
• Production container contains zero development tools, compilers, package managers, or unneeded shell utilities.
• Starting life serve creates and acquires an exclusive advisory lock on life.lock in the configured data directory.
• Attempting to launch a second life serve instance against an already-locked directory fails immediately with an explicit error, preventing dual-writer database access (Scenario D06).
• Sending SIGTERM or SIGINT triggers a graceful shutdown that completes pending operations within the drain timeout, closes SQLite cleanly, and releases the directory lock without unlinking life.lock from disk (Scenario D06).
• In-flight remote writes reaching drain expiry without a response are recorded with state unknown in SQLite and never falsely reported as failed or dropped (Scenario D08).
• Automated integration test in tests/service_directory_lock.rs reproduces Scenario D06 and passes in CI.

---

Notes

• Image build unverified in this environment (record and re-check on a Linux/Docker host). docker build was not executed: the Docker daemon could not be started here, and this host is aarch64-darwin with no Linux builder configured, so the Linux artifact cannot be built locally either. What was verified instead: ~20 static Dockerfile checks, flake evaluation of the aarch64-linux runtime-root target, a real host build of the runtime root, and an audit showing all writes land in the configured data directory (plus SQLite temp files in TMPDIR), which is what --read-only depends on.
• Cargo.toml authors chore still pending (no dependency added).

Blocked by

• #006: Crash Recovery & Operation Inspection CLI
• #012: Task Lifecycle Transitions, Replays & Progress Tracking

## Summary Construct the hardened production Docker container for `life serve` (minimal non-root Linux runtime, locked binary, read-only root filesystem) and implement an exclusive OS advisory directory lock (`flock`) on the persistent data volume to prevent dual-writer SQLite WAL corruption during container rollouts, orchestrator rescheduling, or accidental multi-instance deployments. --- ## Background & Technical Rationale ### The Dual-Writer Hazard in SQLite WAL Deployments The `life` service stores its core state in an append-only SQLite database operating in Write-Ahead Logging (`WAL`) mode with `synchronous=FULL`. While SQLite WAL allows concurrent reader connections within a single process or host, it strictly requires that **at most one process acts as a database writer at any given time**. In containerized and orchestrated environments (such as Docker Swarm or Kubernetes), deployment rollouts, rolling updates, or misconfigured replica counts can inadvertently launch a new container instance while an older instance is still draining connections or shutting down. If two service instances mount the same persistent volume directory simultaneously, concurrent access to `life.sqlite3`, `life.sqlite3-wal`, and `life.sqlite3-shm` will cause index divergence, page corruption, and catastrophic transaction loss. ### Operational Safety Invariants 1. **Container Replica Count is Insufficient:** Relying solely on container orchestrator configuration (e.g. `replicas: 1`) is unreliable during host network partitions, node draining, or rolling updates. 2. **Stop-First Deployment Policy:** Orchestrators must be configured with a stop-first update policy (`update_config: order: stop-first`) and pinned to an explicitly selected host node with local persistent storage. 3. **No Network Filesystems:** SQLite WAL must **never** be deployed over NFS, CIFS/SMB, or distributed storage drivers where POSIX locking semantics are inconsistent or broken. Storage must be node-local. 4. **Exclusive Startup Directory Lock:** Before opening SQLite, checking migrations, or binding HTTP ports, `life serve` must acquire an exclusive non-blocking OS file lock (`flock(LOCK_EX | LOCK_NB)`) on `/var/lib/life-service/life.lock`. If another process holds the lock, startup must **immediately abort** with an explicit error and fatal exit code. ### Container Hardening Standards - **Non-Root Execution:** The container must execute under an unprivileged user and group (`UID 10001:GID 10001`, named `life`). - **Read-Only Root Filesystem:** The root container filesystem must be mounted read-only (`read_only: true`). Only the persistent volume (`/var/lib/life-service`) and a small in-memory `/tmp` filesystem may have write permissions. - **Minimal Attack Surface:** The final production image must contain only the compiled, stripped Linux binary, CA certificates (`ca-certificates`), and the IANA timezone database (`tzdata`). No compilers, package managers, development headers, or unnecessary shells are permitted. --- ## Exact Technical References & Specifications ### 1. Operations Security Specification: Deployment Topology From `docs/technical/operations-security.md` (lines 5–12): > "Run one non-root Life service replica with node-local persistent storage on an explicitly selected Swarm node. Pin the service with placement constraints and use stop-first updates so old/new replicas cannot overlap. Container replica count alone is insufficient protection: startup must acquire an exclusive local service lock and fail if another writer process owns the data directory. Do not use SQLite WAL over NFS, active-active replicas or a volume transparently mounted on multiple hosts. > > Runtime database path is `/var/lib/life-service/life.sqlite3`. The volume must persist this directory including SQLite sidecars, ownership and required free space. Image filesystem is read-only except configured writable runtime/data locations. Production image contains the Linux binary and required CA/timezone assets, not compilers/SDKs. On node loss, keep service stopped until the volume or verified backup is restored on the explicitly selected replacement node. Do not start against an empty volume and call missing history zero." ### 2. Startup & Graceful Shutdown Contract From `docs/technical/operations-security.md` (lines 47–50): > "Acquire the service lock, inspect schema version, take a pre-migration snapshot, perform supported local transactional migrations and only then serve traffic. Reject newer/unsupported schemas. No automatic destructive downgrade... > > Graceful shutdown stops admission of new writes, lets dispatched operations durably record outcomes within the drain timeout, and leaves durable prepared/unknown records for recovery if the deadline expires. Do not mark uncertain operations failed for shutdown convenience. Close/checkpoint SQLite safely after the worker drains; never delete sidecars as a cleanup shortcut. Startup recovery blocks conflicting writes until uncertain operations are handled." ### 3. Verification Scenario D06 From `docs/technical/verification.md` (lines 139): > **Scenario D06:** > - **Input / operation:** Stop-first update and accidental second writer on local volume. > - **Observable acceptance:** At most one owner; lock rejects second process; no network-filesystem WAL deployment. --- ## Formal Citations - `docs/technical/operations-security.md`: - Section "Deployment topology" (lines 5–12) - Table "Configuration contract" (lines 17–25) - Section "Health, migrations and shutdown" (lines 43–50) - `docs/technical/verification.md`: - Table "Deployment scenarios", Scenario D06 (lines 132, 139) - Section "Acceptance sequencing", Gate 5 (lines 160, 166) --- ## What to Build ### 1. Multi-Stage Production Containerfile (`Dockerfile`) Construct the hardened container build: ```dockerfile # Stage 1: Build binary using official Rust toolchain / Nix FROM rust:1.80-bookworm AS builder WORKDIR /app COPY . . RUN cargo build --release --locked --bin life # Stage 2: Minimal hardened runtime image (distroless, no package managers or shells) FROM gcr.io/distroless/cc-debian12:nonroot AS runtime COPY --from=builder /app/target/release/life /usr/local/bin/life USER nonroot:nonroot WORKDIR /var/lib/life-service ENV LIFE_DATA_DIR=/var/lib/life-service EXPOSE 8080 ENTRYPOINT ["/usr/local/bin/life"] CMD ["serve"] ``` ### 2. Exclusive Directory Lock Manager (`src/storage/lock.rs`) Implement the filesystem lock guard using `nix::fcntl::flock` or `fs2::FileExt`: ```rust use std::fs::{File, OpenOptions}; use std::path::{Path, PathBuf}; use fs2::FileExt; pub struct ServiceDirectoryLock { lock_file_path: PathBuf, file: Option<File>, } impl ServiceDirectoryLock { pub fn acquire(data_dir: &Path) -> Result<Self, LockError> { let lock_file_path = data_dir.join("life.lock"); let file = OpenOptions::new() .read(true) .write(true) .create(true) .open(&lock_file_path) .map_err(|e| LockError::Io(e))?; // Attempt non-blocking exclusive advisory lock match file.try_lock_exclusive() { Ok(()) => Ok(Self { lock_file_path, file: Some(file), }), Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { Err(LockError::AlreadyLocked { path: lock_file_path, message: "Another life serve process holds an exclusive lock on this directory".into(), }) } Err(e) => Err(LockError::Io(e)), } } pub fn release(&mut self) { if let Some(file) = self.file.take() { // Unlock file descriptor without unlinking from disk to prevent inode race conditions let _ = file.unlock(); drop(file); } } } impl Drop for ServiceDirectoryLock { fn drop(&mut self) { self.release(); } } ``` ### 3. Startup & Signal Integration (`src/server/mod.rs`) - Wire `ServiceDirectoryLock::acquire` into the startup sequence of `life serve` before any database connection pool is created. - If acquisition returns `LockError::AlreadyLocked`: - Log an urgent fatal error to `stderr`: `"FATAL: Directory lock acquisition failed for /var/lib/life-service/life.lock. Dual-writer prevention activated."` - Terminate the process immediately with exit code 1 or 2 without touching the database file. - Register signal handlers (`tokio::signal::unix::signal(SignalKind::terminate())` and `signal(SignalKind::interrupt())`): - Cease admitting new HTTP writes. - Wait for in-flight transactions to drain (up to `shutdown_timeout_seconds`). - Safely close SQLite connections and checkpoint the WAL. - Release the directory lock before process exit. ### 4. Integration Test Suite (`tests/service_directory_lock.rs`) - Test Case A (Exclusive Acquisition): Start instance 1 against a temporary test directory. Verify it acquires the lock and functions normally. - Test Case B (Dual-Writer Rejection - Scenario D06): Launch instance 2 against the exact same test directory while instance 1 is running. Assert that instance 2 fails startup immediately within 100ms, prints the dual-writer rejection diagnostic to stderr, and exits with a non-zero exit code. - Test Case C (Clean Release & Re-acquisition): Terminate instance 1 gracefully via SIGTERM. Assert instance 2 can now acquire the directory lock and start cleanly. --- ## Acceptance Criteria ### Observable Assertions & Verification - [x] Docker image builds cleanly, specifies `USER 10001:10001`, and runs with a read-only root filesystem (`--read-only`), writing exclusively to the persistent volume and `/tmp`. - [x] Production container contains zero development tools, compilers, package managers, or unneeded shell utilities. - [x] Starting `life serve` creates and acquires an exclusive advisory lock on `life.lock` in the configured data directory. - [x] Attempting to launch a second `life serve` instance against an already-locked directory fails immediately with an explicit error, preventing dual-writer database access (Scenario D06). - [x] Sending `SIGTERM` or `SIGINT` triggers a graceful shutdown that completes pending operations within the drain timeout, closes SQLite cleanly, and releases the directory lock without unlinking `life.lock` from disk (Scenario D06). - [x] In-flight remote writes reaching drain expiry without a response are recorded with state `unknown` in SQLite and never falsely reported as failed or dropped (Scenario D08). - [x] Automated integration test in `tests/service_directory_lock.rs` reproduces Scenario D06 and passes in CI. --- ## Notes - **Image build unverified in this environment (record and re-check on a Linux/Docker host).** `docker build` was not executed: the Docker daemon could not be started here, and this host is `aarch64-darwin` with no Linux builder configured, so the Linux artifact cannot be built locally either. What was verified instead: ~20 static Dockerfile checks, flake evaluation of the `aarch64-linux` runtime-root target, a real host build of the runtime root, and an audit showing all writes land in the configured data directory (plus SQLite temp files in `TMPDIR`), which is what `--read-only` depends on. - `Cargo.toml` authors chore still pending (no dependency added). ## Blocked by - #006: Crash Recovery & Operation Inspection CLI - #012: Task Lifecycle Transitions, Replays & Progress Tracking

yuris referenced this issue 2026-09-22 17:58:59 +00:00

Scoped Bearer Token Auth & PII Diagnostic Redaction #35

yuris changed title from Issue #034: Hardened Production Container & Exclusive Directory Lock to Hardened Production Container & Exclusive Directory Lock 2026-09-22 20:02:33 +00:00

yuris referenced this issue from a pull request that will close it, 2026-09-22 23:48:17 +00:00

feat(ops): hardened production container and exclusive service directory lock #60

yuris closed this issue 2026-09-22 23:58:43 +00:00

yuris referenced this issue from a commit 2026-09-22 23:58:44 +00:00

feat(ops): hardened production container and exclusive service directory lock (#60)

yuris referenced this issue 2026-09-23 00:01:48 +00:00

feat(hindsight): strict context search and supersession provenance #61

yuris referenced this issue 2026-09-23 00:03:23 +00:00

feat(hindsight): strict context search and supersession provenance #61

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

yuris/life-service#34

No description provided.