automaton/tasks/add-state-loop-lock/SPEC.md

# Add `.state.loop` File Lock

Close the tick/approve TOCTOU races flagged in `add-status-brakes/ADVERSARIAL_BUG_REPORT.md` (A6) and `add-loop-runner/ADVERSARIAL_BUG_REPORT.md` (A2, A7). Both reports name the same v1.1 fix: a file lock on `.state.loop` that serializes read-modify-write cycles across processes.

This is a v1.1 hardening task. No new features; no user-visible CLI change. Pure robustness.

## Goal

Add a cross-platform file-lock helper that wraps every `_read_state_loop` → mutate → `_write_state_loop` cycle in `status.py` and `loop-runner.py`. Concurrent ticks (two schedulers firing the same loop) and concurrent approve-vs-tick writes will serialize instead of overwriting each other.

## Background — the race

`_write_state_loop` already does atomic tmp-then-`replace` (status.py:1662). The write itself is atomic. The race is **read-modify-write**:

1. Tick A reads `.state.loop` (count=9).
2. Tick B reads `.state.loop` (count=9).
3. Tick A passes `--check-gate` (count=9 < max=10).
4. Tick B passes `--check-gate` (count=9 < max=10).
5. Tick A runs harness, writes count=10.
6. Tick B runs harness, writes count=10. (Still bounded, but two ticks ran for one increment.)

The `--approve --loop` write vs a concurrent tick's iteration increment is the same shape (status.py A6): approve wins, tick's increment is lost.

The lock closes both by serializing the full read-modify-write critical section.

## Requirements

### R1. New helper: `_loop_lock(loop_path, exclusive=True)`

A context manager (`contextlib.contextmanager` or `__enter__/__exit__` class) that:

- Opens `<loop_path>/.state.lock` (creating it if absent) and holds an OS-level **exclusive** lock for the duration of the `with` block.
- On exit: releases the lock. The `.state.lock` file may be left on disk (it's tiny and idempotent across runs); not garbage-collected.
- **Blocking acquire**: a second acquirer waits until the first releases. No timeout in v1.1 (loop ticks are short; if a tick wedges, the operator notices via `--loop-list` showing stale `last_tick_at` and intervenes manually).
- **Cross-platform**:
  - POSIX (`sys.platform != "win32"`): `fcntl.flock(fd, LOCK_EX)` for acquire, `fcntl.flock(fd, LOCK_UN)` for release.
  - Windows (`sys.platform == "win32"`): `msvcrt.locking(fd, LK_LOCK, 1)` blocking acquire on a 1-byte region; release via `msvcrt.locking(fd, LK_UNLCK, 1)`. `msvcrt` is stdlib on Windows.
- On `BrokenPipeError`/`IOError` from a vanished loop dir mid-hold: surface a clear error `"loop dir vanished mid-lock"` and exit nonzero. Don't mask it.
- File handle is kept open for the life of the `with`; closed in `finally`.

### R2. Wrap every read-modify-write cycle in `status.py`

Locate each `_write_state_loop(...)` callsite in `scripts/status.py` (lines 1721, 1948, 1971, 1993) and confirm each is preceded by a `_read_state_loop(...)` that seeds it. Wrap the read+mutate+write block in `with _loop_lock(loop_path):`. Do NOT wrap the `--create-loop` path (status.py:1819) — there is no prior state to race against; duplicate create is already refused by name (R-of-create-task).

Affected commands in status.py:
- `cmd_check_gate` (halt write) — status.py:1721
- `cmd_pause_loop` (paused) — status.py:1948
- `cmd_resume_loop` (running) — status.py:1971
- `cmd_approve_loop` (halt clear + resumed_count++) — status.py:1993

The lock must cover the read that precedes each of these writes, not just the write. (Wrapping only the write wouldn't close the race — that just makes writes atomic, which they already are.)

### R3. Wrap every read-modify-write cycle in `loop-runner.py`

In `scripts/loop-runner.py`, wrap the read+mutate+write in `cmd_tick`'s step 10 ("atomic state write" per `design/loops/technical.md` §7) and any other `_write_state_loop` callsite (lines 125, 244, 684). Mirror the same `with _loop_lock(loop_path):` pattern.

The lock **must** be held across:
- The `--check-gate` subprocess call's effective decision (i.e. the read of `iteration_count`/`status` it makes), AND
- The subsequent state mutation write.

Since `--check-gate` runs as a subprocess and reads `.state.loop` itself, the lock acquired by the runner blocks the *runner's own* subsequent subprocess read from racing a concurrent approve write, but it cannot lock the *subprocess* itself. This is acceptable: the lock scope we control is the parent runner's read-modify-write; a concurrent tick would block on `.state.lock` at the parent-runner level and the gate call inside it would still see consistent state.

### R4. Idempotence and no-op fast path

If a command reads `.state.loop`, discovers no mutation is needed (e.g. `--pause-loop` on an already-paused loop), it still releases the lock cleanly. The lock MUST always be released, even on early-return code paths inside the `with` block. Use `try/finally` inside the context manager, not inside callers.

### R5. Lock file location

`.state.lock` lives in the loop's own dir (`<loop_path>/.state.lock`), NOT the framework root. Rationale: per-loop granularity; a lock on loop A's tick must not block loop B's approve. Untracked loops (no `.state.loop`) still get a `.state.lock` file on first acquire — that's fine; the file is empty.

### R6. No new pip deps

Use stdlib only: `fcntl` (POSIX), `msvcrt` (Windows), `contextlib`, `sys`, `os`. Both are already conditionally imported elsewhere in the framework (`platform.system()` dispatch in task 5).

### R7. Compatibility with existing atomic write

The existing `_write_state_loop` tmp-then-replace stays. The lock adds a coarse mutex around the read-modify-write cycle; the atomic write provides last-write-wins safety even if some future code path forgets the lock. Defense-in-depth; no regression to the existing atomic semantics.

## Non-goals

- No `--claim-loop-task` (that's task 6).
- No timeout / deadlock detection — out of scope; ticks are short. If a future tick grows long, address then.
- No advisory locking visible to harnesses — internal only; no CLI surface.
- No `outputs.retention` GC (task 3).
- No `blast_radius.base_branch` parameterization (task 4).

## Test plan (`tests/test_state_loop_lock.py`)

New tests, all stdlib, all using `tmp_path`:

1. `test_lock_serializes_concurrent_writes`: two threads kicked off simultaneously, each does read→sleep(0.05)→write under the lock. Assert timestamps don't interleave (one finishes before the other starts its write). Use a shared "interleave detector" (a list append of enter/exit times compared after).
2. `test_lock_releases_on_clean_exit`: acquire+release; the next acquire on the same loop succeeds immediately.
3. `test_lock_releases_on_exception`: `with _loop_lock(p): raise ValueError`; next acquire succeeds.
4. `test_lock_is_per_loop`: two lock acquisitions on two different loop dirs run concurrently without blocking each other (assert both complete within a tightly bounded wall-clock window).
5. `test_no_lock_on_create_loop`: `--create-loop` of a new loop does NOT create a `.state.lock` file (create-path is unwrapped per R2). Then `--check-gate` on it acquires/releases the lock, leaving `.state.lock` behind.
6. `test_pause_loop_serialized_with_concurrent_read`: spawn a thread that holds `_loop_lock` for 0.1s; main thread calls `--pause-loop` and assert it completes after 0.1s (not before). Confirms commands actually acquire the lock.
7. `test_runner_tick_holds_lock_across_state_write`: integration-style — invoke `loop-runner.py --mode tick` against a loop whose tick is artificially delayed, while a parallel `--approve --loop` is held; assert approve completes after the tick. (Skip if it grows flaky — turns into a smoke test asserting the lock file appears.)

Reuse the `_make_loop` helper pattern from `tests/test_status_brakes.py` for loop dir scaffolding.

## Concrete code shape

```python
@contextlib.contextmanager
def _loop_lock(loop_path: Path, exclusive: bool = True):
    lock_file = loop_path / ".state.lock"
    fd = os.open(str(lock_file), os.O_RDWR | os.O_CREAT, 0o644)
    acquired = False
    try:
        if sys.platform == "win32":
            import msvcrt
            msvcrt.locking(fd, msvcrt.LK_LOCK if exclusive else msvcrt.LK_NBLCK, 1)
        else:
            import fcntl
            fcntl.flock(fd, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)
        acquired = True
        yield
    finally:
        if acquired:
            if sys.platform == "win32":
                import msvcrt
                try:
                    msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
                except OSError:
                    pass
            else:
                import fcntl
                fcntl.flock(fd, fcntl.LOCK_UN)
        os.close(fd)
```

Callers:
```python
with _loop_lock(loop_path):
    state = _read_state_loop(loop_path) or _initial_state_loop(name)
    state["status"] = "paused"
    _write_state_loop(loop_path, state)
```

## D-items (decisions locked for this task)

- **D-L1**: blocking acquire, no timeout in v1.1. Ticks short; operator notices via `--loop-list` stale `last_tick_at`.
- **D-L2**: `.state.lock` is per-loop, lives in the loop dir, not garbage-collected.
- **D-L3**: `--create-loop` path is unwrapped (no prior state to race against; create is name-unique-refused).
- **D-L4**: stdlib only (`fcntl` POSIX, `msvcrt` Windows). No `filelock` package.
- **D-L5**: existing atomic write semantics retained (defense-in-depth).

## Risks

- **Deadlock if a path holds the lock and re-enters a function that tries to re-acquire.** Mitigation: `_loop_lock` is not re-entrant — audit every callsite to ensure no nested `_loop_lock` within the same `with` block. POSIX `flock` is re-entrant on the same fd; Windows `msvcrt.locking` is not. Safer to forbid nesting and document it.
- **Linux `flock` on NFS has known caveats.** Out of scope: the loop dir is always local (project root or `~/.automaton`). Document in the helper's docstring.

## Verification

- `python3 -m py_compile scripts/status.py scripts/loop-runner.py`
- `python3 -m pytest tests/test_state_loop_lock.py -v`
- `python3 -m pytest tests/ -q` (full suite must remain green; 433 baseline + new)
- Manual: `python3 scripts/status.py --create-loop t --from-template ci-triage && python3 scripts/status.py --check-gate t` — should succeed and leave `.state.lock` behind on first acquire.