automaton/tasks/runnable-test-suite/SPEC.md

SPEC — runnable-test-suite

Goal

Make python3 -m pytest tests/ -v pass from a clean checkout of ~/.automaton, with deterministic Python deps pinned in the repo and documentation that reflects the actual interpreter that ships on the user's machine.

The prime symptom that proves nothing at all runs today: a fresh clone executes python (per AGENTS.md) and silently fails because macOS only ships python3, and even with the right interpreter the suite fails on No module named pytest.

Requirements (numbered)

1. Add requirements.txt at the repo root pinning pytest (lowest version that supports the syntax used in tests/, which is plain fixtures and tmp_path — pytest ≥ 7.0). No other third-party deps may be added.
2. Provide a venv-based install path: a one-line install in scripts/install.sh (or a new snippet) that creates .venv/ and pip install -r requirements.txt. Must not require sudo and must not pollute the system Python.
3. Make python3 -m pytest tests/ -v exit 0 from a clean checkout after pip install -r requirements.txt (no venv required — system pip3 install -r requirements.txt must also work).
4. Fix every Python file under the repo that fails python3 -m py_compile (currently clean, but must stay clean).
5. Replace every bare python invocation in documentation and prompts with python3 so the documented commands actually run on a stock macOS without a shim.
   • AGENTS.md lines 58, 61, 64, 70
   • README.md lines 203, 206, 209, 212, 213, 216, 219, 222, 225, 228, 229, 230, 242, 245, 324, 338, 353
   • automaton/dashboard/README.md lines 11, 18, 21, 24
   • prompts/orchestrate.md lines 28, 29, 30, 36
   • Any other python (bare) reference found by rg AFTER the first pass
6. Do NOT change python references inside shell scripts that already invoke #!/usr/bin/env python3 shebangs or that explicitly resolve via command -v. Only fix bare python commands that shell out (none expected in scripts/ after audit, but verify).
7. Add a CI step note to CHANGELOG.md under [unreleased] documenting the new requirements.txt and the python3 requirement.
8. Update AGENTS.md "Build & Test Commands" section to reference requirements.txt and use python3 consistently.

Acceptance criteria

Each must pass from a fresh clone with only stock macOS CommandLineTools + pip3:

1. pip3 install -r requirements.txt succeeds.
2. python3 -m pytest tests/ -v exits 0 with N passed (N ≥ 1) and zero error lines.
3. python3 -m py_compile automaton/**/*.py automaton/dashboard/**/*.py scripts/*.py exits 0.
4. bash -n scripts/*.sh exits 0.
5. rg -n "^python |\"python " AGENTS.md README.md automaton/dashboard/README.md prompts/orchestrate.md returns zero matches for a bare python command.
6. Following the install instructions in AGENTS.md verbatim, a new contributor can run the test suite within 60 seconds of clone.

Success contract (streak)

Per the goal mode this task derives from, "done" requires 10 consecutive clean python3 -m pytest tests/ -v runs in a row without any edit between runs. A single failure resets the counter. The cap on attempts is 5; on hitting the cap, stop and report.

Constraints / non-goals

• No new dependencies beyond pytest. Do not add pytest-cov, pytest-mock, tox, etc.
• No virtualenv vendoring. The user creates .venv themselves if they want isolation; system pip3 install -r requirements.txt must also work.
• No changes to existing test logic. If a test is genuinely broken (not just import-failing because pytest is missing), STOP and report — do not patch the test to make it pass. That is the anti-spin rule from #9 in the source article.
• Do not touch any file under tasks/ (per-framework tasks are state, not source).
• Do not modify status.py, vram_detect.py, or any other runtime script's behavior. Only documentation and config files change.
• No Docker, no conda, no pyenv requirements. Stock python3 + pip3 only.
• VRAM-aware scoping: this task fits in ONE sub-task (~9k peak context budget on this 32GB-RAM / no-GPU machine with Model: auto). Do not decompose further. Sub-tasks would exceed the budget on overhead alone.

Recommended implementation approach (high-level)

1. Create requirements.txt with pytest==7.4.4 (last 7.x; works on Python 3.9+).
2. pip3 install -r requirements.txt locally and run the suite; capture every failure.
3. For each failure, decide: import/install issue (fix dep) vs. real code bug (report, do not patch test).
4. Sweep python → python3 in docs/prompts with edit batching.
5. Add install snippet to scripts/install.sh (idempotent; only if .venv doesn't exist).
6. Add a one-line test smoke-check at the end of install.sh: python3 -m pytest tests/ -q || echo "tests deferred".
7. Update CHANGELOG.md [unreleased].
8. Run the streak verifier: 10× python3 -m pytest tests/ -v; stop at first clean streak or 5 attempts.