59 lines
4.9 KiB
Markdown
59 lines
4.9 KiB
Markdown
|
# SPEC — runnable-test-suite
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Make `python3 -m pytest tests/ -v` pass from a clean checkout of `~/.automaton`, with deterministic Python deps pinned in the repo and documentation that reflects the actual interpreter that ships on the user's machine.
|
|||
|
|
|
|||
|
|
The prime symptom that proves nothing at all runs today: a fresh clone executes `python` (per `AGENTS.md`) and silently fails because macOS only ships `python3`, and even with the right interpreter the suite fails on `No module named pytest`.
|
|||
|
|
|
|||
|
|
## Requirements (numbered)
|
|||
|
|
|
|||
|
|
1. Add `requirements.txt` at the repo root pinning `pytest` (lowest version that supports the syntax used in `tests/`, which is plain fixtures and `tmp_path` — pytest ≥ 7.0). No other third-party deps may be added.
|
|||
|
|
2. Provide a venv-based install path: a one-line install in `scripts/install.sh` (or a new snippet) that creates `.venv/` and `pip install -r requirements.txt`. Must not require sudo and must not pollute the system Python.
|
|||
|
|
3. Make `python3 -m pytest tests/ -v` exit 0 from a clean checkout after `pip install -r requirements.txt` (no venv required — system `pip3 install -r requirements.txt` must also work).
|
|||
|
|
4. Fix every Python file under the repo that fails `python3 -m py_compile` (currently clean, but must stay clean).
|
|||
|
|
5. Replace every bare `python ` invocation in documentation and prompts with `python3 ` so the documented commands actually run on a stock macOS without a shim.
|
|||
|
|
- `AGENTS.md` lines 58, 61, 64, 70
|
|||
|
|
- `README.md` lines 203, 206, 209, 212, 213, 216, 219, 222, 225, 228, 229, 230, 242, 245, 324, 338, 353
|
|||
|
|
- `automaton/dashboard/README.md` lines 11, 18, 21, 24
|
|||
|
|
- `prompts/orchestrate.md` lines 28, 29, 30, 36
|
|||
|
|
- Any other `python ` (bare) reference found by `rg` AFTER the first pass
|
|||
|
|
6. Do NOT change `python` references inside shell scripts that already invoke `#!/usr/bin/env python3` shebangs or that explicitly resolve via `command -v`. Only fix bare `python ` commands that shell out (none expected in scripts/ after audit, but verify).
|
|||
|
|
7. Add a CI step note to `CHANGELOG.md` under `[unreleased]` documenting the new `requirements.txt` and the `python3` requirement.
|
|||
|
|
8. Update `AGENTS.md` "Build & Test Commands" section to reference `requirements.txt` and use `python3` consistently.
|
|||
|
|
|
|||
|
|
## Acceptance criteria
|
|||
|
|
|
|||
|
|
Each must pass from a **fresh clone** with only stock macOS CommandLineTools + pip3:
|
|||
|
|
|
|||
|
|
1. `pip3 install -r requirements.txt` succeeds.
|
|||
|
|
2. `python3 -m pytest tests/ -v` exits 0 with `N passed` (N ≥ 1) and zero `error` lines.
|
|||
|
|
3. `python3 -m py_compile automaton/**/*.py automaton/dashboard/**/*.py scripts/*.py` exits 0.
|
|||
|
|
4. `bash -n scripts/*.sh` exits 0.
|
|||
|
|
5. `rg -n "^python |\"python " AGENTS.md README.md automaton/dashboard/README.md prompts/orchestrate.md` returns zero matches for a bare `python ` command.
|
|||
|
|
6. Following the install instructions in `AGENTS.md` verbatim, a new contributor can run the test suite within 60 seconds of clone.
|
|||
|
|
|
|||
|
|
## Success contract (streak)
|
|||
|
|
|
|||
|
|
Per the goal mode this task derives from, "done" requires **10 consecutive clean `python3 -m pytest tests/ -v` runs** in a row without any edit between runs. A single failure resets the counter. The cap on attempts is 5; on hitting the cap, stop and report.
|
|||
|
|
|
|||
|
|
## Constraints / non-goals
|
|||
|
|
|
|||
|
|
- No new dependencies beyond `pytest`. Do not add `pytest-cov`, `pytest-mock`, `tox`, etc.
|
|||
|
|
- No virtualenv vendoring. The user creates `.venv` themselves if they want isolation; system `pip3 install -r requirements.txt` must also work.
|
|||
|
|
- No changes to existing test logic. If a test is genuinely broken (not just import-failing because pytest is missing), STOP and report — do not patch the test to make it pass. That is the anti-spin rule from #9 in the source article.
|
|||
|
|
- Do not touch any file under `tasks/` (per-framework tasks are state, not source).
|
|||
|
|
- Do not modify `status.py`, `vram_detect.py`, or any other runtime script's behavior. Only documentation and config files change.
|
|||
|
|
- No Docker, no conda, no `pyenv` requirements. Stock `python3` + `pip3` only.
|
|||
|
|
- VRAM-aware scoping: this task fits in ONE sub-task (~9k peak context budget on this 32GB-RAM / no-GPU machine with `Model: auto`). **Do not decompose further.** Sub-tasks would exceed the budget on overhead alone.
|
|||
|
|
|
|||
|
|
## Recommended implementation approach (high-level)
|
|||
|
|
|
|||
|
|
1. Create `requirements.txt` with `pytest==7.4.4` (last 7.x; works on Python 3.9+).
|
|||
|
|
2. `pip3 install -r requirements.txt` locally and run the suite; capture every failure.
|
|||
|
|
3. For each failure, decide: import/install issue (fix dep) vs. real code bug (report, do not patch test).
|
|||
|
|
4. Sweep `python ` → `python3 ` in docs/prompts with `edit` batching.
|
|||
|
|
5. Add install snippet to `scripts/install.sh` (idempotent; only if `.venv` doesn't exist).
|
|||
|
|
6. Add a one-line test smoke-check at the end of `install.sh`: `python3 -m pytest tests/ -q || echo "tests deferred"`.
|
|||
|
|
7. Update `CHANGELOG.md` `[unreleased]`.
|
|||
|
|
8. Run the streak verifier: 10× `python3 -m pytest tests/ -v`; stop at first clean streak or 5 attempts.
|