Files

T

Lap Tran bc7daf8590 Restore archived tasks, fix dashboard scroll-reset, bind ornith, add Playwright smoke test

- **Restore 82 completed tasks** from tasks/complete/ back to tasks/ top
  level (all <7 days old per the cleanup policy; premature bulk archive
  was fixed).
- **Dashboard: fix scroll-reset on auto-refresh** — renderBoard rebuilds
  the board via innerHTML every 2s, destroying each column-body's
  scrollTop. Now snapshots column-body scrollTop + board.scrollLeft +
  view.scrollTop before rebuild and restores after (matched by
  PHASE_GROUPS index).
- **Dashboard UI additions** (pre-existing unstaged work): approval
  section cards, transition buttons, inline artifact editor (textarea for
  writing missing SPEC/VERDICT/etc from the detail modal).
- **Bind ornith as Implement model** — config.md: Model explicit to
  omlx/Ornith-1.0-35B-4bit-mlx, context window 32768. Interactive
  autopilot already used ornith via opencode default; now explicit.
- **Fix cleanup stub** — automaton-cleanup.sh had a stale --project arg
  pointing at a pytest temp dir (test isolation leak). Rewired to point
  at ~/.automaton.
- **Fix plist-isolation test** — test asserted host plist doesn't exist,
  but a real install creates it. Now snapshots mtime before run, asserts
  unchanged after (only a write during the test counts as bleed).
- **New Playwright smoke test** (tests/test_dashboard_ui.py) — 2 tests:
  board renders tasks, column scroll survives auto-refresh tick.
  Verified the test fails without the scroll fix (scrollTop resets to 0).
  Skipped via importorskip when playwright is absent (main CI stays
  green).
- **Clarify SI loop scope in README** — new-project onboarding section
  documents the framework-scoped self-improvement loop and options
  (leave/pause/create project loop).
- **CHANGELOG** documents all changes including the known model-divergence
  gap (mde tasks marked complete but per-role model binding was never
  implemented).

2026-06-26 10:05:18 -04:00

4.3 KiB

Raw Permalink Blame History

Framework Self-Consistency Tests

Goal

Add automated tests that enforce the framework's own rules — verifying prompt consistency, stop condition presence, canonical paths, and state machine integrity. These tests lock in the fixes from prior tasks and prevent regression.

Requirements

R1. Add `tests/test_framework_self_consistency.py`

Create a new test file that performs compile-time checks on the framework itself:

A. All delivery prompts have a stop condition block

Assert every .md file in prompts/ that produces a deliverable artifact contains ## Stop Condition (MANDATORY) or a documented equivalent
Exclude orchestrate.md (not a delivery prompt), compaction.md (uses COMPACTION_COMPLETE), adversarial_bug_find.md (now has CONTRACT_MET), workflow.md (reference, not prompt)

B. No hardcoded repository URLs in prompts/contracts/templates

Assert that prompt files don't contain the Gitea URL (10.37.0.86:3003) or any other hardcoded repo URL
install.sh at line 13 is the only allowed location (the install script legitimately needs it)

C. .rules.md contains all mandatory rule sections

Assert .rules.md mentions: Task-Driven Development, VRAM, Changelog, Session Discipline, Scope Confinement, Artifact Integrity

D. .rules.md self-improvement rule has concrete examples

Assert the Self-Improvement section references at least one real failure mode

E. Canonical task path used in all prompts

Assert all {project}/.automaton/tasks/{task-name}/ paths match the canonical format
Assert zero instances of {project}/tasks/ (the deprecated location) — including concrete task names like {project}/tasks/onboarding/

F. pyproject.toml has no stale extras

Assert pyproject.toml does not reference inotify

G. Dashboard CSS theme variables are complete

Assert both :root and [data-theme="light"] sections contain the same set of CSS variable names
This prevents the common bug where a variable is added to one theme but not the other

R2. Add a minimal JS logic test

dashboard.js has 470 lines of untested UI logic. At minimum, test the pure functions:

getTaskDisplayGroup() — review-based group advancement
getFilteredTasks() — filter/sort behavior
STATE_ICONS map completeness (matches TaskState values)

This can be done in Python by parsing the JS file and extracting the function logic, or by adding a small Node.js test with jsdom.

R3. Add verdict parsing regression test

Create a dedicated test file tests/test_parsing.py (or extend test_task.py) with:

PASS verdict mentioning FAIL → DONE (not BLOCKED) — the regression test for the critical bug
PASS verdict mentioning NEEDS_REVIEW → DONE (not BLOCKED)
Structured verdict with ## Status: PASS → DONE
Structured verdict with ## Status: FAIL → BLOCKED
Unstructured verdict with just "FAIL" → BLOCKED (fallback behavior)
Verdict with no status line → RESEARCH (since no SPEC either) or BACKLOG
IMPLEMENTATION.md alone → BUG_FIND (state machine alignment)
Empty VERDICT.md → BLOCKED

R4. CI configuration validation

Add a test that parses .gitea/workflows/ci.yml and asserts:

It runs py_compile on all Python source directories
It runs pytest
It runs bash -n on shell scripts

This catches the case where a new directory is added but CI isn't updated.

Acceptance Criteria

python -m pytest tests/test_framework_self_consistency.py -v passes
All delivery prompts have stop condition blocks (tested by R1.A)
No hardcoded URLs in prompts (tested by R1.B)
.rules.md contains all mandatory sections (tested by R1.C)
Zero deprecated {project}/tasks/ paths in prompts (tested by R1.E)
pyproject.toml has no inotify reference (tested by R1.F)
tests/test_parsing.py includes all regression cases from R3
All existing tests still pass (72/72 minimum)
CI workflow correctly includes all framework source directories (tested by R4)

Non-Goals

Not adding a linter/formatter (ruff/black) — framework policy doesn't require one
Not adding mypy type checking
Not changing the soft-enforcement philosophy — these tests verify prompts and docs, not runtime behavior
Not testing the dashboard server integration (too heavy for unit tests)

4.3 KiB Raw Permalink Blame History