Lap Tran
bc7daf8590
- **Restore 82 completed tasks** from tasks/complete/ back to tasks/ top level (all <7 days old per the cleanup policy; premature bulk archive was fixed). - **Dashboard: fix scroll-reset on auto-refresh** — renderBoard rebuilds the board via innerHTML every 2s, destroying each column-body's scrollTop. Now snapshots column-body scrollTop + board.scrollLeft + view.scrollTop before rebuild and restores after (matched by PHASE_GROUPS index). - **Dashboard UI additions** (pre-existing unstaged work): approval section cards, transition buttons, inline artifact editor (textarea for writing missing SPEC/VERDICT/etc from the detail modal). - **Bind ornith as Implement model** — config.md: Model explicit to omlx/Ornith-1.0-35B-4bit-mlx, context window 32768. Interactive autopilot already used ornith via opencode default; now explicit. - **Fix cleanup stub** — automaton-cleanup.sh had a stale --project arg pointing at a pytest temp dir (test isolation leak). Rewired to point at ~/.automaton. - **Fix plist-isolation test** — test asserted host plist doesn't exist, but a real install creates it. Now snapshots mtime before run, asserts unchanged after (only a write during the test counts as bleed). - **New Playwright smoke test** (tests/test_dashboard_ui.py) — 2 tests: board renders tasks, column scroll survives auto-refresh tick. Verified the test fails without the scroll fix (scrollTop resets to 0). Skipped via importorskip when playwright is absent (main CI stays green). - **Clarify SI loop scope in README** — new-project onboarding section documents the framework-scoped self-improvement loop and options (leave/pause/create project loop). - **CHANGELOG** documents all changes including the known model-divergence gap (mde tasks marked complete but per-role model binding was never implemented).
82 lines
4.9 KiB
Markdown
82 lines
4.9 KiB
Markdown
# SPEC — model-divergence-enforcement
|
|
|
|
## Problem
|
|
|
|
The framework has no computational enforcement of model divergence for conflict-of-interest roles. `status.py:1432-1436` enforces that the *session* playing reviewer differs from the implementer (via `.state.implementer`), but there is no enforcement that the *model* playing bug-finder differs from adversarial-bug-finder, or that the referee model differs from the implementer model. Same-model conflict-of-interest yields rubber-stamping.
|
|
|
|
## Design (locked)
|
|
|
|
Full design is in `design/framework/functional.md` §4 and `design/framework/technical.md` §§2-5,13. This SPEC references those docs and does not repeat them.
|
|
|
|
### Key decisions (from design session 2026-06-25)
|
|
|
|
- **Manifest**: `models.json` with `{default, advised, models:[{name, provider, context_window, location}]}`.
|
|
- **Mode detection**: 0-1 models → single-LLM (advisory once, then silent). 2+ → multi-LLM (hard block). Missing file → single-LLM (backward compatible).
|
|
- **Conflict matrix (locked)**: `code_review≠implement`; `bug_find≠implement`; `adversarial_bug_find≠implement+bug_find`; `referee≠implement+bug_find+adversarial_bug_find`; `loop-verify≠loop-implement`.
|
|
- **Auto-assignment**: default model → next-available on conflict → user override via `--transition --model` or `loop.json roles.<role>.model`. Refuse only if no non-conflicting model exists.
|
|
- **State**: `.state.models` per task recording which model filled which role.
|
|
- **Loop integration**: `loop.json` per-role `model` field + `{model}` substitution in `_invoke_harness`.
|
|
- **Audit**: new `model_divergence` category in `--audit`.
|
|
- **Dashboard**: model badges on task cards.
|
|
- **Detection**: `scripts/detect_models.py` probes opencode.json + localhost endpoints (8080/11434/1234/8000).
|
|
|
|
## Scope
|
|
|
|
This is a **parent task**. It decomposes into 3 sequential subtasks:
|
|
|
|
### Subtask 1: manifest+detection
|
|
- `models.json` schema + loader
|
|
- `scripts/detect_models.py` (probe opencode.json + localhost endpoints)
|
|
- `install.sh` / `update.sh` / `upgrade.sh` integration (run detect_models after VRAM detection)
|
|
- `config.md` `## Available Models` section
|
|
- `prompts/onboarding.md` Step 2d (model config)
|
|
- Tests: `test_model_divergence.py` (manifest loading, single vs multi mode detection, missing file backward compat)
|
|
|
|
### Subtask 2: interactive enforcement+audit
|
|
- `.state.models` schema + writer
|
|
- `status.py --transition --model <name>` (records model, checks conflict matrix in multi-LLM mode)
|
|
- `status.py --claim --model <name>` (refuses on conflict)
|
|
- `CONFLICT_MATRIX` constant + `_check_conflict` helper in `status.py`
|
|
- `status.py --audit` `model_divergence` category
|
|
- Dashboard model badges on task cards
|
|
- Tests: `test_model_divergence.py` (conflict matrix, auto-assign, audit category, dashboard badges)
|
|
|
|
### Subtask 3: loop enforcement+dashboard
|
|
- `loop.json` per-role `model` field (optional, defaults to `models.json default`)
|
|
- `loop-runner.py _invoke_harness` `{model}` substitution
|
|
- `status.py --check-gate` model-divergence check (loop-verify ≠ loop-implement in multi-LLM mode)
|
|
- Loop dashboard badges (model per role)
|
|
- Tests: `test_model_divergence.py` (loop model binding, {model} substitution, check-gate halt)
|
|
|
|
## Dependencies
|
|
|
|
- Subtask 1 → no deps (foundational)
|
|
- Subtask 2 → depends on subtask 1 (needs manifest loader)
|
|
- Subtask 3 → depends on subtask 2 (needs .state.models + conflict matrix)
|
|
|
|
## Out of scope
|
|
|
|
- Rule agents (FW-2, FW-3) — separate backlog items that *consume* this feature
|
|
- Agent tab redesign (FW-1) — separate backlog item, no dependency on this task
|
|
- Model capability inspection (D8: framework never inspects capability/size/provider)
|
|
- `detect_models.py` auto-writing `models.json` without user confirmation (detection is advisory; user confirms the manifest)
|
|
|
|
## Success criteria
|
|
|
|
1. `models.json` missing → single-LLM mode, all model commands are no-ops, existing behavior unchanged.
|
|
2. `models.json` with 1 model → single-LLM mode, advisory printed once if `advised: true`, then silent.
|
|
3. `models.json` with 2+ models → multi-LLM mode, conflict matrix enforced on `--transition --model` and `--claim --model`.
|
|
4. `--audit` flags conflict-matrix violations as `model_divergence` category.
|
|
5. `loop.json` per-role model binding works; `--check-gate` halts on loop-verify = loop-implement in multi-LLM mode.
|
|
6. `detect_models.py` probes opencode.json + localhost endpoints and emits a candidate manifest.
|
|
7. `pytest tests/ -v` green; no regressions.
|
|
|
|
## References
|
|
|
|
- `design/framework/functional.md` §4 (Model-Divergence Enforcement)
|
|
- `design/framework/technical.md` §§2-5 (manifest, state, flags, conflict matrix), §13 (loop integration)
|
|
- `design/framework/README.md` (locked decisions F4, F5)
|
|
- `.agent.md` Agent Configuration (6 phase roles)
|
|
- `scripts/status.py:1432-1436` (existing session-level reviewer≠implementer enforcement)
|
|
- `scripts/loop-runner.py:366-404` (`_invoke_harness`, needs `{model}` substitution)
|