Bootstrap self-improvement loop, decompose model-divergence-enforcement
CI / build (push) Has been cancelled
CI / build (push) Has been cancelled
Self-improvement loop:
- Created via --create-loop --from-template self-improvement
- Scheduled via launchd (3600s interval)
- State: running
model-divergence-enforcement task:
- Research approved, decomposed into 3 sequential subtasks:
1. mde-manifest-detection (models.json + detect_models.py)
2. mde-interactive-enforcement (conflict matrix + --model args + audit)
3. mde-loop-enforcement (loop.json roles + {model} substitution + check-gate)
- Parent at decomposition:approved (stays active until subtasks complete)
- Each subtask has BRIEF.md with scope, deliverables, acceptance criteria
This commit is contained in:
Lap Tran
@@ -0,0 +1 @@
|
||||
decomposition:approved
|
||||
@@ -0,0 +1,2 @@
|
||||
research:approved|2026-06-25T11:11:51.910546+00:00|user
|
||||
decomposition:approved|2026-06-25T11:12:40.642310+00:00|user
|
||||
@@ -0,0 +1,113 @@
|
||||
# DECOMPOSITION — model-divergence-enforcement
|
||||
|
||||
## Method
|
||||
|
||||
Decompose by **dependency layer**, not by file. Each subtask builds on the previous
|
||||
one's foundation. The SPEC (`tasks/model-divergence-enforcement/SPEC.md`) defines 3
|
||||
sequential subtasks with strict dependency ordering.
|
||||
|
||||
## Sub-tasks (3, sequential)
|
||||
|
||||
### subtask-1: `mde-manifest-detection`
|
||||
**Scope:** `models.json` schema + loader + `scripts/detect_models.py` probe + install integration.
|
||||
**Files touched:**
|
||||
- `scripts/detect_models.py` (new — probe opencode.json providers + localhost endpoints 8080/11434/1234/8000)
|
||||
- `scripts/status.py` (add `_load_models_manifest()` helper, `_get_mode()` — single vs multi-LLM)
|
||||
- `scripts/install.sh` (call `detect_models.py` after `vram_detect.py`)
|
||||
- `scripts/update.sh` (same)
|
||||
- `scripts/upgrade.sh` (same)
|
||||
- `config.md` (add `## Available Models` section template)
|
||||
- `prompts/onboarding.md` (Step 2d — model config check)
|
||||
- `tests/test_model_divergence.py` (new — manifest loading, single vs multi mode, missing file backward compat)
|
||||
**Not touched:** `status.py --transition`, `--claim`, `--audit`, `loop-runner.py`, dashboard code.
|
||||
**Acceptance:**
|
||||
1. `models.json` missing → `_get_mode()` returns `"single"`, all model commands are no-ops.
|
||||
2. `models.json` with 0-1 models → `_get_mode()` returns `"single"`.
|
||||
3. `models.json` with 2+ models → `_get_mode()` returns `"multi"`.
|
||||
4. `detect_models.py` probes localhost endpoints and prints a candidate manifest (JSON to stdout).
|
||||
5. `pytest tests/test_model_divergence.py -v` green.
|
||||
6. `pytest tests/ -q` green (no regressions).
|
||||
**Peak context estimate:** ~6k tokens (new script + status.py helper + tests).
|
||||
**Run order:** first. Foundational — subtasks 2 and 3 depend on this.
|
||||
|
||||
### subtask-2: `mde-interactive-enforcement`
|
||||
**Scope:** `.state.models` schema + conflict matrix + `--transition --model` / `--claim --model` enforcement + audit category + dashboard badges.
|
||||
**Depends on:** subtask-1 (needs `_load_models_manifest()` and `_get_mode()`).
|
||||
**Files touched:**
|
||||
- `scripts/status.py`:
|
||||
- `CONFLICT_MATRIX` constant (locked matrix from SPEC §15)
|
||||
- `_check_conflict(current_phase, current_model, next_phase, next_model)` helper
|
||||
- `cmd_transition`: add `--model` arg; in multi-LLM mode, check conflict matrix before allowing transition
|
||||
- `cmd_claim`: add `--model` arg; refuse if model conflicts with existing `.state.models` entry
|
||||
- `cmd_audit`: add `model_divergence` category (scan `.state.models` for violations)
|
||||
- `.state.models` writer (JSON: `{implementer: "model-name", code_reviewer: "model-name", ...}`)
|
||||
- `automaton/dashboard/html/dashboard.js`: model badge on task cards (read from `.state.models`)
|
||||
- `automaton/dashboard/ui/app.py`: include `.state.models` in `/api/tasks` response
|
||||
- `tests/test_model_divergence.py`: conflict matrix tests, auto-assign tests, audit category tests, dashboard badge tests
|
||||
**Not touched:** `loop-runner.py`, `loop.json` schema, `--check-gate`.
|
||||
**Acceptance:**
|
||||
1. Single-LLM mode: `--transition --model <name>` records model but never refuses. Advisory printed once if `advised: true`.
|
||||
2. Multi-LLM mode: `--transition --model <name>` refuses if `<name>` conflicts with `.state.models` for a conflicting phase.
|
||||
3. Multi-LLM mode: `--claim --model <name>` refuses on conflict.
|
||||
4. `--audit` flags `model_divergence` violations (e.g., same model in implementer + code_reviewer).
|
||||
5. Dashboard task cards show model badges when `.state.models` exists.
|
||||
6. `pytest tests/test_model_divergence.py -v` green.
|
||||
7. `pytest tests/ -q` green (no regressions).
|
||||
**Peak context estimate:** ~8k tokens (status.py surgery + dashboard + tests).
|
||||
**Run order:** second, AFTER subtask-1.
|
||||
|
||||
### subtask-3: `mde-loop-enforcement`
|
||||
**Scope:** `loop.json` per-role model field + `{model}` substitution in loop-runner + `--check-gate` model-divergence brake.
|
||||
**Depends on:** subtask-2 (needs `CONFLICT_MATRIX` and `_check_conflict`).
|
||||
**Files touched:**
|
||||
- `scripts/loop-runner.py`:
|
||||
- `_invoke_harness` (line 366-404): add `{model}` placeholder substitution from `loop.json` role config
|
||||
- `_find_work_backlog`: no change (already supports `work_source.area`)
|
||||
- `scripts/status.py`:
|
||||
- `--check-gate`: add model-divergence brake gate (loop-verify model ≠ loop-implement model in multi-LLM mode)
|
||||
- `cmd_install_schedule` / `cmd_create_loop`: validate `loop.json` per-role `model` fields against `models.json`
|
||||
- `templates/loops/`: update loop templates with `roles` schema example
|
||||
- `design/loops/technical.md`: document `{model}` substitution
|
||||
- `tests/test_model_divergence.py`: loop model binding tests, `{model}` substitution tests, check-gate halt tests
|
||||
**Not touched:** interactive `--transition` / `--claim` (already done in subtask-2), dashboard badges (already done in subtask-2).
|
||||
**Acceptance:**
|
||||
1. `loop.json` with `roles.implementer.model: "llama-3.3-70b"` → `_invoke_harness` substitutes `{model}` in harness command.
|
||||
2. `loop.json` without per-role `model` → defaults to `models.json` `default` model.
|
||||
3. `--check-gate` in multi-LLM mode halts loop if loop-verify model = loop-implement model.
|
||||
4. `--check-gate` in single-LLM mode does NOT halt (advisory only).
|
||||
5. `pytest tests/test_model_divergence.py -v` green.
|
||||
6. `pytest tests/ -q` green (no regressions).
|
||||
**Peak context estimate:** ~6k tokens (loop-runner + check-gate + tests).
|
||||
**Run order:** third, AFTER subtask-2.
|
||||
|
||||
## Dependency graph
|
||||
|
||||
```
|
||||
subtask-1 (manifest+detection)
|
||||
│
|
||||
▼
|
||||
subtask-2 (interactive enforcement+audit)
|
||||
│
|
||||
▼
|
||||
subtask-3 (loop enforcement+dashboard)
|
||||
│
|
||||
▼
|
||||
parent model-divergence-enforcement → complete
|
||||
```
|
||||
|
||||
Parent is complete only when ALL three subtasks pass their acceptance criteria AND
|
||||
`pytest tests/ -q` is green.
|
||||
|
||||
## Parent non-goals
|
||||
|
||||
- No rule agents (FW-2, FW-3) — separate backlog items that *consume* this feature.
|
||||
- No agent tab redesign (FW-1) — separate backlog item, no dependency on this task.
|
||||
- No model capability inspection — framework never inspects capability/size/provider (decision D8).
|
||||
- No `detect_models.py` auto-writing `models.json` — detection is advisory; user confirms the manifest.
|
||||
|
||||
## Fallback
|
||||
|
||||
If any subtask hits a blocker (e.g., `status.py` surgery is too large for context budget),
|
||||
it must report back to the Orchestrator via `human_intervention` rather than skipping
|
||||
enforcement logic. Partial enforcement is worse than no enforcement — it creates a
|
||||
false sense of security.
|
||||
@@ -0,0 +1,81 @@
|
||||
# SPEC — model-divergence-enforcement
|
||||
|
||||
## Problem
|
||||
|
||||
The framework has no computational enforcement of model divergence for conflict-of-interest roles. `status.py:1432-1436` enforces that the *session* playing reviewer differs from the implementer (via `.state.implementer`), but there is no enforcement that the *model* playing bug-finder differs from adversarial-bug-finder, or that the referee model differs from the implementer model. Same-model conflict-of-interest yields rubber-stamping.
|
||||
|
||||
## Design (locked)
|
||||
|
||||
Full design is in `design/framework/functional.md` §4 and `design/framework/technical.md` §§2-5,13. This SPEC references those docs and does not repeat them.
|
||||
|
||||
### Key decisions (from design session 2026-06-25)
|
||||
|
||||
- **Manifest**: `models.json` with `{default, advised, models:[{name, provider, context_window, location}]}`.
|
||||
- **Mode detection**: 0-1 models → single-LLM (advisory once, then silent). 2+ → multi-LLM (hard block). Missing file → single-LLM (backward compatible).
|
||||
- **Conflict matrix (locked)**: `code_review≠implement`; `bug_find≠implement`; `adversarial_bug_find≠implement+bug_find`; `referee≠implement+bug_find+adversarial_bug_find`; `loop-verify≠loop-implement`.
|
||||
- **Auto-assignment**: default model → next-available on conflict → user override via `--transition --model` or `loop.json roles.<role>.model`. Refuse only if no non-conflicting model exists.
|
||||
- **State**: `.state.models` per task recording which model filled which role.
|
||||
- **Loop integration**: `loop.json` per-role `model` field + `{model}` substitution in `_invoke_harness`.
|
||||
- **Audit**: new `model_divergence` category in `--audit`.
|
||||
- **Dashboard**: model badges on task cards.
|
||||
- **Detection**: `scripts/detect_models.py` probes opencode.json + localhost endpoints (8080/11434/1234/8000).
|
||||
|
||||
## Scope
|
||||
|
||||
This is a **parent task**. It decomposes into 3 sequential subtasks:
|
||||
|
||||
### Subtask 1: manifest+detection
|
||||
- `models.json` schema + loader
|
||||
- `scripts/detect_models.py` (probe opencode.json + localhost endpoints)
|
||||
- `install.sh` / `update.sh` / `upgrade.sh` integration (run detect_models after VRAM detection)
|
||||
- `config.md` `## Available Models` section
|
||||
- `prompts/onboarding.md` Step 2d (model config)
|
||||
- Tests: `test_model_divergence.py` (manifest loading, single vs multi mode detection, missing file backward compat)
|
||||
|
||||
### Subtask 2: interactive enforcement+audit
|
||||
- `.state.models` schema + writer
|
||||
- `status.py --transition --model <name>` (records model, checks conflict matrix in multi-LLM mode)
|
||||
- `status.py --claim --model <name>` (refuses on conflict)
|
||||
- `CONFLICT_MATRIX` constant + `_check_conflict` helper in `status.py`
|
||||
- `status.py --audit` `model_divergence` category
|
||||
- Dashboard model badges on task cards
|
||||
- Tests: `test_model_divergence.py` (conflict matrix, auto-assign, audit category, dashboard badges)
|
||||
|
||||
### Subtask 3: loop enforcement+dashboard
|
||||
- `loop.json` per-role `model` field (optional, defaults to `models.json default`)
|
||||
- `loop-runner.py _invoke_harness` `{model}` substitution
|
||||
- `status.py --check-gate` model-divergence check (loop-verify ≠ loop-implement in multi-LLM mode)
|
||||
- Loop dashboard badges (model per role)
|
||||
- Tests: `test_model_divergence.py` (loop model binding, {model} substitution, check-gate halt)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Subtask 1 → no deps (foundational)
|
||||
- Subtask 2 → depends on subtask 1 (needs manifest loader)
|
||||
- Subtask 3 → depends on subtask 2 (needs .state.models + conflict matrix)
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Rule agents (FW-2, FW-3) — separate backlog items that *consume* this feature
|
||||
- Agent tab redesign (FW-1) — separate backlog item, no dependency on this task
|
||||
- Model capability inspection (D8: framework never inspects capability/size/provider)
|
||||
- `detect_models.py` auto-writing `models.json` without user confirmation (detection is advisory; user confirms the manifest)
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. `models.json` missing → single-LLM mode, all model commands are no-ops, existing behavior unchanged.
|
||||
2. `models.json` with 1 model → single-LLM mode, advisory printed once if `advised: true`, then silent.
|
||||
3. `models.json` with 2+ models → multi-LLM mode, conflict matrix enforced on `--transition --model` and `--claim --model`.
|
||||
4. `--audit` flags conflict-matrix violations as `model_divergence` category.
|
||||
5. `loop.json` per-role model binding works; `--check-gate` halts on loop-verify = loop-implement in multi-LLM mode.
|
||||
6. `detect_models.py` probes opencode.json + localhost endpoints and emits a candidate manifest.
|
||||
7. `pytest tests/ -v` green; no regressions.
|
||||
|
||||
## References
|
||||
|
||||
- `design/framework/functional.md` §4 (Model-Divergence Enforcement)
|
||||
- `design/framework/technical.md` §§2-5 (manifest, state, flags, conflict matrix), §13 (loop integration)
|
||||
- `design/framework/README.md` (locked decisions F4, F5)
|
||||
- `.agent.md` Agent Configuration (6 phase roles)
|
||||
- `scripts/status.py:1432-1436` (existing session-level reviewer≠implementer enforcement)
|
||||
- `scripts/loop-runner.py:366-404` (`_invoke_harness`, needs `{model}` substitution)
|
||||
@@ -0,0 +1 @@
|
||||
new
|
||||
@@ -0,0 +1,34 @@
|
||||
# BRIEF — mde-interactive-enforcement
|
||||
|
||||
**Parent:** model-divergence-enforcement
|
||||
**Subtask:** 2 of 3 (depends on mde-manifest-detection)
|
||||
**Spec ref:** `tasks/model-divergence-enforcement/SPEC.md` §Subtask 2
|
||||
**Design ref:** `design/framework/technical.md` §§4-5, §12
|
||||
|
||||
## Objective
|
||||
|
||||
Add the conflict matrix, `--transition --model` / `--claim --model` enforcement,
|
||||
`.state.models` tracking, audit category, and dashboard model badges. This is the
|
||||
core enforcement layer for interactive (non-loop) workflows.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. `scripts/status.py`:
|
||||
- `CONFLICT_MATRIX` constant (locked matrix from SPEC)
|
||||
- `_check_conflict(current_phase, current_model, next_phase, next_model)` helper
|
||||
- `cmd_transition`: `--model` arg + conflict check in multi-LLM mode
|
||||
- `cmd_claim`: `--model` arg + conflict check
|
||||
- `cmd_audit`: `model_divergence` category
|
||||
- `.state.models` writer (JSON: `{implementer, code_reviewer, bug_hunter, ...}`)
|
||||
2. `automaton/dashboard/html/dashboard.js` — model badge on task cards
|
||||
3. `automaton/dashboard/ui/app.py` — `.state.models` in `/api/tasks` response
|
||||
4. `tests/test_model_divergence.py` — conflict matrix, auto-assign, audit, badges
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Single-LLM mode: `--model` records but never refuses (advisory once if `advised: true`)
|
||||
- Multi-LLM mode: `--transition --model` refuses on conflict (e.g., same model for implement + code_review)
|
||||
- Multi-LLM mode: `--claim --model` refuses on conflict
|
||||
- `--audit` flags `model_divergence` violations
|
||||
- Dashboard shows model badges when `.state.models` exists
|
||||
- `pytest tests/ -q` green
|
||||
@@ -0,0 +1 @@
|
||||
new
|
||||
@@ -0,0 +1,31 @@
|
||||
# BRIEF — mde-loop-enforcement
|
||||
|
||||
**Parent:** model-divergence-enforcement
|
||||
**Subtask:** 3 of 3 (depends on mde-interactive-enforcement)
|
||||
**Spec ref:** `tasks/model-divergence-enforcement/SPEC.md` §Subtask 3
|
||||
**Design ref:** `design/framework/technical.md` §§10, §13
|
||||
|
||||
## Objective
|
||||
|
||||
Add loop-level model-divergence enforcement: `loop.json` per-role `model` field,
|
||||
`{model}` substitution in `_invoke_harness`, and `--check-gate` model-divergence
|
||||
brake gate.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. `scripts/loop-runner.py`:
|
||||
- `_invoke_harness` (line 366-404): `{model}` placeholder substitution from `loop.json` role config
|
||||
2. `scripts/status.py`:
|
||||
- `--check-gate`: model-divergence brake gate (loop-verify ≠ loop-implement in multi-LLM mode)
|
||||
- `cmd_create_loop` / `cmd_install_schedule`: validate `loop.json` per-role `model` fields
|
||||
3. `templates/loops/`: update loop templates with `roles` schema example
|
||||
4. `design/loops/technical.md`: document `{model}` substitution
|
||||
5. `tests/test_model_divergence.py`: loop model binding, `{model}` substitution, check-gate halt
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `loop.json` with `roles.implementer.model` → `_invoke_harness` substitutes `{model}` in harness command
|
||||
- `loop.json` without per-role `model` → defaults to `models.json` `default` model
|
||||
- `--check-gate` in multi-LLM mode halts if loop-verify model = loop-implement model
|
||||
- `--check-gate` in single-LLM mode does NOT halt (advisory only)
|
||||
- `pytest tests/ -q` green
|
||||
@@ -0,0 +1 @@
|
||||
new
|
||||
@@ -0,0 +1,30 @@
|
||||
# BRIEF — mde-manifest-detection
|
||||
|
||||
**Parent:** model-divergence-enforcement
|
||||
**Subtask:** 1 of 3 (foundational, no deps)
|
||||
**Spec ref:** `tasks/model-divergence-enforcement/SPEC.md` §Subtask 1
|
||||
**Design ref:** `design/framework/technical.md` §§2-3
|
||||
|
||||
## Objective
|
||||
|
||||
Create the model manifest system (`models.json` schema + loader) and model detection
|
||||
script (`scripts/detect_models.py`). This is the foundation for all model-divergence
|
||||
enforcement — subtasks 2 and 3 depend on it.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. `scripts/detect_models.py` — probes opencode.json providers + localhost endpoints
|
||||
(8080/11434/1234/8000), emits candidate manifest as JSON to stdout
|
||||
2. `scripts/status.py` — add `_load_models_manifest()` and `_get_mode()` helpers
|
||||
3. `scripts/install.sh`, `update.sh`, `upgrade.sh` — call detect_models after vram_detect
|
||||
4. `config.md` — `## Available Models` section template
|
||||
5. `prompts/onboarding.md` — Step 2d model config check
|
||||
6. `tests/test_model_divergence.py` — manifest loading, single vs multi mode, missing file
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `models.json` missing → `_get_mode()` returns `"single"`, all model commands are no-ops
|
||||
- `models.json` with 0-1 models → `"single"` mode
|
||||
- `models.json` with 2+ models → `"multi"` mode
|
||||
- `detect_models.py` probes localhost and prints candidate JSON
|
||||
- `pytest tests/ -q` green
|
||||
Reference in New Issue
Block a user