Lap Tran
bc7daf8590
- **Restore 82 completed tasks** from tasks/complete/ back to tasks/ top level (all <7 days old per the cleanup policy; premature bulk archive was fixed). - **Dashboard: fix scroll-reset on auto-refresh** — renderBoard rebuilds the board via innerHTML every 2s, destroying each column-body's scrollTop. Now snapshots column-body scrollTop + board.scrollLeft + view.scrollTop before rebuild and restores after (matched by PHASE_GROUPS index). - **Dashboard UI additions** (pre-existing unstaged work): approval section cards, transition buttons, inline artifact editor (textarea for writing missing SPEC/VERDICT/etc from the detail modal). - **Bind ornith as Implement model** — config.md: Model explicit to omlx/Ornith-1.0-35B-4bit-mlx, context window 32768. Interactive autopilot already used ornith via opencode default; now explicit. - **Fix cleanup stub** — automaton-cleanup.sh had a stale --project arg pointing at a pytest temp dir (test isolation leak). Rewired to point at ~/.automaton. - **Fix plist-isolation test** — test asserted host plist doesn't exist, but a real install creates it. Now snapshots mtime before run, asserts unchanged after (only a write during the test counts as bleed). - **New Playwright smoke test** (tests/test_dashboard_ui.py) — 2 tests: board renders tasks, column scroll survives auto-refresh tick. Verified the test fails without the scroll fix (scrollTop resets to 0). Skipped via importorskip when playwright is absent (main CI stays green). - **Clarify SI loop scope in README** — new-project onboarding section documents the framework-scoped self-improvement loop and options (leave/pause/create project loop). - **CHANGELOG** documents all changes including the known model-divergence gap (mde tasks marked complete but per-role model binding was never implemented).
78 lines
3.5 KiB
Markdown
78 lines
3.5 KiB
Markdown
# Bug Report: harden-parse-verdict
|
|
|
|
Bug_find phase observations. Each non-blocking unless marked BLOCKER.
|
|
|
|
## O1 — `bool(raw_pass.strip())` fallback for "0" / "1" strings
|
|
|
|
For `{"pass": "0", "score": 0.5}`, the new path strips → "0" → not "true"/
|
|
"false" → `bool("0")` = True (non-empty string is truthy).
|
|
|
|
Pre-fix: `bool("0")` was also True (same).
|
|
|
|
This is **consistent with v1** — no behavior change. A verifier emitting
|
|
`pass: "0"` intending "false" gets `True` in v1 AND in the hardened
|
|
implementation. The SPEC says non-`true`/`false` strings fall through to
|
|
`bool(...)`, which is truthy for non-empty. Not a regression.
|
|
|
|
**Not a bug** — documented behavior per SPEC R1 + D-V1. If users want
|
|
strict numeric-string handling, that's a separate future task (out of
|
|
scope for v1.1 harden-parse-verdict).
|
|
|
|
## O2 — `float("nan")` serializes back as `NaN` to `.state.loop`
|
|
|
|
When the verifier emits `NaN` as the score, `parse_verdict` returns
|
|
`score=0.5` (without writing to disk by itself). But this score is part
|
|
of `verdict` which gets written via `_write_state_loop(loop_path, state)`
|
|
into `.state.loop` as JSON. Since the clamp converts NaN to 0.5 BEFORE
|
|
the verdict is stored, `.state.loop` gets `0.5`, not `NaN`. No NaN leaks
|
|
into the loop state.
|
|
|
|
**Not a bug** — confirmed via tracing: `parse_verdict` returns the clamped
|
|
dict; `cmd_tick` then stores `state["last_verdict"] = verdict` (a clean
|
|
dict with `score: 0.5`); the JSON round-trip is clean.
|
|
|
|
## O3 — `_gate_score_plateau`'s threshold unaffected
|
|
|
|
`_gate_score_plateau` (status.py) reads `score_history` and decides a halt
|
|
when last N scores are within some delta. With clamped scores, the plateau
|
|
detection range is now strictly `[0, 1]` instead of `[any, any]`. Brief
|
|
review:
|
|
|
|
- Pre-fix: a verifier could emit `score: 1.5` across N ticks; plateau
|
|
detection sees a flat line at 1.5; halt fires. Expected behavior.
|
|
- Post-fix: the same verifier's 1.5 clamps to 1.0 across N ticks; plateau
|
|
sees flat line at 1.0; halt fires. Same outcome.
|
|
|
|
- Pre-fix: a verifier emits alternating `0.9` and `1.1`; plateau sees a
|
|
bimodal history [0.9, 1.1, 0.9, 1.1] — NOT plateau (variation > epsilon).
|
|
- Post-fix: alternating `0.9` and `1.0` (1.1 clamps to 1.0); plateau sees
|
|
[0.9, 1.0, 0.9, 1.0] — still variation above a small epsilon — still NOT
|
|
plateau. Same outcome in this scenario.
|
|
|
|
Edge case: a verifier emits all `1.0` and `0.99` (vs 1.0 and 1.0 clamped).
|
|
The clamp DOES change plateau detection in this case — `1.0 1.0 1.0`
|
|
looks more plateau-like than `1.0 0.99 0.99`. Could cause halt earlier than
|
|
prior. Documented as a desirable side effect (clamp reduces the verifier's
|
|
untrustworthiness from inflating scores; plateau detection is more
|
|
honest).
|
|
|
|
**Not a bug** — improved behavior. Documented in CHANGELOG.
|
|
|
|
## O4 — Verifier prompt hasn't been updated
|
|
|
|
`prompts/loop-verifier.md` still asks the model to emit JSON with
|
|
`"pass": true/false` and `"score": 0.0-1.0`. The runner now defensive-coerces,
|
|
but the prompt's contract is unchanged. Was the prompt already
|
|
JSON-typed-booleans-only? Let me check.
|
|
|
|
**`prompts/loop-verifier.md` review**: still says "Output: strict JSON, no
|
|
prose" with example shape. The prompt explicitly tells the model to emit
|
|
JSON booleans — no mention of string-typed `pass`. So the v1 contract was
|
|
strict; the O6 finding was a defense-in-depth concern, not a present-fault.
|
|
The harden task adds belt-and-suspenders without changing the contract.
|
|
|
|
**Not a bug** — the prompt remains authoritative. No edit needed.
|
|
|
|
## Verdict
|
|
|
|
**No blockers.** Proceed to adversarial_bug_find. |