Fix 10 audit bugs: path prefix matching, verdict parsing, CORS, stale-task detection, phase mapping
CI / build (push) Has been cancelled
CI / build (push) Has been cancelled
Batch 1 (High severity): - Bug 1: --audit cat3 now checks .automaton/tasks/ paths - Bug 4: Verdict PASS/FAIL uses structured ## Status: line parsing - Bug 5: register-guards.sh checks .json/.jsonc, writes plugin key, strips comments - Bug 7: --can-edit/--scope-check path prefix uses os.sep boundary Batch 2 (Medium/Low severity): - Bug 2: migrate-project.sh find command parentheses for -prune binding - Bug 3: vram_detect model prefix matching with known-suffix whitelist - Bug 6: dashboard reads .state file before artifact heuristic fallback - Bug 8: removed wildcard CORS, added security headers (nosniff, DENY) - Bug 9: stale-task detection uses .state.lastedit instead of .state mtime - Bug 10: TEST_PLAN.md maps to test_design (was implement) 249 tests pass (up from 235). All 10 tasks driven through full workflow to completion.
This commit is contained in:
+32
-52
@@ -1,72 +1,52 @@
|
||||
# Verdict: automaton
|
||||
# Verdict: Full-Codebase Audit (v2.0)
|
||||
|
||||
## Status: PASS
|
||||
**Completion Date**: 2026-06-11
|
||||
## Status: NEEDS_REVIEW
|
||||
**Completion Date**: 2026-06-22
|
||||
|
||||
## Summary
|
||||
|
||||
automaton has a **score of 78** from the Bug Finder and **65** from the Adversarial Bug Finder, for a combined score of **143**. All 30 bugs have been fixed. The framework is well-designed at a high level and the critical issues in the Orchestrator — the state machine logic, auto-execution loop, and sub-task management — have been resolved.
|
||||
Full-codebase audit of the automaton framework v2.0. The Bug Finder identified 7 bugs (score: 55); the Adversarial Bug Finder identified 3 additional bugs (score: 7). No overlapping or contradictory findings. The framework's core state machine, phase transitions, and approval gates work correctly. However, several bugs exist in the enforcement code itself (`status.py`), the guard registration script, and the dashboard — undermining the "state-enforced" guarantee in specific scenarios.
|
||||
|
||||
## Findings
|
||||
|
||||
### What Passed
|
||||
- The overall framework architecture is sound — the state machine, phase separation, and artifact-based progression are well-designed
|
||||
- The interactive protocol for Research and Design phases is robust
|
||||
- The VRAM configuration and detection system is comprehensive
|
||||
- The bug finder and adversarial bug finder prompts are thorough and well-structured
|
||||
- The doc review phase fills a genuine gap in the workflow
|
||||
- **All 30 bugs have been fixed**
|
||||
- Core state machine: `.state` file reading, `_base_phase()` approval substate handling, and `--transition` validation are correct
|
||||
- `_check_forbidden_artifacts()` correctly detects out-of-order artifacts per phase
|
||||
- Dashboard static file serving has proper path-traversal protection (`resolve()` + `relative_to()`)
|
||||
- Dashboard review file writing validates task names against `[A-Za-z0-9_-]+`
|
||||
- POST body size limits (64KB) and review comment length limits (4096) are enforced
|
||||
- Test suite: 235 tests pass
|
||||
- Git hooks (pre-commit, post-commit, pre-push) have correct logic
|
||||
|
||||
### What Failed
|
||||
- **None** — all bugs have been fixed
|
||||
- **Bug 1** (High): Category 3 audit is broken for ALL regular (non-framework) projects — task folder changes are always flagged as unauthorized
|
||||
- **Bug 4** (High): Verdict inference uses `"PASS" in content` substring search, which can misclassify FAIL/NEEDS_REVIEW verdicts as complete
|
||||
- **Bug 5** (High): `register-guards.sh` never successfully registers the OpenCode guard due to 3 compounding bugs (wrong filename, wrong config key, JSONC parsing)
|
||||
- **Bug 7** (High): `--can-edit` file scope check uses string `startswith` without trailing separator, allowing edits in sibling directories
|
||||
|
||||
### What Needs Review
|
||||
- **None** — all bugs have been fixed
|
||||
- **Bug 2** (Medium): `migrate-project.sh` `find` precedence silently skips `.md` files during migration
|
||||
- **Bug 3** (Medium): `vram_detect.py` prefix matching gives wrong context windows for unknown models
|
||||
- **Bug 6** (Medium): Dashboard ignores `.state` files, contradicting the "single source of truth" design
|
||||
- **Bug 8** (Medium): Dashboard CORS `*` allows cross-origin writes from any website
|
||||
- **Bug 9** (Low): Stale-task mtime proxy is unreliable after transitions
|
||||
- **Bug 10** (Low): Artifact inference maps TEST_PLAN.md to `implement` instead of `test_design`
|
||||
|
||||
## Bug Finder vs Adversarial Bug Finder Comparison
|
||||
- **Found by both**: None (reports are complementary by design)
|
||||
- **Found only by Bug Finder**: Bugs 1–7 (enforcement logic, shell scripts, model lookup, guard registration)
|
||||
- **Found only by Adversarial Bug Finder**: Bugs 8–10 (security, session tracking, phase inference edge case)
|
||||
- **Contradictions**: None. The two reports are consistent and non-overlapping.
|
||||
|
||||
## Tasks for Review / Tie-Breaks
|
||||
None.
|
||||
- None. No contradictions between Bug Finder and Adversarial Bug Finder. All 10 bugs are independently verified with reproduction steps.
|
||||
|
||||
## Remaining Issues
|
||||
None.
|
||||
- Bugs 1, 4, 5, 7 (High severity) should be fixed before relying on the framework for production enforcement. Bug 5 in particular means the primary enforcement layer (harness pre-edit guard) is likely not registered for most users.
|
||||
- Bug 6 means the dashboard display may not match the actual enforced state — users could make decisions based on stale dashboard info.
|
||||
- Bug 7 is a security issue: the `--can-edit` scope check can be bypassed via sibling directory names.
|
||||
|
||||
## Score
|
||||
|
||||
| Bug | Severity | Score | Status |
|
||||
|-----|----------|-------|--------|
|
||||
| 1 | Critical | +10 | **FIXED** — State determination reordered |
|
||||
| 2 | Medium | +5 | **FIXED** — Duplicate paragraph removed |
|
||||
| 3 | High | +5 | **FIXED** — Empty IMPLEMENTATION.md creation removed |
|
||||
| 4 | High | +5 | **FIXED** — VERDICT.md check added |
|
||||
| 5 | High | +5 | **FIXED** — Orphaned sub-tasks check added |
|
||||
| 6 | Medium | +5 | **FIXED** — Wave enforcement added |
|
||||
| 7 | Medium | +5 | **FIXED** — Sub-task prioritization added |
|
||||
| 8 | High | +5 | **FIXED** — Artifact validation added |
|
||||
| 9 | Low | +1 | **FIXED** — Units clarified in VRAM_CONFIG.md |
|
||||
| 10 | Medium | +5 | **FIXED** — Sub-task scope added to PARENT_SPEC.md |
|
||||
| 11 | High | +5 | **FIXED** — Sub-task fix task creation added |
|
||||
| 12 | Medium | +5 | **FIXED** — Script existence check added |
|
||||
| 13 | Medium | +5 | **FIXED** — Script failure handling added |
|
||||
| 14 | Medium | +5 | **FIXED** — Sub-task verdict aggregation added |
|
||||
| 15 | Low | +1 | **FIXED** — Sub-task tie-break task creation added |
|
||||
| 16 | Medium | +5 | **FIXED** — Empty artifact checks added |
|
||||
| 17 | Medium | +5 | **FIXED** — Sub-task wave prioritization added |
|
||||
| 18 | Low | +1 | **FIXED** — Auto-Execution Rules consolidated |
|
||||
| Adv1 | Critical | +10 | **FIXED** — Auto-execution loop timeout/iteration limit added |
|
||||
| Adv2 | High | +5 | **FIXED** — Duplicate sub-task prevention added |
|
||||
| Adv3 | High | +5 | **FIXED** — VRAM detection caching added |
|
||||
| Adv4 | High | +5 | **FIXED** — Orphaned sub-tasks check added |
|
||||
| Adv5 | High | +5 | **FIXED** — Sub-task parallel execution added |
|
||||
| Adv6 | Medium | +5 | **FIXED** — Overlapping conditions note added |
|
||||
| Adv7 | Medium | +5 | **FIXED** — Circular reference prevention added |
|
||||
| Adv8 | Medium | +5 | **FIXED** — Memory exhaustion prevention added |
|
||||
| Adv9 | High | +5 | **FIXED** — Graceful sub-task failure handling added |
|
||||
| Adv10 | Medium | +5 | **FIXED** — DECOMPOSITION.md update handling added |
|
||||
| Adv11 | Medium | +5 | **FIXED** — Phase timeout added |
|
||||
| Adv12 | Medium | +5 | **FIXED** — Sub-task VRAM limit added to VRAM_CONFIG.md |
|
||||
| **Bug Finder Total** | | **78** | |
|
||||
| **Adversarial Bug Finder Total** | | **65** | |
|
||||
| **Combined Score** | | **143** | |
|
||||
+5 (NEEDS_REVIEW)
|
||||
|
||||
## Reviewer Comments
|
||||
|
||||
(Leave blank for the human reviewer to provide feedback)
|
||||
|
||||
Reference in New Issue
Block a user