hermes/automaton
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Test Design phase: test_design.md prompt, workflow state machine, orchestrator updates, implement/referee reference TEST_PLAN.md

Browse Source
This commit is contained in:
gitea
2026-06-11 09:25:45 -04:00
parent fec11d29dc
commit f4587886b9
10 changed files with 193 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BUG_REPORT.md
+13 -2
View File
@@ -2,7 +2,7 @@
## Summary
A comprehensive review of the agent-framework project identified **12 bugs** (after removing the visualize*.html files, which eliminated 8 visualization-related bugs). Remaining issues range from critical (Orchestrator is not a real state machine) to low (unused placeholders).
A comprehensive review of the agent-framework project identified **13 bugs** (after removing the visualize*.html files, which eliminated 8 visualization-related bugs). Remaining issues range from critical (Orchestrator is not a real state machine) to low (unused placeholders).
---
@@ -96,6 +96,16 @@ A comprehensive review of the agent-framework project identified **12 bugs** (af
---
### Bug 14: No test design phase — tests written on the fly during implementation
- **Severity**: Medium
- **Location**: `prompts/implement.md`, `prompts/referee.md`, `prompts/workflow.md`
- **Description**: Tests are written during the implementation phase as part of the TDD loop. There is no formal Test Design phase before implementation starts. The spec says "acceptance criteria are testable" but doesn't require test cases to be defined upfront. This means:
- The implementer writes tests on the fly, so there's no test specification to verify against later
- The referee checks "Are edge cases covered?" and "Do all tests pass?" but doesn't have a test spec to compare against
- There's no way for the user to sign off on test coverage before implementation starts
- **Reproduction**: Start "Implement add user auth". The implementer writes tests during the TDD loop. No one has reviewed or approved the test coverage before code was written.
- **Suggested Fix**: **RESOLVED** — Added a Test Design phase (optional) between Design and Implement. Produces TEST_PLAN.md — an explicit test specification that the implementer follows. The referee checks that all test cases from the TEST_PLAN.md are implemented.
### Bug 11: Orchestrator contradiction — Autopilot mode task creation from verdicts
- **Severity**: Medium
- **Location**: `prompts/orchestrate.md`, "Tasks from bug verdicts" section
@@ -129,4 +139,5 @@ A comprehensive review of the agent-framework project identified **12 bugs** (af
| 11 | Medium | +5 | **RESOLVED** — Split Orchestrator verdict task creation into Manual/Autopilot modes |
| 12 | Medium | +5 | **RESOLVED** — Added `update.sh` script, upgrade option in onboarding, update instructions in README |
| 13 | Low | +1 | **RESOLVED** — Autopilot is now Enabled by default (was Disabled) |
| **Total** | | **53** | |
| 14 | Medium | +5 | **RESOLVED** — Added Test Design phase (optional) between Design and Implement |
| **Total** | | **58** | |