98 lines
4.0 KiB
Markdown
98 lines
4.0 KiB
Markdown
# Implementation: harden-parse-verdict
|
|
|
|
## SCOPE
|
|
|
|
Closed `add-loop-runner/BUG_REPORT.md` O6 (pass-string coercion bug) plus
|
|
the un-noted sibling issue (no score clamping). Pure-function change to
|
|
`parse_verdict` in `scripts/loop-runner.py`. No CLI surface change; no
|
|
schema change; no new deps (math is stdlib).
|
|
|
|
## FILES TOUCHED
|
|
|
|
- `scripts/loop-runner.py`
|
|
- Added `import math`.
|
|
- `parse_verdict(text)`: replaced `verdict["pass"] = bool(data.get("pass"))`
|
|
with explicit string-vs-bool dispatch:
|
|
- bool passed through → `bool(True)` = True; `bool(False)` = False (unchanged).
|
|
- `"true"` (any case, leading/trailing whitespace stripped) → True.
|
|
- `"false"` (any case, leading/trailing whitespace stripped) → False.
|
|
- Any other string → `bool(raw_pass.strip())` (empty → False; non-empty → True).
|
|
Preserves old `bool(...)` truthy semantics for `"yes"` / etc.
|
|
- Replaced `verdict["score"] = float(data.get("score", 0.0))` with:
|
|
- `try: score = float(data.get("score", 0.0))` /
|
|
`except (TypeError, ValueError): score = 0.5`
|
|
(TypeError for non-numeric types like None/list/dict; ValueError for
|
|
non-numeric strings like "great").
|
|
- `if not math.isfinite(score): score = 0.5` (catches NaN, Infinity,
|
|
-Infinity returned by some Hermes-style recursive decoders).
|
|
- `score = max(0.0, min(1.0, score))` (clamp to `[0, 1]`).
|
|
- Updated docstring to spell out the new contract: `pass` accepts
|
|
bool OR `"true"`/`"false"` strings (case-insensitive); `score` is
|
|
clamped to `[0, 1]` with NaN/non-finite → 0.5.
|
|
|
|
## D-ITEMS Locked
|
|
|
|
- **D-V1**: `"true"` / `"false"` strings → bool via case-insensitive
|
|
equality. Other strings defer to current `bool(...)` for backwards
|
|
compat (`pass: "yes"` stays truthy).
|
|
- **D-V2**: `score` NaN / non-finite → `0.5`.
|
|
- **D-V3**: `score` non-numeric string → `0.5`.
|
|
- **D-V4**: No opt-out flag for clamping. Strict emitters unaffected.
|
|
- **D-V5**: Tests are pure-functional; no subprocess.
|
|
|
|
## BUG FOUND AND FIXED INLINE
|
|
|
|
While running `tests/test_parse_verdict.py`, the test
|
|
`test_other_truthy_string_pass` failed on the first iteration. The
|
|
implementation had:
|
|
|
|
```python
|
|
if isinstance(raw_pass, str):
|
|
verdict_pass = raw_pass.strip().lower() == "true"
|
|
```
|
|
|
|
This treats EVERY non-`"true"` string as False — including `"yes"`,
|
|
which used to be True via `bool("yes")`. SPEC R1 explicitly says
|
|
non-`true`/`false` strings fall through to `bool(...)` for backwards
|
|
compat. Fixed to the three-way branch:
|
|
|
|
```python
|
|
if isinstance(raw_pass, str):
|
|
lower = raw_pass.strip().lower()
|
|
if lower == "true":
|
|
verdict_pass = True
|
|
elif lower == "false":
|
|
verdict_pass = False
|
|
else:
|
|
verdict_pass = bool(raw_pass.strip())
|
|
```
|
|
|
|
This is consistent with SPEC R1 wording. All 22 tests pass after the fix.
|
|
|
|
## TESTS
|
|
|
|
New file `tests/test_parse_verdict.py` — 22 tests across 5 classes:
|
|
|
|
- `TestStrictBaseline` (2): bool `pass` true/false; preserves existing semantics.
|
|
- `TestPassStringCoercion` (6): `"true"`/`"false"` strings, case-insensitive,
|
|
surrounding whitespace, empty string, other truthy string.
|
|
- `TestScoreClamping` (9): clamped high (1.5→1.0), clamped low (-0.3→0.0),
|
|
edges (0.0, 1.0), NaN → 0.5, Infinity → 0.5, non-numeric string "great"
|
|
→ 0.5, numeric string "0.75" → 0.75, missing score → 0.0, None score →
|
|
0.5.
|
|
- `TestFenceBlockStillWorks` (2): existing fence-block path with bool pass;
|
|
fence path with string-pass + clamped score.
|
|
- `TestOptionalKeysPreserved` (3): reasons+next_hint combination, missing
|
|
reasons → [], non-list reasons coerced to [].
|
|
|
|
## TEST COUNT
|
|
|
|
- Baseline: 447 passed (post-`add-state-loop-lock`).
|
|
- New: +22 in `tests/test_parse_verdict.py`.
|
|
- Final: **469 passed**, 0 regressions.
|
|
|
|
## PIPELINE TO COMPLETION
|
|
|
|
research → research:awaiting_approval → research:approved → implement.
|
|
Next: → code_review → code_review:awaiting_approval → code_review:approved
|
|
→ bug_find → adversarial_bug_find → doc_review → referee → complete. |