6.2 KiB
Adversarial Bug Report: harden-parse-verdict
Probed parse_verdict with non-contract inputs. Each attack vector
hypothesized, tested, verdict given.
A1 — pass as Python types (None, list, dict, int) — type-confusion
Hypothesis: A verifier emitting non-string non-bool pass values
(e.g. {"pass": null}, {"pass": [false]}, {"pass": 0}) could yield
surprising verdicts.
Test: 9-row sweep via json.dumps (Python None → JSON null,
Python True/False → JSON true/false):
| Input | Output | Notes |
|---|---|---|
pass: null |
pass=False |
bool(None) = False; preserved from v1. |
pass: [] (empty list) |
pass=False |
bool([]) = False; preserved. |
pass: [false] (list with False) |
pass=True |
bool([False]) = True (non-empty list is truthy). Surprising but documented Python semantics. v1 returned same. Not a regression. |
pass: [true] |
pass=True |
Same. |
pass: {} (empty dict) |
pass=False |
bool({}) = False; preserved. |
pass: 0 (int) |
pass=False |
bool(0) = False; preserved. |
pass: 1 (int) |
pass=True |
bool(1) = True; preserved. |
pass: -1 (int) |
pass=True |
bool(-1) = True (non-zero); preserved. |
pass: 1.5 (float) |
pass=True |
bool(1.5) = True (non-zero); preserved. |
Verdict: PASS — no regression for any non-string non-bool type. RESET
behavior matches v1's bool(...) semantics. The SPEC's three-way
string/bool branch handles strings explicitly; everything else falls
through to v1's bool(...).
A2 — score as Python non-numeric types
Hypothesis: score: [], score: {}, score: [1, 2], score: "high"
should default to 0.5 per R3 (TypeError / ValueError caught).
Test:
| Input | Output |
|---|---|
score: [] |
score=0.5 (TypeError caught by float([])) |
score: {} |
score=0.5 (TypeError caught) |
score: [1, 2] |
score=0.5 (TypeError caught) |
score: "high" |
score=0.5 (ValueError caught) |
score: None |
score=0.5 (TypeError caught) |
Verdict: PASS — R3's except (TypeError, ValueError) catches all
non-numeric types; defaults to 0.5 (D-V3). Confirmed.
A3 — score as out-of-range numeric strings
Hypothesis: A verifier emitting score: "2.0" (an out-of-range
numeric STRING) bypasses the clamp because R3's except arm never fires
and R2's clamp applies after — but is the clamp correctly triggered?
Test:
| Input | Output |
|---|---|
score: "2.0" |
score=1.0 (parse to 2.0, clamp to 1.0) |
score: "-0.5" |
score=0.0 (parse to -0.5, clamp to 0.0) |
score: "0.75" |
score=0.75 (parse to 0.75, no clamping) |
Verdict: PASS — clamping applies to all numeric inputs regardless of
whether they came in as JSON numbers or numeric strings. Confirmed in
the SPEC test plan (test_score_numeric_string_ok and the inline fix).
A4 — score as JSON literal NaN / Infinity / -Infinity
Hypothesis: Some Hermes-style recursive decoders emit the bare
tokens NaN / Infinity / -Infinity (rejected by strict JSON but
accepted by Python's json.loads with the default parse_constant).
float(NaN) succeeds (returns math.nan). The math.isfinite check
catches it.
Test:
| Input | Output |
|---|---|
score: NaN (bare token) |
score=0.5 (isfinite catches; D-V2 default) |
score: Infinity (bare token) |
score=0.5 |
score: -Infinity (bare token) |
score=0.5 |
Verdict: PASS — D-V2 documented neutral default. The math.isfinite
guard fires before the clamp so the NaN doesn't propagate through max /
min.
A5 — score as JSON booleans (true / false)
Hypothesis: A verifier erroneously using "score": true instead of
"score": 0.8 would yield float(True) = 1.0 in Python (no exception),
then clamp to 1.0 (no change). The result is "the verifier said pass
with a perfect score" — incorrect but not a crash. Is this OK?
Test:
| Input | Output |
|---|---|
score: true |
score=1.0 (float(True) → max(0, min(1, 1.0)) → 1.0) |
score: false |
score=0.0 (float(False) → 0.0) |
Verdict: PASS — float(True) is well-defined in Python. A verifier
mis-typing score: true produces a deterministic 1.0 (not a crash; not
NaN). Score-plateau gate will see consistent 1.0 across ticks → halt as
score_plateau. Reasonable downstream behavior; documented quirk.
A6 — pass short strings ("t", "T", "f")
Hypothesis: A verifier abbreviating pass: "t" or pass: "T" might
be misread as True (since SPEC only says "true"/"false" exact match
maps to True/False). Per SPEC R1, other strings fall through to
bool(...), which is truthy for non-empty.
Test:
| Input | Output |
|---|---|
pass: "t" |
pass=True (abstract: bool("t") = True; not "true") |
pass: "T" |
pass=True |
pass: "f" |
pass=True (truthy; surprising!) |
Verdict: PASS — documented behavior. Risk: a verifier emitting
pass: "f" intending "false" gets True. Same as v1. The SPEC's
contract is to use full true/false strings or JSON booleans. This
abbreviated-string case is undocumented but not a regression; future
prompt work (out of scope for this task) should discourage abbreviations.
A7 — Combined: pass: "false" string with score: NaN literal — full
harsh-path coverage
Hypothesis: A both-broken verdict still yields a parseable dict with coerced defaults rather than None.
Test: {"pass": "false", "score": NaN} literal — parse_verdict
returns {"pass": False, "score": 0.5, "reasons": [], "next_hint": ""}.
Verdict: PASS — both coercion paths fire; documented defaults applied.
A8 — Whitespace-only strips: newline + tab in pass value
Hypothesis: A verifier emitting pass: "\n true " (whitespace-wrapped)
should yield True after .strip().
Confirmed via test test_pass_with_surrounding_whitespace — " true "
strips cleanly. Newline/tab characters not explicitly tested but
str.strip() defaults to all whitespace; newlines strip too.
Verdict: PASS.
No BLOCKERS
A1-A8 are all documented behaviors per SPEC R1+R2+R3 + D-V1/D-V2/D-V3.
All inputs that would have caused silent corruption (the bool("false")=True
bug) or crashes (TypeError from non-numeric scores) are now handled
defensively. Recommend proceeding to doc_review.