Source: qa/guards-report.md
# Mechanical guards for CASE files — validation report (Fable2, 2026-11-10)
Trigger: QA1's audit of Flash1 (`neov2/qa/flash1-QA1.md`): 10 of 14 sampled files needed a fix although every quote was
verbatim. Guards: `neov2/tools/guards.py`, run by `casedsl.py check` (warnings) and `casedsl.py check --strict` (errors).
All deterministic: regular expressions, offsets, the IR's citation spans, the route taxonomy. No model.
**Caveat first.** The guards were written and tuned with QA1's 14 files and the 165 older files in view, so the rates below
are in-sample. The 103 other Flash files were not used for tuning and have no gold labels; their flags are listed, not scored.
"False positive" on the older files is my own reading of each flagged quote, not an independent review.
## The guards
| code | flags | waivable |
|---|---|---|
| `G1-syllabus` | a quote, or a declared opinion span, lies before the opinion of the Court begins (syllabus, "Held:" headnote, counsel list) | no |
| `G1-no-opinion` | strict only: the file declares no `opinion` span, so quote-in-opinion cannot be checked | no |
| `G2-recital` | a `fact` quote reads as procedure ("X moved for summary judgment", "The district court granted", "We review … de novo") or as doctrine ("Section 1 … provides", "To establish …, a plaintiff must", "This case presents") | yes |
| `G2-history-held` | a `held` quote recounts history ("the jury returned a verdict", "The district court dismissed") | yes |
| `G3-cited-case` | a `fact` quote describes another case: "In X v. Y, the Court …", "But, the Court held …", or a full case citation inside the span with no party of this case named | yes |
| `G4-route-fit` | no term of the route's name, definition or include rule occurs anywhere in the opinion | yes |
| `G4-pattern-copied` | a pattern of 3+ content words is identical to another case file's pattern for the same element, and at least one of its words occurs nowhere in this opinion | yes |
| `G5-no-fact` | an applied block held satisfied `[pleaded|triable|proved]` quotes no `fact` | yes |
Waiver: `? guard: <code> <reason>` inside the block. It needs a reason, it is printed as `waived guard[...]` on every
check, and `vtool.py guards` counts waivers per code so a reviewer can sample them first.
### Where the opinion starts (G1)
I looked at how the corpus marks it rather than assuming. Sampling every one of the 47 reporters in
`docroot/ir/caselaw` (553 records): the IR text of 46 starts at the opinion — head matter was dropped upstream; five
New York / Florida reporters open with the literal header `OPINION OF THE COURT` at offset 0. Only CourtListener-sourced
Supreme Court records (`/cl/<id>`, `/us/6xx/<page>`) carry a syllabus. On all 226 such records:
| | records |
|---|---:|
| no syllabus (orders, short per curiams) | 83 |
| syllabus present | 143 |
| — opinion start found ("Justice X delivered the opinion of the Court", "Per Curiam.") | 143 |
| — not found | 0 |
If the start is ever not found (an IR cut off inside the syllabus), the guard falls back to the end of the syllabus
line-up ("X, J., delivered the opinion of the Court, in which …") and says the offset is approximate; with neither it
raises a file-level flag asking for a hand-declared span.
Two traps the detector handles: in preliminary prints the running header "Opinion of the Court" appears *inside* the
syllabus pages, so it is not used as a marker; and the page-proof watermark is interleaved with the text
("Page Justice Proof Pending Barrett delivered Publication the opinion of the Court"). The older `/us/NNN/PPPP-01`
records start at "Justice X delivered the opinion of the Court" (offset ≈ 15) and have nothing to flag.
A whole-text span (`opinion op : majority @0..N`, what the mechanical v1→v1.1 upgrade writes) does not defeat the
guard: a span that begins inside the syllabus is itself flagged.
## Result on QA1's 14 audited files
| QA1 finding a guard could see | files | flagged |
|---|---:|---:|
| all quotes from the syllabus (Olivier, Medical Marijuana, Williams) | 3 | 3 (11 of 11 quotes) |
| procedure or doctrine quoted as `fact` (Atrium, Premier, Connecticut Fine Wine) | 3 | 3 |
| procedural history quoted as `held` (Mongol Nation) | 1 | 1 |
| a precedent's facts quoted as this case's facts (Connecticut Fine Wine: *Fisher*) | 1 | 1 |
| variant forced from another case (Mongol Nation, Fleetwood) | 2 | 2 (`G4-pattern-copied`) |
| satisfied block with no fact (Fleetwood) | 1 | 1 |
All 8 FIX files with a mechanically visible defect are flagged. Within them, one of Atrium's two rule recitations is
caught as `G2-recital` and the other only through `G3-cited-case` (it cites a case); one extra `G2-recital` in Premier
("Premier contends … that the Defendants conspired …") is a plaintiff's contention and arguably a false alarm.
QA1 findings **no guard sees** (they need legal reading): wrong element (Chemehuevi: plaintiff standing coded as
`person`), omitted element or block (Chemehuevi, Fleetwood), inverted polarity on a defense (Kemp), basis `[law]` where
`[no-evidence]` was right and a mid-opinion standard quoted as `held` (Premier). Kemp is flagged by nothing.
QA1's 4 PASS files: 3 clean; **1 false positive** — Eveland, `G4-pattern-copied`: its pattern is the same as Kamenesh's
and QA1 judged it a correct fit. That is the guard's known weakness (below). File-level false-positive rate on PASS
files 1/4; the sample is too small to mean much.
## False-positive rate on the 165 older (Fable / Astra / Argon) files
2,182 quotes, 1,088 of them `fact`, 831 blocks. `G1` is silent (correctly: 0 syllabus quotes; the one older slip-format
case, Kousisis, quotes the opinion).
| guard | flags | files | rate | my reading of each flag |
|---|---:|---:|---|---|
| `G2-recital` | 5 | 5 | 0.46 % of fact quotes | 4 genuine (lower-court ruling, "moved for acquittal", "Petitioners appealed", a question presented, each quoted as `fact`), 1 false (a party's summary-judgment concession) |
| `G2-history-held` | 0 | 0 | 0 | — |
| `G3-cited-case` | 2 | 2 | 0.18 % of fact quotes | 1 genuine (a generalisation resting on a precedent), 1 false (a statement about state practice with a state citation) |
| `G4-route-fit` | 0 | 0 | 0 | — |
| `G4-pattern-copied` | 4 | 4 | 0.5 % of blocks | 4 false: each older file is the *original* that a later Flash file copied |
| `G5-no-fact` | 1 | 1 | 0.1 % of blocks | genuine gap (satisfied `[pleaded]`, held quote only) |
| any | 12 | 12 | 7.3 % of files | 6 of 12 flags false → 3.6 % of files carry a false flag |
Unaudited Flash files (103; not used for tuning): 46 flags in 33 files — `G5-no-fact` 21, `G4-pattern-copied` 10,
`G1-syllabus` 4 (Lindke v. Freed and O'Connor-Ratcliff v. Garnier quote the syllabus; neither was in QA1's sample),
`G2-history-held` 4, `G4-route-fit` 4 (four § 841 files claim the `scheduling` route in opinions that never mention a
schedule), `G2-recital` 2, `G3-cited-case` 1. I read the two syllabus hits against the source text and they are real; the
rest are unverified.
## What was tried and dropped
* "Include-rule keywords absent from the opinion" exactly as specified: 46 flags corpus-wide, 16 on older files, nearly
all false — include rules are examples ("cities, counties"), not vocabulary. Kept only in the weak form (no term of
name + definition + include anywhere), which fires 4 times, all on the Flash `scheduling` blocks.
* "Most pattern words absent from the opinion": 18 flags on older files, all paraphrase (annotators describe facts in
their own words). Replaced by the copied-pattern test, which is what caught both of QA1's forced variants.
* "Decided block without a fact": 43 flags on older files (rulings of law, derivative holdings). Narrowed to
*satisfied* on allegations or evidence: 1.
* Bare `Id.` / `Ibid.` as a cited-case signal: 7 false flags on older Supreme Court files where `Id.` points at the
complaint or appendix. Dropped.
## Known limits
* `G4-pattern-copied` cannot tell the original from the copy (older originals get flagged once someone copies them), and
two cases can legitimately share a fact pattern (Eveland). It is the guard most likely to need a waiver.
* G2 and G3 are opening-phrase heuristics. A recital in the middle of a quote, or a precedent described without a
citation or an "In X, the Court" opener, passes.
* Nothing here checks element choice, polarity, basis, or completeness. A file can pass `--strict` and still be wrong;
these guards remove the cheap failures so reviewers can spend their reading on the rest.
* `--strict` on today's corpus reports 356 errors, 272 of them `G1-no-opinion`: only 10 of 282 files declare opinion
spans (8 of those 10 are strict-clean). Strict is a gate for new files, not a claim about old ones.
## Reproduce
```
export NEOV2_CORPUS=/path/to/neochartdocroot
python3 neov2/qa/guards_eval.py # the tables above -> neov2/qa/guards-eval.json
python3 neov2/qa/guards_eval.py --list old --code G3 # every flag of one group / code
python3 neov2/variants/vtool.py guards [--strict] # per-code flags and waivers over all case files
python3 -m unittest neov2/tools/tests/test_guards.py
```