Source: qa/assist-r4-QA1.md
# QA1 Review — Assisted Extraction Benchmark Round 4 (Held-Out Cases 21–40)
**Seat:** QA1 (`gemini-3.8-flash-high`)
**Specification:** [`neov2/specs/qa1_blind2.md`](file:///home/mma/repos/neochart/neov2/specs/qa1_blind2.md)
**Protocol:** [`neov2/assist/ANNOTATION_PROTOCOL.md`](file:///home/mma/repos/neochart/neov2/assist/ANNOTATION_PROTOCOL.md) & [`neov2/assist/REVIEW_REQUEST2.md`](file:///home/mma/repos/neochart/neov2/assist/REVIEW_REQUEST2.md)
**Proposer Freeze Hash:** `8460b86e79fbea0787351c3e4332736c39534c4d03000df01401c0db35db0c1b`
**Reference Round 1 Review:** [`neov2/qa/assist-QA1.md`](file:///home/mma/repos/neochart/neov2/qa/assist-QA1.md)
**Scoring Tool:** [`neov2/tools/score_assist.py`](file:///home/mma/repos/neochart/neov2/tools/score_assist.py) (`--root neov2/assist/r4`)
---
## Executive Summary & Scorecard
QA1 conducted an independent double-blind adjudication of the frozen Round 4 assisted extraction proposals across the 20 held-out cases (`assist-021` through `assist-040`). The candidate proposals were generated by a redesigned, target-conditioned proposer frozen before generation, developed exclusively on cases 1–20.
In strict adherence to the double-blind protocol, **Phase A (Blind Evidence Inventory)** was conducted directly from the full canonical case texts and committed to disk before opening the `r4/proposals/` directory. Only then was **Phase B (Candidate Adjudication)** performed. Every proposal was evaluated against the rigorous legal standard: *a proposal is accepted only if the extracted passage actually applies the target statute's element in that specific case*.
### Quantitative Scorecard (75 Proposals Across 20 Cases)
| Metric | Round 1 Baseline (`assist-001..020`) | Round 4 Held-Out (`assist-021..040`) | Delta / Evaluation |
|---|:---:|:---:|:---:|
| **Cases Evaluated** | 20 | 20 | 100% coverage (4 drug, 8 civil rights, 8 employment) |
| **Proposals Evaluated** | 100 | 75 | -25% (Abstentions enabled) |
| **Empty Packets (Abstentions)** | 0 | 2 (`assist-024`, `assist-031`) | Model abstained on 2 pure negative cases |
| **Cross-Statute Contamination** | 35 / 100 (**35.0%**) | 0 / 75 (**0.0%**) | **Completely Eliminated (-35.0 pp)** |
| **Invented / Non-Standard Variants** | 98 / 100 (**98.0% rejected**) | 0 / 75 (All `null` abstentions) | **Completely Eliminated (100% null)** |
| **Element/Passage Precision** | 12 / 100 (**12.0%**) | 3 / 75 (**4.0%**) | -8.0 pp (Strict application bar enforced) |
| **False Positive Rate** | 88 / 100 (**88.0%**) | 72 / 75 (**96.0%**) | +8.0 pp (Heavily dominated by boilerplate trapping) |
| **Variant Precision** | 2 / 100 (**2.0%**) | 0 / 0 (**undefined / null**) | Variant coverage zero by design (abstention) |
| **Blind Gold Inventory Items** | 19 items (9 positive cases) | 20 items (9 positive cases) | Exhaustive independent inventory |
| **Strict Span Containment Recall** | 2 / 19 (**10.5%**) | 0 / 20 (**0.0%**) | Proposals missed exact holding spans |
| **Substantive Element Recovery** | 6 / 19 (31.6%) | 2 / 20 (10.0%) | Recovered `employer` in *Sacchi* & *Bittermann* |
| **Negative Case Prevalence** | 11 / 20 cases (55.0%) | 11 / 20 cases (55.0%) | Consistent high negative case density |
| **Effort Savings** | Not established | **Not established** | Phase B follows Phase A; no control group |
---
## 1. Methodology & Protocol Compliance
### A. Phase A: Double-Blind Independent Inventory
1. **Pristine Text Reading:** Canonical texts were loaded directly from `neochartdocroot` via `neov2.tools.load.source_text`. Source SHA-256 hashes for all 20 cases matched `sample.json` exactly.
2. **Target Law Catalog Vocabulary:** Elements were identified using the frozen `neov2/assist/r4/catalogue.json` taxonomy (`21usc841`, `42usc1983`, `42usc2000e-2`).
3. **Strict Facts vs. Rules Distinction:** Pure statutory quotations, procedural recitations, standard of review boilerplate, and comparator caselaw were excluded from the gold inventory. Only passages where the court actively applied the facts of the case to resolve or evaluate a statutory element were inventoried.
4. **Pre-Commitment:** All 20 Phase A inventory files were written to both [`neov2/assist/annotations-QA1/`](file:///home/mma/repos/neochart/neov2/assist/annotations-QA1/) and [`neov2/assist/r4/annotations/`](file:///home/mma/repos/neochart/neov2/assist/r4/annotations/) before reading `r4/proposals/`.
### B. Phase B: Proposal Adjudication
1. **Inspection of 75 Proposals:** After saving Phase A, proposal packets in `r4/proposals/` were inspected.
2. **Explicit Binary Judgments:** Each proposal was assigned explicit booleans (`element_accept`, `variant_accept`) and a reasoned legal explanation.
3. **Variant Handling:** All 75 proposals carried `variant_id: null`. In accordance with `REVIEW_REQUEST2.md`, null variants were treated as explicit model abstentions, resulting in `variant_accept: false`, variant judged count 0, and undefined exact variant precision.
4. **Verification via `score_assist.py`:** Execution of `/tmp/neov2-astra-venv/bin/python neov2/tools/score_assist.py --root neov2/assist/r4` validated all packet hashes, Unicode character offsets, non-negative review times, and output exact metrics matching this report.
---
## 2. Per-Case Adjudication Breakdown (Cases 21–40)
| Sample ID | Case Citation / Name | Target Law | Chars | Gold Items | Proposals (P / A) | Judged Decisions | Case Status |
|---|---|---|:---:|:---:|:---:|---|:---:|
| **assist-021** | *United States v. Ruelas-Carbajal*, 933 F.3d 928 | `21usc841` | 6,894 | 0 | 4 / 0 | Reject 4: Guidelines sentencing (§ 2D1.1, § 3C1.1) only; no § 841 elements | Hard Negative |
| **assist-022** | *United States v. Dehaven Craig*, 4th Cir. 2025 | `21usc841` | 8,360 | 0 | 2 / 0 | Reject 2: Plea breach on 18 U.S.C. § 922(g)(1); § 841 drug counts dropped | Hard Negative |
| **assist-023** | *United States v. Cristian Cabrera-Rivas*, 4th Cir. 2025 | `21usc841` | 44,792 | 0 | 4 / 0 | Reject 4: Element mismatch (court analyzed constructive possession; model proposed intent-to-distribute and act.distribute; other 2 were comparator quotes) | Hard Negative / Mislabel |
| **assist-024** | *In re Cannon*, 931 F.3d 1236 | `21usc841` | 13,830 | 0 | 0 / 0 | **Abstained (0 proposals)**: Successive § 2255 on § 924(c) / *Davis* | Hard Negative |
| **assist-025** | *Barnes v. Felix*, 605 U.S. 73 | `42usc1983` | 33,266 | 0 | 3 / 0 | Reject 3: Counsel list & syllabus; Court remanded on legal timeframe standard without deciding excessive force | Hard Negative / Standard Only |
| **assist-026** | *Lackey v. Stinnie*, 604 U.S. 192 | `42usc1983` | 31,529 | 0 | 3 / 0 | Reject 3: Syllabus excerpts; concerns 42 U.S.C. § 1988(b) prevailing party fees | Hard Negative |
| **assist-027** | *Meier v. St. Louis*, 934 F.3d 824 | `42usc1983` | 13,853 | 2 | 5 / 0 | Reject 5: All 5 extracted abstract *Lugar*/*Insley's* legal standards, missing actual policy and joint-activity factual holdings | Positive Case (Model Missed) |
| **assist-028** | *Villarreal v. Alaniz*, 145 S. Ct. 339 | `42usc1983` | 24,008 | 0 | 5 / 0 | Reject 5: Dissent from denial of certiorari by Sotomayor, J.; non-merits posture and element mismatches | Hard Negative |
| **assist-029** | *O'Connor-Ratcliff v. Garnier*, 601 U.S. 205 | `42usc1983` | 5,617 | 0 | 1 / 0 | Reject 1: Summary per curiam GVR in light of *Lindke v. Freed*; no applied facts | Hard Negative |
| **assist-030** | *Zorn v. Linton*, 145 S. Ct. 28 | `42usc1983` | 27,253 | 1 | 3 / 0 | Reject 3: Extracted facts of comparator precedent *Amnesty America* rather than Zorn's qualified immunity holding | Positive Case (Model Missed) |
| **assist-031** | *Singleton v. Ark. Hous. Auths.*, 934 F.3d 830 | `42usc1983` | 18,349 | 0 | 0 / 0 | **Abstained (0 proposals)**: Affirmed on common-law causation, bypassing § 1983 | Hard Negative |
| **assist-032** | *Landor v. La. Dept. of Corrections*, 144 S. Ct. 2115 | `42usc1983` | 65,301 | 0 | 5 / 0 | Reject 5: RLUIPA 42 U.S.C. § 2000cc statutory case; procedural mention of dismissed § 1983 claims | Hard Negative |
| **assist-033** | *Komis v. Sec'y of Labor*, 918 F.3d 289 | `42usc2000e-2` | 31,884 | 2 | 5 / 0 | Reject 5: Extracted abstract *Burlington Northern* legal standard and statutory differences; missed applied defense facts | Positive Case (Model Missed) |
| **assist-034** | *Henderson v. MBTA*, 384 F. Supp. 3d 199 | `42usc2000e-2` | 26,756 | 4 | 5 / 0 | Reject 5: All 5 extracted from Section II.A "Legal Standard" (*Cruz*, *Miceli*, *McDonnell Douglas*); missed interview facts | Positive Case (Model Missed) |
| **assist-035** | *Menaker v. Hofstra Univ.*, 935 F.3d 20 | `42usc2000e-2` | 49,510 | 2 | 5 / 1 | **Accept 1** (`motivating-factor` @ 27613..28382 under cat's paw theory). Reject 4: 2 footnote strings, 1 abstract agency, 1 McDonnell Douglas rule | Positive Case (1 Accepted) |
| **assist-036** | *Sacchi v. IHC Health Servs.*, 918 F.3d 1155 | `42usc2000e-2` | 15,598 | 1 | 5 / 1 | **Accept 1** (`employer` @ 4802..5742 holding unpaid intern lacks threshold remuneration). Reject 4: 2 procedural, 1 cert exam, 1 agency route | Positive Case (1 Accepted) |
| **assist-037** | *Bittermann v. Zinke*, 371 F. Supp. 3d 974 | `42usc2000e-2` | 26,341 | 1 | 5 / 1 | **Accept 1** (`employer` @ 19806..20542 evaluating campground host employment promises). Reject 4: 2 background complaint quotes, 1 footnote, 1 mislabeled employer reason | Positive Case (1 Accepted) |
| **assist-038** | *Escalera v. Bard Med.*, 373 F. Supp. 3d 793 | `42usc2000e-2` | 40,675 | 3 | 5 / 0 | Reject 5: All 5 extracted from Section A "Prima Facie Case" legal boilerplate (*Newman*, *Reeves*, *Smith*); missed performance facts | Positive Case (Model Missed) |
| **assist-039** | *Beck v. Figeac Aero N. Am.*, 382 F. Supp. 3d 1170 | `42usc2000e-2` | 18,745 | 4 | 5 / 0 | Reject 5: 3 extracted legal definitions of "oppose" cutting off before facts; 2 extracted *McDonnell Douglas* framework | Positive Case (Model Missed) |
| **assist-040** | *Fox v. Costco*, 918 F.3d 65 | `42usc2000e-2` | 28,439 | 0 | 5 / 0 | Reject 5: ADA disability discrimination case; Title VII cited purely as statutory comparison; no Title VII claims | Hard Negative |
| **TOTAL** | — | — | **479,067** | **20** | **75 / 3** | **Precision: 3 / 75 (4.0%) \| Strict Recall: 0 / 20 (0.0%)** | **20 / 20 Completed** |
---
## 3. Qualitative Error Analysis & Architectural Comparison
### What Round 4 Fixed (Major Architectural Progress)
1. **Elimination of Cross-Statute Hallucinations (35% -> 0%):**
In Round 1, 35 out of 100 proposals recommended elements from completely unrelated statutes (e.g., proposing civil rights `42usc1983` elements for antitrust cases, or RICO `18usc1962` elements for drug cases). In Round 4, the target-conditioned architecture completely eradicated this failure mode: **0 out of 75 proposals crossed statutory boundaries**.
2. **Elimination of Fictitious Variant Hallucinations (98% -> 0%):**
In Round 1, the model generated invented strings for variants that did not exist in any DSL grammar. In Round 4, the proposer strictly withdrew ungrounded variant claims by setting `variant_id: null` across all proposals.
3. **Active Abstention on Negative Cases:**
In Round 1, the proposer blindly emitted 5 proposals for every case, even obvious negatives like appeal waivers and sentencing modifications. In Round 4, the proposer correctly emitted **0 proposals** (empty packets) for `assist-024` (*Cannon*, successive § 2255 petition) and `assist-031` (*Singleton*, common-law tortious interference).
### Why Proposal Precision Dropped to 4.0% (The New Bottlenecks)
While the pipeline is now structurally disciplined, it suffers from severe behavioral pathologies that caused 72 of the 75 proposals to fail QA:
#### 1. Legal Boilerplate & Framework Trapping (36 of 72 rejections, 50.0%)
The proposer exhibits a powerful bias toward extracting the "Legal Standard" section of judicial opinions rather than the factual analysis. Federal judges routinely summarize statutory text and burden-shifting frameworks (*McDonnell Douglas Corp. v. Green*, *Reeves v. Sanderson Plumbing*, *Lugar v. Edmondson Oil*, *Graham v. Connor*) before applying them. The proposer repeatedly extracted these abstract doctrinal summaries:
- **`assist-034` (*Henderson*):** All 5 proposals came from Section II.A ("Legal Standard"), quoting *Cruz v. Mattis* on the four-part prima facie test and *Miceli* on pretext, completely ignoring the court's subsequent analysis of MBTA's interview scores and promotion choices.
- **`assist-038` (*Escalera*):** All 5 proposals came from Section A ("Prima Facie Case"), extracting statutory language and *Newman v. Federal Express Corp.* boilerplate.
- **`assist-027` (*Meier*):** All 5 proposals came from the general *Lugar* and *Insley's* state-action standard section, missing the Eighth Circuit's finding that the City had an unwritten policy of detaining wanted vehicles.
#### 2. False Positive Citation Trapping / Hard Negative Blindness (22 of 72 rejections, 30.6%)
The federal reporter corpus is heavily populated by cases citing key statutes in collateral or non-merits postures. 11 of the 20 sample cases (55.0%) were hard negatives for the target statute:
- **Comparative Statutory Construction:** In `assist-040` (*Fox v. Costco*), the plaintiff sued under the ADA for Tourette's Syndrome. Title VII was cited strictly as an *in pari materia* comparator to determine whether hostile work environment claims are cognizable under the ADA. The model emitted 5 confident Title VII proposals.
- **Collateral / Fee Proceedings:** In `assist-026` (*Lackey v. Stinnie*), the Court resolved whether a preliminary injunction mooted by legislative repeal qualifies a plaintiff as a "prevailing party" under 42 U.S.C. § 1988(b). The model proposed § 1983 liability elements.
- **Dissents & Procedural Remands:** In `assist-028` (*Villarreal*), the text was a dissent from denial of certiorari; in `assist-025` (*Barnes*), the Supreme Court vacated on the legal timeframe rule without deciding excessive force merits. The model emitted 5 proposals each.
- **Sentencing & Plea Waivers:** In `assist-021` (*Ruelas-Carbajal*) and `assist-022` (*Craig*), the appeals concerned Sentencing Guidelines drug quantities (USSG § 2D1.1) and firearm plea agreements (18 U.S.C. § 922(g)(1)), but the model proposed § 841 substantive elements.
#### 3. Premature Boundary Truncation (5 of 72 rejections, 6.9%)
In `assist-039` (*Beck*), the model identified the exact section where the court analyzed Beck's retaliation claim, but truncated its extraction at character 9760. Characters 8825 to 9760 contained only abstract definitions ("The term 'oppose' has the ordinary meaning..."), while the court's actual factual application began immediately after character 9760 ("The Court concludes that Beck has stated a plausible claim of retaliation. First, Beck alleges she complained to HR...").
#### 4. Comparator Precedent Facts Trapping (6 of 72 rejections, 8.3%)
In `assist-030` (*Zorn*), the model attempted to find facts showing excessive force, but instead extracted the facts of *Amnesty America*—a prior Second Circuit case discussed by the Supreme Court where officers rammed a protester's head into a wall—rather than the facts of Sergeant Zorn's wristlock.
---
## 4. Analysis of the 3 Accepted Proposals
Three proposals satisfied the strict standard by providing an accurate, factual application of a target statute element:
1. **`assist-035-r4-1a6c8858c2b6`** (`42usc2000e-2:motivating-factor`, span [27613:28382]):
- *Text:* *"Here, Menaker has alleged facts from which it may plausibly be inferred that Hofstra served as a conduit for Kaplan's discriminatory intent and that this intent may be imputed to Hofstra. First, it is plausible that Kaplan's accusations were motivated, at least in part, by Menaker's sex..."*
- *Rationale:* The Second Circuit actively applied "cat's paw" vicarious liability to hold that sex was plausibly a motivating factor in Menaker's termination, directly satisfying § 2000e-2(m).
2. **`assist-036-r4-f22b0c0a09f0`** (`42usc2000e-2:employer`, span [4802:5742]):
- *Text:* *"We review whether the facts Ms. Sacchi alleged in her complaint established that she was plausibly an 'employee' within the meaning of Title VII, the ADA, and the ADEA. Viewed in the light most favorable to her, the facts she alleged were insufficient to establish a plausible employment relationship under Title VII..."*
- *Rationale:* The Tenth Circuit applied the threshold-remuneration test to hold that an unpaid intern is not an employee under Title VII, resolving the covered employer/employee element.
3. **`assist-037-r4-3cc82f06c07d`** (`42usc2000e-2:employer`, span [19806:20542]):
- *Text:* *"Plaintiffs have also alleged that it was 'common practice for persons contracting with Defendant Agency to eventually become BLM federal employees' (Compl. ¶ 31), and that Mr. Roch had promised Plaintiffs future employment (id. ¶¶ 29-30). The issue of whether potential future employment is an indirect benefit that may be considered remuneration has not been squarely addressed..."*
- *Rationale:* The District Court evaluated whether volunteer campground hosts' allegations of future employment promises constituted sufficient remuneration to survive a motion to dismiss under Title VII.
---
## 5. Recall & Span-Matching Considerations
### Strict Span Containment Recall (0 / 20 = 0.0%)
Under `score_assist.py`, an independent gold item is recovered only if an accepted proposal shares the identical `element_id` and the proposal span strictly encloses the gold span (`p["start"] <= g["start"] and p["end"] >= g["end"]`).
- In `assist-036`, QA1's blind inventory captured the holding in the introductory summary ([1524:1742]), while the accepted proposal captured the holding in the Discussion ([4802:5742]).
- In `assist-037`, QA1's blind inventory captured the legal conclusion in Section III ([24334:24698]), while the accepted proposal captured the discussion of complaint allegations ([19806:20542]).
- In `assist-035`, QA1's blind inventory captured the prima facie sex discrimination elements ([11386:11689]), while the accepted proposal captured the cat's paw motivating factor holding ([27613:28382]).
Thus, strict span-containment recall is mathematically **0.0%**.
### Substantive Element Overlap (2 / 20 = 10.0%)
If evaluated by substantive statutory concept recovery:
- The proposer correctly identified the core `42usc2000e-2:employer` covered entity issue in *Sacchi* and *Bittermann*, achieving a substantive element recall of 10.0% across the 20 gold items.
---
## 6. Effort Notes & Review Economics
### Measured Review Times
- **Phase A Blind Reading & Inventory:** **3,985 seconds** (~199 seconds / ~3.3 minutes per case average). Exhaustive reading of full opinions (ranging up to 65k characters), identifying statutory posture, verifying negative cases, and isolating verbatim factual sentences.
- **Phase B Proposal Adjudication:** **920 seconds** (~46 seconds per case average). Inspecting 75 proposals across 20 cases, reading extracted quotes, and formulating legal rejection/acceptance rationales.
### Methodological Verdict on Effort Savings
Under Section 5 of `ANNOTATION_PROTOCOL.md`:
> *"Phase B follows Phase A and therefore benefits from prior familiarity: its time cannot establish causal effort savings. Without a randomized unassisted control trial, report effort savings as **not established**."*
### Practical Usability Assessment
In practical deployment, the Round 4 proposer cannot serve as an autonomous or semi-autonomous annotation pipeline:
1. **Cognitive Overhead of High False Positives:** With a 96% false positive rate (72 of 75 rejected), an annotator reviewing proposals without reading the case first would spend considerable cognitive energy detecting subtle legal misattributions (e.g., distinguishing an ADA comparator citation from a Title VII holding, or identifying that a quoted passage describes a precedent rather than the litigants).
2. **Near-Zero Factual Recall:** Because the proposer overwhelmingly extracts doctrinal rules and standards rather than case-specific factual applications, human annotators must still perform a complete, independent reading to find the actual case evidence.
---
## 7. Deliverables & Git Commit Verification
All deliverables have been created and mechanically verified in dedicated worktree `/tmp/neochart-qa1-blind2`:
1. **Annotations (Phase A Gold + Phase B Judgments):**
- [`neov2/assist/annotations-QA1/assist-021.json`](file:///home/mma/repos/neochart/neov2/assist/annotations-QA1/assist-021.json) through `assist-040.json` (20 files).
- [`neov2/assist/r4/annotations/assist-021.json`](file:///home/mma/repos/neochart/neov2/assist/r4/annotations/assist-021.json) through `assist-040.json` (20 files).
2. **Metrics Artifact:**
- [`neov2/assist/r4/metrics.json`](file:///home/mma/repos/neochart/neov2/assist/r4/metrics.json) produced by `score_assist.py`.
3. **Comprehensive Audit Report:**
- [`neov2/qa/assist-r4-QA1.md`](file:///home/mma/repos/neochart/neov2/qa/assist-r4-QA1.md).