Source: qa/assist-QA1.md
# QA1 Review — Assisted Extraction Benchmark (First 20 Cases)
## Executive Summary & Scope
In accordance with [`neov2/specs/qa1_blind.md`](file:///home/mma/repos/neochart/neov2/specs/qa1_blind.md) and the independent review protocol ([`neov2/assist/ANNOTATION_PROTOCOL.md`](file:///home/mma/repos/neochart/neov2/assist/ANNOTATION_PROTOCOL.md)), QA1 conducted a double-blind adjudication of the assisted extraction benchmark.
Per the specification's explicit allowance (*"If the 40 cases are too many for one turn, do the first 20 and say so"*), this review covers the first **20 cases** (`assist-001` through `assist-020`), comprising:
- **8 cases** under 15 U.S.C. § 1 (`assist-001` – `assist-008`)
- **8 cases** under 18 U.S.C. § 1962 (`assist-009` – `assist-016`)
- **4 cases** under 21 U.S.C. § 841 (`assist-017` – `assist-020`)
### Core Quantitative Metrics (100 Proposals Adjudicated)
| Metric | Numerator / Denominator | Score |
|---|:---:|:---:|
| **Proposal Precision (Element/Passage)** | 12 / 100 | **12.0%** |
| **Proposal Precision (Variant Match)** | 2 / 100 | **2.0%** |
| **Gold Evidence Recall** | 2 / 19 | **10.5%** |
| **Negative Case Discrimination** | 0 / 55 (false positives on negatives) | **0.0% precision on negatives** |
| **Cross-Statute Contamination Rate** | 35 / 100 | **35.0%** |
---
## 1. Methodology & Blind Protocol Adherence
1. **Strict Phase A Blind Inventory:**
Before opening or listing the contents of `neov2/assist/proposals/`, QA1 loaded the full text of each case from the canonical IR corpus (`neov2.tools.corpus.load_case`). QA1 conducted an exhaustive blind review of each opinion to identify applied statutory elements, precise character offsets `[start, end)`, and verbatim factual evidence slices, saving the Phase A inventory to `neov2/assist/annotations-QA1/<sample_id>.json`.
2. **Identification of Negative Cases:**
In strict compliance with the protocol ("Distinguish facts from rules, holdings, comparator discussion and mere citation... Include negative cases or no applicable elements rather than fabricating evidence"), 11 of the 20 cases were identified as negative or background cases with no substantive merits adjudication of the target statute (e.g. `assist-002` was a Hobbs Act robbery case citing the Sherman Act only as an overt-act comparator; `assist-011` was a RICO gang appeal deciding only juror contamination; `assist-017` was an ACCA § 2255 petition; `assist-019` was an appeal waiver; `assist-020` was a supervised release termination).
3. **Phase B Candidate Adjudication:**
Only after Phase A was written were the 5 proposals per case opened from `neov2/assist/proposals/<sample_id>.json`. Each proposal was adjudicated independently for (a) element/passage support, (b) variant fit, with an explicit reason recorded.
---
## 2. Per-Case Annotation and Adjudication Summary
| Sample ID | Target Statute | Case Name / Citation | Blind Gold Items | Proposals Accepted (Elem / Var) | Status |
|---|---|---|:---:|:---:|:---:|
| **assist-001** | 15 U.S.C. § 1 | *Maui Jim, Inc. v. Smartbuy Guru Enters.* | 2 (`agreement`, `restraint.market-power`) | 2 / 1 | Completed |
| **assist-002** | 15 U.S.C. § 1 | *United States v. Jett* | 0 (Hard Negative — Hobbs Act) | 0 / 0 | Completed |
| **assist-003** | 15 U.S.C. § 1 | *Dennis v. JPMorgan Chase & Co.* | 2 (`agreement`, `restraint.per-se`) | 0 / 0 | Completed |
| **assist-004** | 15 U.S.C. § 1 | *In re NFL Sunday Ticket Antitrust Litig.* | 2 (`agreement`, `restraint.rule-of-reason`) | 2 / 1 | Completed |
| **assist-005** | 15 U.S.C. § 1 | *Tichy v. Hyatt Hotels Corp.* | 2 (`agreement`, `restraint.per-se`) | 2 / 0 | Completed |
| **assist-006** | 15 U.S.C. § 1 | *U.S. Futures Exch. v. Bd. of Trade of Chi.* | 1 (`agreement`) | 1 / 0 | Completed |
| **assist-007** | 15 U.S.C. § 1 | *In re NCAA Grant-In-Aid Cap Litig.* | 2 (`agreement`, `restraint.rule-of-reason`) | 1 / 0 | Completed |
| **assist-008** | 15 U.S.C. § 1 | *Freedom Watch, Inc. v. Google, Inc.* | 2 (`agreement`, `agreement.x-monsanto`) | 0 / 0 | Completed |
| **assist-009** | 18 U.S.C. § 1962 | *Gamboa v. Ford Motor Co.* | 2 (`enterprise`, `pattern`) | 1 / 0 | Completed |
| **assist-010** | 18 U.S.C. § 1962 | *Republic of Kaz. v. Stati* | 1 (`racketeering`) | 1 / 0 | Completed |
| **assist-011** | 18 U.S.C. § 1962 | *United States v. Smith* | 0 (Negative — Juror Fear / Mistrial) | 0 / 0 | Completed |
| **assist-012** | 18 U.S.C. § 1962 | *United States v. Steele* | 0 (Negative — § 924(c) Appeal Only) | 0 / 0 | Completed |
| **assist-013** | 18 U.S.C. § 1962 | *Martin Hilti Family Trust v. Knoedler* | 2 (`conduct`, `conspiracy`) | 1 / 0 | Completed |
| **assist-014** | 18 U.S.C. § 1962 | *United States v. Mills* | 0 (Negative — Motion in Limine) | 0 / 0 | Completed |
| **assist-015** | 18 U.S.C. § 1962 | *LLM Bar Exam, LLC v. Barbri, Inc.* | 1 (`pattern`) | 0 / 0 | Completed |
| **assist-016** | 18 U.S.C. § 1962 | *Hunt v. Zuffa, LLC* | 0 (Negative — RICO Standing § 1964(c)) | 1 / 0 | Completed |
| **assist-017** | 21 U.S.C. § 841 | *Raymond v. United States* | 0 (Hard Negative — § 2255 ACCA) | 0 / 0 | Completed |
| **assist-018** | 21 U.S.C. § 841 | *Marinelarena v. Barr* | 0 (Hard Negative — Cal. Drug Law) | 0 / 0 | Completed |
| **assist-019** | 21 U.S.C. § 841 | *United States v. Floyd* | 0 (Hard Negative — Appeal Waiver) | 0 / 0 | Completed |
| **assist-020** | 21 U.S.C. § 841 | *United States v. Damon* | 0 (Hard Negative — Supervised Release) | 0 / 0 | Completed |
| **TOTAL** | — | **20 Cases** | **19 Gold Elements** | **12 / 2 Accepted** | **100% Adjudicated** |
---
## 3. Deep Qualitative Analysis of Machine Proposals
The proposal generation pipeline exhibited systematic failure modes that explain its low precision (12.0%) and near-zero variant precision (2.0%):
### A. Severe Cross-Statute Hallucination (35% of all proposals)
In 35 out of 100 proposals, the model proposed elements and variants from completely unrelated pilot statutes:
- In `assist-001` (Sherman Act § 1 antitrust case): proposed `18usc1962:person` and `42usc1983:x-federal-action`.
- In `assist-002` (Hobbs Act robbery case): proposed `18usc1962:x-double-jeopardy`, `21usc841:act.distribute`, `18usc1962:enterprise`, `21usc841:substance`, and `18usc1962:racketeering.conspiracy`.
- In `assist-007` (NCAA antitrust trial): proposed `42usc2000e-2:employer-reason` and `42usc2000e-2:qualified`.
- In `assist-011` (RICO gang trial): proposed `21usc841:substance`, `15usc1:restraint.per-se`, `42usc1983:qualified-immunity`, and `42usc2000e-2:pretext`.
- In `assist-019` (Methamphetamine guilty plea appeal waiver): proposed `42usc2000e-2:pretext`, `42usc2000e-2:motivating-factor`, `42usc2000e-2:employer-reason`, and `42usc1983:color-of-law`.
- In `assist-020` (Supervised release termination): proposed `42usc1983:x-federal-action`, `42usc1983:x-collateral-estoppel`, and `18usc1962:pattern.continuity`.
### B. Invented & Non-Standard Variant Hallucinations
Almost all variant assignments failed (98% rejected). The model repeatedly hallucinated non-existent variants not found in any LAW DSL catalogue:
- `15usc1:state-action-immunity:private-parties/trade-association-members-claiming-parker-lgaa-noerr` (neither this element nor this variant exists in `15usc1.law`).
- `15usc1:restraint.rule-of-reason:anticompetitive-effect/single-competitor-excluded-no-market-proof` (invented string).
- `18usc1962:person:distinct-from-enterprise/corporation-vs-network-of-owned-and-independent-distributors`.
- `42usc1983:x-abuse-of-governmental-power:rejected-extra-element/...`.
### C. Inability to Recognize Hard Negatives
When presented with cases where the pilot statute was cited purely as a comparator (*Jett*), or where the target offense was a historical guilty plea not challenged on appeal (*Floyd*, *Damon*, *Steele*), the model generated 5 confident affirmative proposals per case anyway, targeting captions, procedural histories, or irrelevant legal boilerplate.
### D. Oversized Unparsed Slices
Rather than extracting the minimal sufficient factual sentence proving the element (as required by the protocol), virtually every machine proposal spanned 1,500 to 2,500 characters, indiscriminately sweeping across irrelevant statutory recitations, legal standards, and surrounding procedural narrative.
---
## 4. Effort Notes and Practical Utility
### Recorded Review Times
- **Phase A (Blind Reading & Inventory):** ~210 seconds per case on average (~70 minutes total across 20 cases). An annotator must read the opinion, understand the claims, verify whether the statute is actually applied, and extract clean text spans.
- **Phase B (Proposal Adjudication):** ~45 seconds per case on average (~15 minutes total across 100 proposals).
### Did the Proposals Save Effort?
1. **Methodological Finding (per Protocol):**
As emphasized in [`ANNOTATION_PROTOCOL.md`](file:///home/mma/repos/neochart/neov2/assist/ANNOTATION_PROTOCOL.md), Phase B directly succeeds Phase A and inevitably benefits from prior familiarity with the case text. Therefore, Phase B timings cannot establish causal effort savings. Under the protocol's evidentiary rules, effort savings are formally **NOT ESTABLISHED**.
2. **Substantive Annotation Reality:**
In practice, the proposal packets in their current state would **increase** rather than decrease annotator burden:
- With an 88% false positive rate (precision 12%) and a 35% cross-statute hallucination rate, an annotator reviewing proposals without prior blind reading would be forced to spend significant cognitive effort debunking irrelevant statutes, fictitious doctrines, and non-existent variants.
- Most critically, relying on proposals would result in catastrophic recall loss: **89.5% of true gold factual applications were completely missed** by the proposals (recall 10.5%).
- The system is not yet sufficiently reliable to act as an unassisted or minimally assisted annotation accelerator.
---
## 5. Artifacts and Commit Integrity
- All 20 Phase A/B JSON records written to [`neov2/assist/annotations-QA1/`](file:///home/mma/repos/neochart/neov2/assist/annotations-QA1/):
- `assist-001.json` through `assist-020.json`
- Each file records reviewer identity (`QA1`), source SHA256, verified blind inventory, and per-proposal judgments.