Source: assist/r4/ERROR_ANALYSIS.md
# Round 4 proposer error analysis (development cases 1–20 only) This is a development-only audit of `assist-001` through `assist-020`. It reads only the QA1 report, those twenty QA1 annotation files, those twenty proposal files, the corresponding canonical IR records, the frozen matcher catalogue and the Round 3 sample metadata. Cases 21–40 and their source text were not opened. No annotation, proposal, LAW file or source text was changed. ## Integrity checks All 20 proposal source pointers resolve to the canonical IR record. Every proposal SHA256 matches the full source, and every proposal `[start,end)` slice matches its quoted text. Every QA1 gold SHA256 and quote offset matches the same source. The only structural anomaly is `assist-017-p8e0e028dbb`: it is present in the proposal packet but has no corresponding QA1 judgment. The 100 QA1 judgment records therefore contain 99 recognized proposal IDs plus one unknown ID in place of a judgment for that candidate. This does not change QA1's reported 12 accepted proposals, but the candidate should not be silently counted as rejected. QA1's totals reconcile at the case and proposal level: | Measure | Audit | |---|---:| | Cases | 20 | | Proposals | 100 | | Gold evidence items | 19 | | Accepted element/passage proposals | 12 | | Accepted exact variants | 2 | | QA1 reported element precision | 12/100 = 12.0% | | QA1 reported variant precision | 2/100 = 2.0% | The stated 2/19 recall does not follow from a strict source-span containment test on the JSON. One accepted proposal (`assist-013`, conspiracy) contains its gold span, while two other accepted proposals overlap a gold span but begin or end outside it. Counting accepted same-element overlaps gives three candidate hits; requiring complete containment gives one. QA1's 2/19 likely uses a different matching convention. The report should preserve 2/19 as QA1's independent metric and label the exact matching convention before using it for regression comparisons. This audit does not substitute a new gold metric. Gold element IDs without a law prefix normalize against the target law in `assist/sample.json`, never against the first retrieved proposal. Under that normalization all 19 gold elements exist in the active frozen catalogue. The earlier apparent missing elements were an artifact of using a cross-statute first proposal as the prefix. No gold item is evidence that the catalogue is missing an element. ## Why proposals were false positives The proposal generator selected the best retrieved fact for every catalogue element, then emitted up to five elements regardless of whether the case actually discussed the target statute. This creates a positive proposal quota for negative/background cases. QA1's narrative report identifies 11/20 cases as negative or background, but the machine-readable QA1 files contain nine empty `gold` arrays. That discrepancy must remain explicit; this audit uses the JSON for counts and does not infer two additional gold negatives. The nine JSON negative/background cases still received five candidates each. Across rejection reasons, categories are non-exclusive. The audit counted 47 cross-statute or law-mismatch reasons, 18 procedural/comparator/background reasons, 2 explicit irrelevant/unsupported reasons, and 32 other rejection reasons. These add above 88 rejected proposals because one reason can support multiple categories and because the QA rationale vocabulary is heterogeneous. QA1's independently reported cross-statute contamination is 35/100; the reason-text count is a broader diagnostic and must not replace QA1's number. The source mechanics explain the errors: * The frozen facts index contains factual passages from all five pilot laws, but retrieval is not conditioned on the case's cited statute. Generic terms such as “enterprise,” “pattern,” “agreement,” “market,” “conduct,” and “person” therefore retrieve semantically similar passages from another law. In this audit, 48/100 proposal `law_id`s differ from the sample's target law; QA1's stricter adjudicated contamination count is 35/100. * Proposals are long windows (median 1,650 characters; mean 1,621; range 161–2,487) rather than minimal evidence spans. The window often includes a statute recitation, a legal standard, procedural history and a comparator discussion. A correct term match inside a window does not establish that the case applied the proposed element. * Only 25/100 proposal quotes contain an explicit USC section anchor. The remaining proposals rely on semantic similarity alone, so a headnote-like rule statement or an opinion's discussion of another statute can win. * Variant strings in the packets came from the frozen facts/catalogue retrieval database; they were not fabricated source text by the proposer. QA1 still found several strings non-standard or unavailable for the target LAW catalogue, and exact variant precision was 2/100. The redesigned proposer must validate every returned variant against the target law's declared IDs, and emit `variant_id: null` with an explicit `unmapped_variant_candidate` note when no exact catalogue entry fits. This distinguishes provenance from legal validity. * The generator does not distinguish applied merits discussion from `comparator`, `background`, `rule-announced`, procedural, dissent or historical passages. QA1's examples include Hobbs Act comparator discussion, juror contamination, §924(c) appeal material, appeal waiver and supervised release proceedings that merely cite a pilot statute. ## Why evidence was missed The retrieval index is built from prior annotated fact quotes, and the query is the entire new opinion chunk. A gold fact may use case-specific language absent from the nearest neighbor quotes. The best passage for a catalogue element can therefore be a doctrinal rule or a neighboring statute instead of the factual sentence QA1 marked. The fixed five-proposal quota also spends the budget on unrelated elements and cross-statute neighbors. Long windows dilute the signal and make exact source alignment poor. Element labels are assigned after global hybrid ranking rather than by an element-conditioned query. No citation-anchor gate requires the passage to discuss the target section, and no role filter removes procedural or comparator text. These mechanisms explain both the 88% false-positive rate and the low evidence recall without implying a model hallucinated text: all quoted windows were real source slices. ## Design requirements for the frozen Round 4 proposer The next proposer should be frozen before generating cases 21–40 and its design hash recorded in every packet: 1. Resolve the target statute from sample metadata and require a citation or discussion anchor for that statute (including subsection forms) before proposing an element. A mere citation can be retained as context but should not pass the substantive-evidence gate by itself. 2. Condition retrieval on one declared law/element at a time. Query with the law element's declared name, existing variants and rule vocabulary, then rerank candidate windows by lexical element cues and target-statute anchors. 3. Restrict variant output to exact IDs in the active LAW catalogue. If none fits, emit `variant_id: null` with an explicit `unmapped_variant_candidate` note; never invent a variant ID. 4. Classify passage role before output (`applied`, `rule-announced`, `comparator`, `background`, `procedural`, `dissent`, or `unknown`) and make only `applied`/reviewable `rule-announced` candidates eligible by policy. 5. Replace 1,500–2,500 character windows with sentence-aligned minimal spans, preserving exact Unicode offsets and the surrounding context separately. 6. Allow zero proposals. Negative/background cases must be able to return an empty packet; a fixed five-element quota makes negative discrimination impossible. The first 20 cases are development data only. Any dev-set precision/recall after these changes must be reported separately from the blind 21–40 result. QA1's labels remain independent and unchanged; this analysis does not certify a legal holding or approve a proposer redesign.