Source: assist/r5/POSTMORTEM.md
# One-page post-mortem: why embedding retrieval failed QA1's independent Round 4 held-out review reports **3/75 element/passage precision (4%) and 0/20 strict gold-span recall**, with only two gold concepts recovered elsewhere in the opinions. Two cases received no proposals. Exact variants were deliberately withheld. The failed result is preserved; the next experiment will not retune R4 using these labels. **Boilerplate trapping.** An element description resembles its legal test. Embedding a LAW checklist and retrieving nearby language therefore ranks statutory recitations and standard-of-review paragraphs highly. Lexical cues such as “plaintiff,” “evidence,” and “here” also occur in these paragraphs. QA1 reports that most rejections were generic legal standards. Tightening citation anchors removed cross-law output by construction, but did not establish that this court applied the element to these parties. Syllabi and counsel sections can mention the right statute without supplying operative findings. **Wrong unit of retrieval.** A 55-word segmentation plus overlapping windows uses text length as a proxy for legal structure. It can split the factual premise from the court's conclusion or combine one with an unrelated citation. The model sees similar wording, not a reliably identified court/party/claim/ opinion relationship. Comparator precedent facts are especially dangerous: they are concrete, semantically relevant and nevertheless about another case. A target-section gate also misses scope mismatches such as Title VII versus ADA analysis, §1983 versus §1988 fees, or §841 versus sentencing rules. **Applied versus announced.** The essential question is relational: what did THIS court decide about THIS element of THIS law on THESE parties' facts? A remand announcing a legal framework, a historical conviction, a procedural ruling and an applied factual finding can all share the same vocabulary. The frozen procedural filters were crude proxies; they could reject mixed cases and retain boilerplate. Removing exact variants prevented unsupported transfer of CASE-specific labels, but did not fix element attribution. **Recall needs careful interpretation.** Strict containment measures recovery of QA1's chosen evidence span. It gives zero when an accepted proposal captures a different passage for the same issue. QA1 explicitly identified that effect in Sacchi and Bittermann, and reported 2/20 substantive concept recovery. That nuance does not rescue a system with 72/75 rejected proposals. Neither shorter excerpts nor faster second-pass review establishes annotator savings; QA1 read the opinions before adjudicating, with no randomized control. **The next test.** Give a fresh LLM agent the entire provided canonical IR opinion and the target law's element checklist. Ask it to distinguish current application from quoted standards, comparator facts and procedural history; allow abstention and require a unique contiguous verbatim quote for every candidate. Mechanically locate exact quotes, reject rather than repair invalid ones, and retain all outputs as unreviewed. Freeze the prompt/protocol before 20 new cases, meter provider tokens and time per case, and send the unchanged batch to QA1. This tests whether whole-opinion interpretation improves the proposal mechanism; it does not assume an LLM's reasoning or quote fidelity makes its legal labels correct. Source: `neov2/qa/assist-r4-QA1.md`. No R4 labels or source text changed.