Source: qa/matcher-QA2.md
# QA2 review — matcher evaluation ## Overall verdict: PASS for the stated, narrow retrieval claim The evaluation is reproducible on its frozen snapshot and supports the descriptive results for this curated 60-case annotation set. It does not establish prospective law prediction or a reliable no-match detector. The report mostly states those limits; the findings below clarify what the numbers can support. ## Re-run and leakage review — PASS, with scope limits Re-ran the committed evaluation from `evaluation_snapshot.json` with the committed negative set, using a separate embedding cache and the native Lucene path. It completed 60 positive folds and 10 corpus-negative proxy cases. The snapshot SHA-256 and model identity match the committed `evaluation.json`; all reported recall and top-1/top-3 counts reproduced exactly. The rerun took 208.528 seconds (median query 2.845 seconds) with the separate cache; this is not comparable to the report's warm-cache timing. The fold removes the held-out `case_id` before deriving labeled fact rows; the evaluator asserts that the held-out ID is excluded and absent from retrieved evidence. It also excludes all cases whose full-source hash exactly matches the query. I independently checked the frozen snapshot: there are no multi-case identical-source-hash groups, and all 60 folds produce zero indexed rows from their excluded case/hash set. Lucene creates a fresh temporary index per fold, so document frequency/IDF is fit only on that fold's training facts. Embeddings use per-vector L2 normalization, not a corpus-fitted scaler; cache keys contain the input text and pinned model identity, not labels. Held-out query chunks are embedded as queries, as expected, but their annotated fact rows are not sent as training vectors. This is a clean **case-level computational leave-one-out**, not a blind external test. The authors selected and annotated the same 60 opinions, and CASE facts/labels were derived after reading those opinions. That makes this an in-sample human-curated retrieval benchmark even though the held-out case's label rows are filtered correctly at runtime. The report discloses the hand-selected sample and calls the metric annotation retrieval; keep that scope prominent. The full opinion is also the query, including its statute citations, law names, procedural history, holdings, and discussions of other cases. That makes law ranking easier than deciding applicability from an incoming fact-only record, as the report acknowledges. Model pretraining overlap with these opinions remains unknown. ## BM25 baseline — PASS for a fixed lexical comparator BM25 and the hybrid use the same case chunks, training-fact text, StandardAnalyzer, and per-chunk top-20 lexical results. The Lucene index is rebuilt for every fold. The BM25 baseline uses raw scores scaled by the maximum score within that query case; this is a single positive multiplier for its within-case aggregation and does not change BM25 ordering. Hybrid adds the top-20 vector channel with fixed reciprocal-rank fusion. The implementation contains no parameter-search loop, and the rerun reproduces the committed counts. This is fair as a transparent BM25-only comparator for the implemented chunk/query settings, not as a tuned or exhaustive lexical baseline. The code/report do not provide an external record that these settings were frozen before inspecting the labels, so the “not tuned” history cannot be independently verified from the artifacts alone. ## Top-1 comparison and paired significance — PASS The rerun gives BM25 54/60, vector 48/60, and hybrid 58/60 top-1. BM25 and hybrid each reach 60/60 top-3; vector reaches 56/60. For hybrid versus BM25, 6 cases change from wrong to right, 2 change from right to wrong, 52 are right for both, and none are wrong for both. The two-sided exact McNemar test on those paired outcomes gives `p = 0.2890625`. Thus the observed 4-case net gain (6.7 percentage points) is not statistically significant at usual thresholds and should not be described as a demonstrated general improvement. The report already states this correctly. The hybrid's two top-1 errors are `/f-supp/772/0583-01` (gold `42usc1983`, predicts `42usc2000e-2`) and `/us/526/0040-01` (gold `42usc1983`, predicts `18usc1962`). ## Element/variant recall and denominator — PASS, add a support-conditioned view The reported 39.1% is exactly 75/192 top-five element hits with the gold law supplied; the end-to-end numerator is also 75/192 in this run. The denominator includes labels that the model cannot score in a fold: 46/192 gold element labels have no supporting fact label in the remaining training cases. Since element scores are propagated only from labeled training facts, these labels have a zero-recall ceiling. Among the 146 element labels with training support, the hybrid hits 75/146 (51.4%); the overall ceiling from label coverage is 146/192 (76.0%). Keep 39.1% as the conservative all-label metric, but display the support-conditioned number next to it so it is not mistaken for performance on labels the model had a chance to retrieve. The same issue is larger for variants: 228/235 exact gold variants are unseen in their fold, leaving only 7 with any training support. Hybrid's 1/235 (0.4%) is correct as an all-label exact-match rate, while 1/7 (14.3%) is the seen-variant rate and 7/235 (3.0%) is the coverage ceiling. The report explicitly gives the 228 unseen count and correctly says exact variants are mostly unavailable; adding the two rates would make that result easier to interpret. Neither metric measures whether a nearby variant is legally analogous. Element and variant targets are the authored CASE annotations, not independently adjudicated legal ground truth. QA1/QA2 review has found some case-level law/element scope and holding-label issues in this same annotation corpus. The report properly limits itself to annotation retrieval; its scores must not be read as legal correctness or applicability accuracy. ## Negative results — PASS on the stated proxy set All 10 selected cases still receive a top-ranked pilot law. The fixed cosine `>= 0.5` diagnostic accepts all 60 positives and all 10 negatives; the negative max-cosine range is 0.526–0.586. That is zero specificity on this set and shows the tested threshold cannot support abstention. The cases are citation-screened corpus negatives, not expert-adjudicated non-applicability examples, and the report labels them a proxy rather than legal negatives. Keep the result scoped to this small selection; it does not estimate specificity on the wider case population. ## Recommended report additions - Show support-conditioned recall beside 75/192 element recall and 1/235 exact-variant recall. - State in the results summary that leave-one-out is over labels from a hand-selected, human-reviewed collection, rather than a blind prospective test. - Keep the negative result phrased as failure on the 10 citation-screened proxies, not proof that a legal applicability classifier has zero specificity generally. ## Verification evidence - Rerun: `/tmp/neov2-qa2-evaluation.json` (temporary output; repository evaluation artifacts were not changed). - Frozen snapshot hash and model identity match `neov2/matcher/evaluation.json`. - Re-run counts: BM25 54/60 top-1; vector 48/60; hybrid 58/60; exact paired McNemar `p = 0.2890625`; 60/60 positives and 10/10 negative proxies pass the `>= 0.5` cosine gate. - The matcher implementation, frozen evaluation artifacts, case annotations, and corpus were not edited.