Source: qa/matcher-v2-QA2.md
# QA2 — independent review of matcher v2 evaluation **Pinned tree:** `c8534448fb90ccc86373efcc751885d5ba38aa09` (`origin/master`). I reviewed the v2 report, protocol, freeze/snapshot/evaluation artifacts, retrieval and statistics code, then reproduced the statistics and an initial 20-fold subset. No matcher source or evaluation artifacts were changed; only this QA report is in scope for commit. ## Findings ### Leakage controls — **PASS, with the cache distinction below** - The pinned freeze contains 165 task rows, 164 canonical source IDs, and 164 groups. It found no additional alias edges or near-duplicate candidates. The two *United States v. Kitchen* tasks are the same case ID/group, assigned to §841 and §922; group-level any-hit metrics union those two labels. All 180 authored input-file hashes in `freeze.json` match the files in the pinned tree. - Grouping is label-blind. It links canonical IDs, source hashes, normalized full text, reporter citations, caption/date pairs and the fixed five-word-shingle rule. The negative/positive alias audit also reports zero cross-set aliases and near-duplicate candidates. These are observed-data checks, not proof that every possible alias was found. - Each query fold excludes its entire alias group before training-fact selection. Training facts are restricted to `is_instance == 1`; the extra guard removes exact normalized held-out fact quotes and training quotes of at least ten words found verbatim in the query. In the recorded run, all 164 queries show zero held-out training-cache key overlap; the added quote guard removed zero passages. The full-corpus 20-fold rerun below also had zero overlap. - Training vectors are text/model-keyed, and each fold’s active training cache contains exactly the keys for that fold’s permitted fact rows. The fold-local cache is deleted after the query. A shared `content-model-pool` and `query-only` cache do retain deterministic embeddings across folds, including embeddings of text that is held out in another fold. These vectors contain no labels, scores, or fitted corpus state, and held-out vectors never enter that fold’s active training cache. Thus this is cache reuse, not supervised-label leakage. The report should retain this distinction when saying “held-out vectors” are excluded. - Gold element labels are used after ranking for measurement; ranking receives only each fold’s permitted fact rows and the fixed law/element catalogue. The run validates the frozen snapshot and code hashes, and source IR hashes for all 164 query records. ### BM25 baseline — **PASS as a fixed lexical reference** BM25 and vector retrieval use the same fold-specific applied-fact rows, full-opinion chunks and top-20-per-chunk candidate depth; Lucene is rebuilt within each fold so held-out documents cannot affect IDF. The hybrid is fixed RRF, with no parameter search. BM25 is therefore a fair, reproducible baseline for this comparison, though it is not a tuned lexical system. All methods aggregate the best element scores and average the top three per law, dividing by three. This common rule disadvantages catalogues with fewer than three supported elements; the report discloses it. The comparison is also limited to these 15 authored law catalogues and this curated training corpus. ### Statistics and headline claims — **PASS with the reported endpoint distinction** Running `/tmp/neov2-search-venv/bin/python neov2/matcher/v2/report_results.py` recomputed the statistics. `REPORT.md`, `summary.json`, and `evaluation.json` retained their pre-run SHA-256 values (`06221e09…`, `5bc1d950…`, and `3d283de0…`, respectively). - The task-level top-1 counts are 78/165 hybrid versus 63/165 BM25 (47.3% vs 38.2%, a 9.1 percentage-point descriptive difference). The paired test is correctly run at the independent opinion-group level: Kitchen’s two statute tasks form one group, and an opinion is a hit if either associated law is in the top 1. That view is 78/164 vs 63/164. Exact McNemar gives `p=0.0769`, Holm-adjusted `p=0.2308` across the three top-1 method comparisons; the top-1 gain is not statistically established. - Top-3 is 133/165 vs 107/165 on task rows; the opinion-group any-hit view is 133/164 vs 107/164. The paired group McNemar test is appropriate: BM25-only/hybrid-only successes are 4/30, exact `p=6.16e-6`, within-endpoint Holm `p=1.85e-5`. The report’s qualified claim that the top-3 gain survives this correction is accurate. Holm adjustment is only within the three comparisons for each endpoint, as disclosed; it is not a single correction over every reported endpoint. - The 2,000-draw bootstrap resamples opinion groups and holds out-of-fold outcomes fixed. This is a reasonable interval for this curated collection, not a refit or a prospective test. The paired direction tests use group-level outcomes and exact binomial sign/McNemar calculations. - The report separates 259 gold element labels into 183 with training support and 76 without it. Hybrid recall is 135/259 = 52.1% over all gold and 135/183 = 73.8% conditional on support; the support ceiling is 70.7%. This is the right separation. Its 24.2% precision is an annotation-retrieval measure: unannotated predictions are nonmatches, not proven legal errors. ### Negative gate — **PASS as a cautionary diagnostic only** The threshold results use eight clean citation-negative proxies from the original ten IDs; two were contaminated after rescreening against all 15 laws. At the original 0.5 threshold, all 8/8 proxies pass (zero specificity). At 0.6, 0/8 pass but only 78/164 association opinions are accepted. The conclusion that the original gate fails is supported. These eight citation-negative cases are not expert-adjudicated non-applicability labels, are not newly held out, and cannot validate a legal abstention threshold; the report says so. ## Independent 20-fold reproduction I ran the frozen evaluator on the first 20 sorted opinion groups: ```bash OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=2 \ NEOV2_EMBED_CACHE=/tmp/neov2-model LMLUCENE_BIN=/tmp/lmlucene-linux-x64 \ /tmp/neov2-search-venv/bin/python neov2/matcher/v2/evaluate.py run \ --corpus /home/mma/repos/neochart/neochartdocroot \ --cache /tmp/neochart-matcher2-qa2-cache --limit 20 ``` The 20 rerun groups, 20 task rows, and 20 query audits exactly matched the corresponding prefix of the frozen full-run `evaluation.json` on all ranking outputs, gold/seen labels, training-row and cache-key hashes, exclusions, copied-passage guards and cosine values; wall-clock seconds were excluded. The subset’s group top-1/top-3 counts were BM25 7/20 and 14/20, vector 11/20 and 16/20, hybrid 12/20 and 18/20. The full run’s held-out training-cache overlap is zero for all 164 queries. The rerun used only `/tmp` caches; its partial checkpoint was removed. ## What is the law-level matcher for if `cites_to` exists? For the simple question “which statutes does this opinion cite?”, the `cites_to` graph is the direct, more auditable answer; the matcher should not replace it. A useful additional role is second-stage triage: rank which cited laws appear to be applied to the live dispute, surface analogous element-level fact passages from prior opinions, and flag plausible legal issues that citation extraction misses. It can help a reviewer navigate evidence, but its low precision and curated annotation target do not support automated legal-applicability or liability conclusions. The v2 evaluation has no explicit `cites_to` baseline, so it establishes no incremental value over that graph for identifying cited statutes. ## Overall assessment The report’s main numeric claims reproduce. The strongest result is the paired top-3 improvement; the top-1 difference is uncertain. Leakage controls correctly isolate labels and evidence at the fold level, while shared deterministic vector caches are inference caches rather than independent fold storage. The negative-gate result is a warning about eight proxies, not a validated gate. The matcher’s defensible purpose is evidence retrieval and citation triage, not replacing explicit citation edges or deciding legal applicability. **Reproduction evidence:** frozen input manifest matched 180/180 files; 164/164 recorded queries report zero held-out training-cache overlap; 20/20 rerun groups matched; recomputed statistics/artifact hashes were unchanged. No repository artifacts other than this report are intended for commit.