Source: matcher/REPORT.md
# Astra3 Round 2: matcher evaluation — 2026-11-10 Hybrid retrieval improves law ranking on this selected sample: **58/60 top-1 versus 54/60 for raw Lucene BM25**. Both reach 60/60 top-3. Element recovery remains limited, exact variants are mostly unseen in training, and the tested negative detector fails. This is an evidence retrieval PoC, not a calibrated legal applicability classifier. ## Results | Method | Law top-1 | Law top-3 | Element recall@5, gold law supplied | End-to-end element recall@5 | Satisfied-element recall@5 | Variant recall@5 | |---|---:|---:|---:|---:|---:|---:| | bm25 | 54/60 (90.0%) | 60/60 (100.0%) | 73/192 (38.0%) | 67/192 (34.9%) | 47/118 (39.8%) | 1/235 (0.4%) | | vector | 48/60 (80.0%) | 56/60 (93.3%) | 67/192 (34.9%) | 55/192 (28.6%) | 43/118 (36.4%) | 3/235 (1.3%) | | hybrid | 58/60 (96.7%) | 60/60 (100.0%) | 75/192 (39.1%) | 75/192 (39.1%) | 48/118 (40.7%) | 1/235 (0.4%) | Hybrid fixes six BM25 errors and introduces two different errors. The paired two-sided exact McNemar p-value is 0.289: this small curated sample does not establish a general improvement. Hybrid does **not** improve top-3 or exact-variant recall over BM25; conditional element recall rises by only 2/192. No scoring parameters were tuned on these labels. Per-statute top-1 counts: | Statute | Cases | BM25 | Vector | Hybrid | |---|---:|---:|---:|---:| | 15 U.S.C. § 1 | 12 | 9 | 9 | 12 | | 18 U.S.C. § 1962 | 12 | 12 | 12 | 12 | | 21 U.S.C. § 841 | 12 | 9 | 12 | 12 | | 42 U.S.C. § 1983 | 12 | 12 | 5 | 10 | | 42 U.S.C. § 2000e-2 | 12 | 12 | 10 | 12 | Hybrid errors: - `/f-supp/772/0583-01`: gold `42usc1983`, predicted `42usc2000e-2`. - `/us/526/0040-01`: gold `42usc1983`, predicted `18usc1962`. ## Evaluation contract and leakage controls All 60 known cases are evaluated once using complete canonical IR text, sentence-aligned at approximately 250 words. In each fold the whole held-out case is removed from the facts database before indexing. Exact matching source hashes also exclude aliases. Lucene is rebuilt per fold, so held-out documents cannot influence BM25 document frequencies. Model/cache inputs contain text and model identity only; no case labels enter embeddings. Only CASE evidence with `kind=fact` is indexed. Rules, held passages, glosses, LAW quotes, law titles, element names and variant labels are not retrieval text. Repeated facts with identical case/offsets/text are deduplicated, retaining all annotations. Opposing or not-reached neighbor holdings remain labeled as such; a neighbor is not a finding about the new case. The frozen export contains 5 laws, 121 metadata elements, 60 cases, 1172 evidence records (572 factual), 494 instances and 288 variants. `evaluation_snapshot.json` is a curated annotation export, not a raw corpus dump. All CASE source pins and verbatim slices were verified while building it. `sources.json` records the authored inputs. `evaluation.json` contains all per-case outcomes, source hashes, exclusions via training counts, gold labels and unseen-label counts. Law gold is the authored pilot-law association. Element gold is the set of reached (`satisfied` or `not`) elements supported by factual quotes; rule-only, held-only and not-reached annotations are excluded from this particular metric. The satisfied-only column narrows that denominator further. Conditional recall takes the top five positive-score elements within the gold law. End-to-end recall counts those hits only if the correct law ranks first. Variant recall uses the top five positive-score variants within the gold law. These metrics measure annotation retrieval, not correctness of legal outcomes. One case (`/f-supp-2d/427/0894-01`) has no quoted evidence: it remains in law accuracy, with no element denominator. There are 46 held-out element labels and 228 of 235 exact variant labels with no supporting fact in the remaining cases. All stay in recall denominators. The variant catalogue is therefore too case-specific for strong leave-one-out exact-label recovery. Full opinions contain law names, citations, doctrinal discussion and quotations of other opinions. Thus law accuracy is optimistic relative to fact-only intake. Known cases are hand-selected, not a random test distribution. Pretraining overlap is unknown. LAW metadata is available to name outputs but is never scored, including interpretation rules authored from the test cases. QA1/QA2 legal review is separate. ## Retrieval and scores Native lmlucene v0.1.19 supplies BM25 with StandardAnalyzer. Per chunk, the first 200 distinct alphabetic tokens form an OR query; top 20 fact hits are retained. Vector retrieval uses exact NumPy cosine over normalized 384d embeddings, also top 20. Hybrid uses fixed reciprocal-rank fusion `0.5 * (61/(60+r_bm25) + 61/(60+r_vector))`, with missing-channel contributions zero. The scaling puts the best possible fusion score at one; it is not probability calibration. For each fact, retain its best chunk score; for each annotated element/variant retain the best fact score. A law score is the sum of its three highest element scores divided by three (missing elements contribute zero). BM25 baseline uses raw Lucene scores divided by the maximum raw score in that query case; vector baseline uses nonnegative raw cosine. Both share the same aggregation. This avoids the artificial ties produced by a rank-only single-channel baseline. The earlier rank-only trial is not the reported BM25 baseline. CLI results contain every pilot law and catalogue element/variant, the actual query chunks and offsets, and linked neighbor evidence with source URL, source pointer, Unicode offsets, exact quote and full-source SHA256. Scores cannot be compared as liability probabilities across methods or cases. No Boolean legal-rule evaluation or new variant generation is claimed. ## Genuine local embeddings Windows-host Ollama was unreachable in Round 1. This round uses CPU ONNX Runtime with `Xenova/all-MiniLM-L6-v2`, pinned to revision `751bff37182d3f1213fa05d7196b954e230abad9`. The model is Apache-2.0; download URLs, SHA256 hashes and pooling parameters are in `model.json`. Every artifact hash is verified before loading. No synthetic lmsearch embeddings are used; no model bytes enter Git. Tokenizer windows contain at most 256 tokens with stride 32. Attention-mask mean pooling produces window vectors; window vectors are averaged and the final vector is L2-normalized. All overflow windows are included, preserving passage tails. CPU inference batches at most 16 windows. This is an explicit long-text extension of ordinary MiniLM truncation, not a claim of equivalence to one full-context model pass. Cache keys include text, model/tokenizer hashes, revision and pooling parameters. Cache files use atomic replacement. ## Negative findings Ten fixed corpus cases were selected from non-pilot citation lists and screened against extracted citations and full-text pilot-citation patterns. Their IDs and selection criteria are in `negative_cases.json`. The evaluator rechecks those conditions and excludes overlap with the 60 known cases. “No detected citation” is a corpus-negative proxy, not proof that no pilot law could apply. Citation extraction/regex can miss citations. All 10 negatives still receive a top-ranked pilot law. A predeclared diagnostic threshold of max cosine ≥0.5 accepts **60/60 positives and 10/10 negatives**: zero specificity on this set. Negative maximum cosines range from 0.526 to 0.586. Max-over-many-chunks similarity and within-query ranking do not provide a safe abstention rule. The CLI therefore does not pretend that a high score proves applicability. A calibrated no-match model needs a separate, larger adjudicated negative/development set; this task did not tune a threshold after observing negatives. ## Reproduce ```bash uv venv /tmp/neov2-search-venv uv pip install --python /tmp/neov2-search-venv/bin/python -r neov2/matcher/requirements.txt # Native Lucene setup: neov2/specs/astra3-results.md (LMLUCENE_BIN/LMSCRIPT_BIN/LMLUCENE_SCHEMA). # Immutable evaluated data, independent of further annotation edits: /tmp/neov2-search-venv/bin/python neov2/tools/match.py /path/to/new-case.txt \ --db neov2/matcher/evaluation_snapshot.json /tmp/neov2-search-venv/bin/python neov2/tools/evaluate_matcher.py \ --db neov2/matcher/evaluation_snapshot.json # Or build a fresh annotation snapshot (changes the evaluated dataset): python3 neov2/tools/matcher_data.py /tmp/neov2-search-venv/bin/python neov2/tools/match.py /path/to/new-case.txt ``` `NEOV2_EMBED_CACHE` controls model storage (default `/tmp/neov2-model`); matcher embedding cache defaults to ignored `neov2/matcher/cache/`. SQLite defaults to ignored `neov2/db/matcher.sqlite`. It has explicit metadata-only LAW elements and is separate from the canonical loader database, whose schema requires statutory anchors. The adapter accepts the committed block LAW format and the indentation format through its existing parser; it never fabricates statutory offsets. Atomic construction verifies CASE quotes, references and stable authored source files. The frozen JSON can be queried without rebuilding the database or importing the indentation LAW parser. Other agents continued revising CASE/LAW annotations during this task; those later revisions are not retroactively included in this evaluation. The fresh-snapshot command was also verified against the later current files, but its scores require a new evaluation. Final evaluation runtime: 93.323s; median per-case query 1.276s with warm embedding cache. A synthetic new-case CLI smoke test completed in 1.240s and ranked RICO first; that smoke test is not an additional accuracy measurement. First-time model download and embedding costs are excluded from warm timings. Validation: nine focused tests (leakage/exclusion, exact query slices, corrupt quote rollback, missing-reference rollback, unseen-label denominators, model identity cache separation, genuine semantic shape/norms and long-text batching); 60 native leave-one-out queries; ten verified corpus negatives; actual new-text CLI smoke; Ruff, Pyright and syntax checks. No corpus edits or website deployment. Independent QA1/QA2 review remains pending.