Source: xref/out/audit.md
# Missed-citation audit — 2026-11-10 300 case pages were selected by seeded reservoir sampling from 2,023,221 caselaw entries in the local corpus manifest (seed `20261110`). This is a sample of hosted records, not unique judicial decisions; parallel publications can duplicate opinions. All input files were opened read-only. No corpus build or deployment was performed. ## Evidence and offsets `audit_sample.json` records the selected URLs and SHA-256 hashes of their HTML and decoded opinion text. `audit_candidates.jsonl` contains 10,263 regex/eyecite candidates with exact quotes, context, URL, character offsets, and overlapping link destinations. Offsets refer to concatenated HTML-decoded data nodes inside `div.opinion-body`, **not raw HTML bytes or downloaded source offsets**. Whitespace is preserved. Reconstruct this representation with `audit.Opinion`; verify the stored hash before using an offset. No source text was rewritten. A candidate is “linked” when its interval overlaps an existing anchor. That is a screening signal, not proof of a correct target. Unlinked candidates include ambiguous names, unsupported legal bodies, and absent hosted targets; they are not all extraction failures. Detectors cannot establish a complete gold denominator, so full-corpus recall and end-to-end before/after link precision are **not measured**. ## Classified candidates | Family | Unlinked | Linked | Interpretation / proposed addition | |---|---:|---:|---| | Full case citation | 2,792 | 2,578 | Some missing variants/targets; 35 occurrences also expose malformed canonical keys. Fix the key formatter. | | Short case citation | 380 | 550 | Use resolved full-case antecedents; no new context-free rule. | | Eyecite Id. | 492 | 301 | Antecedent resolution and target availability matter. | | Regex-only Id. | 225 | 10 | Includes unresolved/unsupported antecedents. | | Eyecite supra | 184 | 48 | Keep resolution scoped to context. | | Regex-only supra | 43 | 0 | Candidate only; no unique target established. | | U.S.C.A. / U.S.C.S. | 30 | 0 | Implement canonical-code alias handling; abstain on plural numeric ranges. All sampled variants are U.S.C.A. | | Other full law cite | 147 | 204 | Mix of supported and unsupported bodies; requires target inventory. | | Bare section | 1,780 | 134 | State-code and federal ambiguity; do not globally map a bare number. | | “Section 1983” | 4 | 0 | Candidate for context-gated federal civil-rights resolution; not implemented. | | Title VII | 96 | 0 | Act-level family, not a unique section; needs an act landing target. | | RICO | 2 | 0 | Federal/state ambiguity; needs family and jurisdiction context. | | “the Act” | 124 | 0 | Anaphora; retain unresolved unless an antecedent is established. | | Session law | 12 | 0 | Needs Statutes at Large / public-law inventory and mapping. | | CFR | 71 | 0 | Needs regulation inventory; never coerce to USC. | The complete breakdown, including unknown/reference/journal candidates, is in `audit_summary.json`. Short references can overlap other candidates; these totals are detector occurrences, not unique legal propositions. ## Hand-checked measurements Astra inspected 67 quoted mentions and their contexts. Selection deliberately concentrates on the proposed fixes: all 30 annotated-code mentions, 15 of the 35 broken full-case keys selected by stable candidate ID, 10 other full-case controls, and two examples from each of six alias/regulation/session-law families. This diagnostic sample is **not representative**, and the labels are engineering annotations, not legal review. `audit_hand_checked.json` records each expected canonical key and the inspection rationale. Statute scoring rolls subsections up to sections. Nine mentions have no single section target: three numeric ranges and six act-level/anaphoric references. Their correct outcome is abstention. The other 58 have one expected key each, including six families the patch leaves unsupported. | Targeted extraction metric | Original `91980f0` | Patched | |---|---:|---:| | Correct keys (TP) | 10 | 52 | | Incorrect keys (FP) | 15 | 0 | | Missing keys (FN) | 48 | 6 | | Precision: TP / (TP + FP) | 40.00% | 100.00% | | Recall: TP / (TP + FN) | 17.24% | 89.66% | These are **citation-key extraction** measurements on isolated quoted mentions, not deployed hyperlink results. All 42 additional correct keys in this targeted set were checked against their source contexts. Annotated-code singular-reference precision is 27/27; three plural numeric ranges correctly abstain. Case-key repair precision is 15/15 in the checked broken-key subset. The ten ordinary-case controls remain correct. U.S.C.S. and negative-format behavior are covered by synthetic regression tests, not empirical sample claims. The unchanged six misses are two contextual “Section 1983” references, two session-law cites, and two CFR cites. Among the 27 singular annotated-code mentions, only 17 have an exact canonical key in the current citation index. Ten lack that key. Even an available key may refer to a later statute version: this sample includes historical Bankruptcy Act § 501 and transferred provisions. **Historical identity is not established by key equality.** This patch does not solve version-aware linking. ## Implemented authoring patch - `citations.py`: recognize explicitly titled U.S.C.A./U.S.C.S. forms, including spaced abbreviations; preserve lookup normalization and decline plural numeric ranges. - `build_site.py`: recognize those forms in single/list link matching, preserving the exact displayed citation. The patch does not create target inventory. - `eyecite_extract.py`: construct case keys from parsed volume, corrected reporter, and page. The former `find(page)` truncated citations such as `137 F.3d 1` or `363 F.3d 363` when page digits occurred in the volume. - `test_eyecite_extract.py`: regression coverage for canonical keys, correction failures, annotated forms, ambiguous suffixes, range abstention, and verbatim rendered text. Existing IR caches store extraction results. Applying this code alone does not refresh them. A future separately scoped build must invalidate/reparse affected IR and refresh graph/resolution/rendering; a source-SHA-only cache reuse would retain the old keys. No deployment to neochart.com was requested or performed. ## Reproduce From repository root, with Python 3.12 and the pinned authoring audit dependencies: ```sh uv venv /tmp/neov2-astra-venv --python 3.12 uv pip install --python /tmp/neov2-astra-venv/bin/python -r neov2/xref/requirements-audit.txt /tmp/neov2-astra-venv/bin/python neov2/xref/audit.py --reuse-sample /tmp/neov2-astra-venv/bin/python neov2/xref/evaluate.py --output neov2/xref/out/audit_after.json mkdir -p /tmp/neov2-astra-baseline git show 91980f0:neochartdocroot/authoring/citations.py > /tmp/neov2-astra-baseline/citations.py git show 91980f0:neochartdocroot/authoring/eyecite_extract.py > /tmp/neov2-astra-baseline/eyecite_extract.py /tmp/neov2-astra-venv/bin/python neov2/xref/evaluate.py --authoring /tmp/neov2-astra-baseline --output neov2/xref/out/audit_before.json python3 neov2/xref/stats.py --as-of 2026-11-10 ``` Omit `--reuse-sample` to reselect from the manifest with the recorded seed. Statistics use only the standard library. Audit/evaluation fail rather than silently replacing eyecite with regex-only results. The system Python lacks eyecite; use the isolated environment. ## Verification and independent review 32 relevant tests pass across citations, rendering/key regression, resolver, statistics, audit sampling/offsets, and metric accounting. Python compilation and `git diff --check` pass. Independent QA1 and QA2 found no blocking issue; both required the targeted-metrics limitation above. These are code/evidence reviews, not substantive legal signoff. No source-law or case files were changed.