Source: xref/out/r2/report.md
# Astra Round 2: citation resolution and stratified audit ## Result The research resolver annotates exact source spans for case short forms/Id., explicit USC-backed section references, Pub. L./Statutes at Large, CFR, and named acts. Ambiguity is retained as an unresolved record. Authoring integration is an **unapplied opt-in patch**; QA1 review is requested, not approved. Nothing was rebuilt or deployed on neochart.com. ## Sampling and provenance - Phase one: SRS reservoir of 10,000 caselaw manifest records from 2,023,221; seed 20261110. - Phase two: 2,000 pages allocated across 915 actual rendered court-label/decision-decade strata, with at least one per represented stratum and remaining capacity allocated proportionally. Court labels are whitespace-normalized page metadata, not reporter proxies or a canonical court registry. - Every frame record retains its page hash. Selected pages are checked before decoding. The frame, quotas, selected URLs, HTML/opinion SHA-256 values, and weights are committed. - The case record is the sampling unit; duplicate judicial opinions across publishers may remain. No claim of a complete unique-decision census is made. - Final resolver SHA-256: `ee25f313096dbd2dd9102047711f9a50328e81d81c1cc954b80c4469b20cc686`. ## Precision checks 350 resolved predictions were checked against exact source text/context: 50 in each of seven families. The final resolved Act-alias population in the 2,000-page sample is only 32, so all 32 were checked and supplemented with 18 targeted resolved aliases from unused phase-one pages. Supplemental records are excluded from all recall estimates and main audit counts. This satisfies the 50-check target without pretending the supplemental sample is random within the main audit. | Family | Checked | Correct | Observed precision | |---|---:|---:|---:| | act_anaphor | 50 | 50 | 100.0% | | case_id | 50 | 50 | 100.0% | | case_short | 50 | 50 | 100.0% | | cfr | 50 | 50 | 100.0% | | named_act | 50 | 50 | 100.0% | | session_law | 50 | 50 | 100.0% | | statute_short | 50 | 50 | 100.0% | These are engineering annotation checks, not legal signoff or deployed-link precision. A 50/50 result does not establish perfect population accuracy (a working binomial Wilson interval is approximately 92.9–100%, ignoring within-opinion clustering). The Act supplement is targeted; its pooled result is descriptive only. Names and statutory text were checked as written; no claim of correct historical version, hosted target availability, or OCR repair is made. The first development snapshot flagged 11 of 50 Act-anaphor labels as malformed, incomplete, or referring to a different act (39/50 accepted). Examples include “Yet the Act,” “II of the Act,” and an AWPA reference incorrectly assigned to FLCRA. The final rule requires an explicit same-paragraph parenthetical binding such as `Social Security Act (the Act)` and logs other anaphors unresolved. Before/after labels are preserved; this is in-sample hardening, not a held-out performance comparison. **Act identities are opinion-local.** For example, Illinois and federal enactments can share the name “Religious Freedom Restoration Act.” `identity_scope: opinion` prevents treating these display labels as globally interchangeable legal keys. No URLs are produced; jurisdiction/version evidence is needed for a later join. ## Independent recall estimate and uncertainty The recall gold was selected without detector output: 40 random pages from phase two, with up to two random nonempty newline text units on each page. One page had only one unit, yielding 79 reviewed units. Headings were retained rather than filtering for citation-rich prose. Reviewers enumerated the seven target-family mentions and used the full opinion to establish antecedents. Root source checks corrected pin-page identities and accepted verified parallel reporter citations. The gold contains 33 target mentions, 30 with independently identifiable referents. Weights are `(frame stratum size / selected stratum size) × (page text units / selected text units)`. The common gold-page selection factor cancels in the ratio. Intervals use 2,000 bootstrap resamples of whole reviewed pages. They are conditional on the phase-one frame and omit phase-one sampling variance, annotation error, and unseen pattern families. Sparse events make the intervals exploratory; they are not a definitive whole-corpus recall bound. | Metric | Weighted estimate | Conditional 95% page-bootstrap interval | |---|---:|---:| | detection_recall | 84.38% | 70.74–92.97% | | resolution_recall | 19.06% | 4.03–58.82% | Detection requires an overlapping span in the same family; it does not require exact span boundaries or a correct target. Resolution additionally requires the manually identified canonical key or a source-verified parallel reporter key. Unknown referents are excluded only from the resolution denominator. Full case citations/full USC citations are outside this new-family recall denominator. Remaining misses include state session laws, bare case-name references, decimal state/standards sections, nonlocal act aliases, and cases where conservative statute/intervening-authority guards decline an otherwise interpretable short reference. Some unresolved candidates are ordinary English “act,” not legislation. Negative findings are preserved; the low resolution result is not concealed by counting abstentions as successes. ## Main-sample annotation counts | Family/status | Occurrences | |---|---:| | act_anaphor:resolved | 32 | | act_anaphor:unresolved | 1,173 | | case_full:resolved | 39,166 | | case_full:unresolved | 3 | | case_id:resolved | 2,222 | | case_id:unresolved | 3,040 | | case_short:resolved | 5,788 | | case_short:unresolved | 2,845 | | cfr:resolved | 534 | | named_act:resolved | 89 | | named_act:unresolved | 3,051 | | session_law:resolved | 335 | | statute_full:resolved | 2,577 | | statute_short:resolved | 328 | | statute_short:unresolved | 11,608 | ## Statistics v2 See `../stats_v2.md` and the six companion CSV exports: decade rankings, title aggregates, top recent statutes within each title, velocity, most-cited cases, and co-citation pairs. Inputs are the uncapped graph, not capped inverse/importance lists. Counts deduplicate each case-record/section edge and roll subsections up; co-citation pairs count a case only once. - Top co-citation pair: 28 USC §1331 with 42 USC §1983 (6,017 case records); §841/§846 follows (5,331). These are discovery leads, not evidence of element equivalence. - Twombly is the most-cited case (19,435), followed by Iqbal (16,945). - Velocity compares complete 2016–2020 and 2021–2025 periods, normalized per 1,000 case records. Denominators are 73,807 versus **244**; coverage/court-mix changes dominate interpretation. Do not read the rising/falling tables as population-level legal trends. - The graph includes 406 citation keys outside the expected USC title range 1–54. They are flagged as possible normalization/OCR artifacts, not silently accepted as validated statutes. ## Files and reproduction Machine candidates are compressed JSONL: `audit_v2_candidates.jsonl.gz`. Quotes and offsets refer to concatenated HTML-decoded data nodes inside `div.opinion-body`; they are not HTML byte offsets. Verify `opinion_sha256` and reconstruct with `audit.Opinion`. Original source text is never rewritten. From repository root: ```sh uv venv /tmp/neov2-astra-venv --python 3.12 uv pip install --python /tmp/neov2-astra-venv/bin/python -r neov2/xref/requirements-audit.txt /tmp/neov2-astra-venv/bin/python neov2/xref/audit_v2.py --reuse-frame --gold 80 /tmp/neov2-astra-venv/bin/python neov2/xref/audit_v2.py --supplement-act-review /tmp/neov2-astra-venv/bin/python neov2/xref/review_metrics_r2.py python3 neov2/xref/stats_v2.py --as-of 2026-11-10 ``` Omit `--reuse-frame` to select a fresh deterministic frame from the unchanged manifest. The manual labels are committed evidence; rerunning the sampler does not manufacture new labels. The final precision file intentionally includes the supplemental cases rather than the raw packet’s unresolved filler. The authoring patch adds `citation_mentions.py` and a `NEO_RESEARCH_MENTIONS=1` opt-in IR annotation in `parse_docs.py`. It has been applied and compiled only in an isolated temporary fixture. It preserves source `text`. Existing source-SHA IR caches will need deliberate research reparse/version handling if this patch is later approved; merely setting the environment variable does not refresh cached records. ## Verification and review status 58 relevant tests pass. The patch applies, compiles, and embeds byte-identical resolver code. All 350 precision quotes and 79 gold text units/events were checked against 246 original decoded source pages. Final audit hashes match the fixed resolver version and sampled pages. Source law/cases and deployed authoring files are unchanged. **QA1 requested via `neov2/qa/astra-r2-request.md`; Lead dispatch is pending.** Internal engineering checks are not a substitute for that independent review.