Source: xref/out/r3/REBUILD_PLAN.md
# Round 4 update — fragment-safe href checks; full rebuild still blocked
This update supersedes the Round 3 sample counts below. `target_href()` now checks the page path before `#fragment`, then preserves the fragment exactly in the emitted href. Plain links and missing-page rejection remain unchanged; fragment-only current-page links are checked against the current file. Unit tests first reproduced the defect, then passed with the fix.
## Exact sample reruns
Both historical manifests were pinned by case ID, not reselected. The QA1 inventory is taken verbatim from `neov2/qa/xref-r3-QA1.md`; it is disjoint from Astra's 80 cases and includes the same 18 SCOTUS cases.
| Current stored pages vs corrected renderer | Pages | Changed | Added anchors | Removed anchors | Text / non-anchor changes | Gate |
|---|---:|---:|---:|---:|---:|---|
| Exact Astra Round 3 set | 80 | 10 | 23 | 0 | 0 / 0 | PASS |
| Exact QA1 Round 3 set | 80 | 10 | 23 | 0 | 0 / 0 | PASS |
| Additional QA1 advisory `/f3d/559/0274-01` | 1 | 1 | 1 | 1 | 0 / 0 | **BLOCKED** |
The two requested 80-case runs therefore have 46 added anchors across 20 pages. Research annotations still change zero card inputs or HTML bytes. The added fragments increase Astra's prior total from4 to23 and QA1's prior total from1 to23. Exact hashes, manifests, diffs and gate results are under `../r4/`.
### Why the advisory case remains blocked
The previous implementation dropped the link to `18 U.S.C. § 924(c)` because it checked `index.html#C` as a filename. The corrected implementation keeps a link, but its href is now the citation index's `index.html#C`, whereas the stored page links to `index.html` without the fragment. This is a **retarget**, counted as one removed/one added anchor by the strict gate. The gate must continue rejecting it; it is not an allowed pure addition.
A separate read-only DOM inspection found that the current section924 target page has no `id="C"` or `<a name="C">`. The fragment-preservation fix does not validate or create DOM destinations. `fragment_target_observation.json` records the target hash and finding. Resolving the stale index/target-anchor relationship or approving a retarget requires a separate scoped decision; neither the corpus nor citation index was edited in this round.
**Do not run or publish a full candidate rebuild on the strength of the two passing samples.** The known advisory failure must be resolved or explicitly handled through the operator's approval process, without weakening the general preservation gate. No full re-emit or production deployment has been performed.
## QA1 notes addressed
- Fragment/path bug: fixed in the tracked authoring renderer with regressions for `#C`, encoded fragments, no fragment, same-page fragments, empty targets, and missing files (including a misleading filename containing `#C`).
- Gate robustness: the operator command below calls the same tested `verify_candidate_tree()` used against all three real scratch trees. It rejects missing/unexpected pages, absent source pages, text/whitespace corruption, removed or retargeted anchors, and changes outside anchors. A dedicated test covers base-href → `#C` retarget rejection.
- Annotation default/caching: unchanged. `NEO_RESEARCH_MENTIONS` remains default-off; caches are not backfilled and renderers do not consume annotations.
- Resource guidance: retained. Updated 80-case runs took about12.9s each including cached lookup load, with4.46/5.29s annotation work and0.68/0.56s two-pass rendering. This remains consistent with the conservative100GiB /4–12h planning allowance; no full-corpus impact count or timing guarantee is asserted.
- Optional test dependencies: full Python authoring/xref suites were rerun; five corpus-fixture checks were rerun read-only. Only unrelated CairoSVG/Pillow image tests remain unrun. No dependencies were added.
Reproduce the bounded runs with new empty scratch directories:
```sh
cd /home/mma/repos/neochart
/tmp/neov2-astra-venv/bin/python neov2/xref/scratch_reemit.py --sample 80 \
--sample-manifest neov2/xref/out/r4/astra_80_manifest.json --scratch /tmp/neov2-r4-astra-repeat
/tmp/neov2-astra-venv/bin/python neov2/xref/scratch_reemit.py --sample 80 \
--sample-manifest neov2/xref/out/r4/qa1_80_manifest.json --scratch /tmp/neov2-r4-qa1-repeat
# Expected exit2: preserve the advisory retarget evidence, do not waive it.
/tmp/neov2-astra-venv/bin/python neov2/xref/scratch_reemit.py --sample 1 \
--sample-manifest neov2/xref/out/r4/reported_fragment_case_manifest.json --scratch /tmp/neov2-r4-advisory-repeat
```
---
# Historical Round 3 baseline and operator command (subject to Round 4 block)
## What was landed
QA1 passed the Round 2 resolver (`neov2/qa/xref-r2-QA1.md`). The authoring integration is now in the existing tracked `parse_docs.py` and `eyecite_extract.py` files. With `NEO_RESEARCH_MENTIONS=1`, freshly parsed case IR receives `research_mentions`; default parsing remains unchanged. The loader reuses the byte-identical reviewed resolver at `neov2/xref/resolve_mentions.py`, instead of introducing the patch's new untracked `authoring/citation_mentions.py`. This honors the tracked-authoring-files constraint and avoids duplicating the reviewed rules. A complete repository checkout is required for the opt-in path.
**The annotation field is not a link-generation feature.** `emit_pages.build_card()` and the renderer do not consume it. Adding only this field changes zero rendered pages or links when all other inputs are held fixed. Existing source-SHA parse caches do not invalidate merely because the opt-in environment variable changes; cached records are not silently backfilled. No cache was deleted, no full reparse/re-emit was run, and no production/S3/CloudFront operation was performed.
## Measured scratch re-emit and preservation proof
A bounded run used the actual `emit_pages.page_view/build_card` and `build_site.render_section_page` leaf-rendering functions, existing IR/refs, the global cached link maps, and the real hosted-section inventory. It wrote only under `/tmp/neov2-r3-emit-final`.
Selection: the forty fresh QA1 cases, matched by citation against the corpus manifest, plus forty seeded Round 2 cases (seed 20261113). This is a validation sample, not a representative impact estimator. Missing sample IR would fail the run rather than silently reduce coverage.
| Comparison | Pages changed | Anchors added | Anchors removed | Text differences |
|---|---:|---:|---:|---:|
| Cached IR rendered before vs after adding annotations | 0 / 80 | 0 | 0 | 0 |
| Current stored pages vs scratch annotated rendering | 2 / 80 | 4 | 0 | 0 |
The four current-page additions are existing U.S.C.A. recognition improvements from Round 1, **not effects of the new annotation field**:
| Current page | Additions |
|---|---:|
| `catalog/cases/f2d/volume_298/0017_01/index.html` — Division No. 14, ORT v. Leighty | 3 |
| `catalog/cases/f-supp/volume_171/0454_01/index.html` | 1 |
Proof is stronger than whitespace-normalized text comparison:
1. Original IR dictionaries and their exact `text` strings remain unchanged; every annotation quote equals the original string slice at its offsets.
2. Renderer card inputs before/after annotations are equal, and both HTML outputs are byte-identical on every sampled case.
3. Compared with current pages, all decoded HTML data (including whitespace) is equal.
4. Removing only anchor tags leaves byte-identical HTML; all pre-existing anchor spans, attributes, and destinations remain present. Thus the only observed differences are additions of `<a>`/`</a>` wrappers.
5. Each sampled current file was re-read and hash-checked after rendering. Code hashes, input file metadata, page hashes, counts, and timings are in `reemit_evidence.json`; the complete two-page diff is `sample_anchor_additions.diff`.
This proves the gate for these 80 cases. It does **not** establish full-corpus impact counts or historical-version correctness of linked statutes. A full reparse, refreshed resolver graph, changed template, or rebuilt global lookup map is a different input set and must pass its own full diff gate. No statute pages or navigation pages were included in this case-leaf sample.
Reproduce the bounded proof, with a new empty scratch directory:
```sh
cd /home/mma/repos/neochart
/tmp/neov2-astra-venv/bin/python neov2/xref/scratch_reemit.py \
--docroot /home/mma/repos/neochart/neochartdocroot/docroot \
--scratch /tmp/neov2-r3-operator-sample --sample 80
```
The tool refuses more than 200 cases, output outside a dedicated `/tmp` directory, or nonempty scratch output. It uses the real hosted-section inventory explicitly, so an isolated checkout cannot silently change normalization behavior.
## Time, memory, and disk planning
Measured on the 80-case run:
- Total: **17.94 seconds**, including 5.26s manifest selection, 0.63s reading 480.70 MB of selected IR/refs shards, and 6.87s loading lookups/configuration.
- Annotation extraction: **4.43s**. Two leaf-rendering passes: **0.55s**.
- Peak process RSS: **2.44 GB**. This is a bounded-process measurement, not the full emitter's memory ceiling.
- One HTML output set: **1,275,698 bytes**. Compact IR annotation growth: **520,047 bytes**.
- Read-only corpus IR plus refs currently totals **52,884,354,344 bytes** across 25,452 shard/sidecar files.
Simple case-count scaling to 2,023,221 records gives approximately **30.05 GiB** for one candidate HTML set and **1.94 hours** of leaf-rendering CPU work. Full-emitter global scans, JSON decoding, page creation, filesystem overhead, graph/map construction, and shared-host load are additional: budget **4–12 hours as a planning allowance, not a measured completion guarantee**. Reserve **at least 100 GiB of free output space** for candidate pages, filesystem overhead, indexes, manifests, and headroom. A full process can need substantially more memory than the sample because it retains global maps, page stubs, and hashes.
Backfilling annotations across all cases would separately add roughly **12.25 GiB** to IR and take **31.14 hours for the annotation extraction alone** at this sample rate. A full reparse also repeats existing extraction and source reads; its total was not benchmarked. There is no page-rendering benefit from that backfill today, so **do not force a full reparse for this annotation-only integration**.
These extrapolations use a small deliberately selected case sample and can be wrong by large factors. They do not project how many pages or links will change. `/tmp` is a 16.4 GB tmpfs on this host and had only about 3.8 GB free during verification; it is suitable for the bounded sample, **not a full run**. The operator-only full candidate command below uses a fresh directory on `/srv/lmdata`, outside every corpus path.
## Exact operator-only full case re-emit command
**Not executed. Run only after the operator approves the full scratch run.** Use a checkout containing this Round 3 change and the existing pinned authoring environment. This command reads existing cached case IR; it neither reparses sources nor backfills annotations. It renders all case leaves into a new candidate directory, excluding navigation and statute pages. It does not publish anything.
```sh
set -euo pipefail
R3_REPO=/home/mma/repos/neochart
R3_PY=/tmp/neov2-astra-venv/bin/python
R3_SOURCE=/home/mma/repos/neochart/neochartdocroot/docroot
export R3_SOURCE R3_REPO
"$R3_PY" - <<'PY'
import shutil
assert shutil.disk_usage('/srv/lmdata').free >= 100 * 1024**3, 'Need at least 100 GiB free for the candidate run'
PY
R3_SCRATCH=$(mktemp -d /srv/lmdata/neov2-r3-approved-XXXXXX)
export R3_SCRATCH
export HOSTED_USC_SECTIONS="$R3_SOURCE/hosted_usc_sections.json"
cd "$R3_REPO/neochartdocroot"
"$R3_PY" authoring/emit_pages.py \
--docroot "$R3_SOURCE" \
--ir "$R3_SOURCE/ir/caselaw" \
--out "$R3_SCRATCH/pages" \
--books "$R3_REPO/neochartdocroot/authoring/books.json" \
--full --no-nav
PYTHONPATH="$R3_REPO/neov2/xref:$R3_REPO/neochartdocroot/authoring" "$R3_PY" - <<'PY'
import json, os
from pathlib import Path
from scratch_reemit import verify_candidate_tree
source = Path(os.environ['R3_SOURCE']).resolve()
candidate = Path(os.environ['R3_SCRATCH']).resolve() / 'pages'
assert source not in candidate.parents and source != candidate
# Verify the complete intended cached-IR address set, not just files that
# happened to appear. Current orphan pages outside this IR set are untouched.
expected = set()
for shard in (source / 'ir/caselaw').glob('*.jsonl'):
if shard.name.endswith('.refs.jsonl'):
continue
with shard.open() as handle:
for line in handle:
record = json.loads(line)
if record.get('kind') == 'case':
expected.add(record['address'])
try:
counts = verify_candidate_tree(source, candidate, expected)
except (ValueError, OSError) as error:
raise SystemExit(f'BLOCKED: {error}')
(candidate.parent / 'full_diff_gate.json').write_text(json.dumps(counts, indent=2) + '\n')
print(counts)
print('Candidate only. No production publication is authorized by this check.')
PY
```
Do not substitute `build_v2.sh`, omit `--out`, or point output/IR scratch paths at the corpus: those would change the operation's scope. The CLI rebuilds global maps from the full case IR while the sample uses the existing cached maps; the full diff gate is therefore essential. A passing gate still requires QA2 verification and separate operator publication approval. No S3, CloudFront, deployment, or copy-to-production command is included.
## Tests, court-level statistics, and review handoff
- Main suite: **194 passed, 7 skipped** across all existing Python authoring tests and xref tests, including the prior 58 checks and new regressions.
- Five fixture-dependent tests were then run against the read-only original corpus: **5 passed**. Thus **199 distinct tests passed**; the only remaining skips are the unrelated favicon/social-image tests requiring optional CairoSVG/Pillow packages. No dependencies were added for them.
- Fresh-page regressions use exact Pinson §552 and Chester CFR/Public Law/Stat. excerpts. Jacobs' plural CFR list is tested as an unsupported form that must not invent a resolved section. Synthetic parser fixtures separately verify opt-in/default behavior and verbatim source text. The fresh source check corrected an earlier test comment that had attributed Chester's session-law cites to Jacobs.
- `test_scratch_reemit.py` rejects text/whitespace changes, removed or retargeted anchors, and non-anchor markup changes.
- `court_levels.{json,csv,md}` adds the 2017–2026 breakdown from **22,243** citation-bearing case records and actual rendered court labels. It took approximately **12.1s** for metadata counting (graph load/hash extra). **570** records remain unknown, explicitly retained; no reporter-based guessing. New York trial/appellate naming, DC's highest court, Texas criminal appeals, and federal Supreme labels are handled separately.
- QA1's Round 2 verdict is PASS. **Round 3 QA2 verification is requested, not claimed complete.**
Reproduce only the read-only court breakdown:
```sh
python3 neov2/xref/stats_v2.py --court-levels --as-of 2026-11-10 \
--docroot /home/mma/repos/neochart/neochartdocroot/docroot \
--out neov2/xref/out/r3
```