Source: assist/r4/REPORT.md
# Round 4 proposer: development results and frozen held-out handoff The redesign reduces noisy output but **has not demonstrated reliable extraction**. Development improvements are modest; independent quality of the revised packets and all held-out precision/recall remain unknown. ## Development only: cases001–020 | Measure | Original | Frozen revision | |---|---:|---:| | Proposals |100|66| | Proposals on the9 empty-gold JSON cases |45|13| | Empty packets among those9 cases |0|6| | Cross-target-law proposal IDs |48|0 (enforced)| | Median passage characters |1,650|860| | Maximum passage characters |2,487|1,198| | Same-element full gold-span coverage |1/19|2/19| | Independently adjudicated precision |QA1 reported12/100|unknown| | Exact variant assignments |100|0 (abstained)| These are development diagnostics after three iterations. The count reduction is not a measured false-positive reduction: unlisted gold spans do not make all other passages incorrect, and the revised proposals have not been independently judged. Strict coverage gains only one gold item. Gold-containing-proposal rate must not be advertised as precision. Variant abstention eliminates unsupported specificity, not the need for variant extraction. QA1's report states88% false proposals and2/19 recall; its independent report is preserved. Mechanical auditing found100 judgment records but99 recognized IDs, 9 empty gold arrays rather than11 in the narrative, and1/19 full-containment recall using explicit law-normalized element IDs. These conventions/counts are recorded in `ERROR_ANALYSIS.md` and `dev_metrics.json` instead of editing labels. The original CASE-derived variant strings were retrieved, not invented text; their transfer to unrelated cases or LAW catalogues was often unsupported. The redesign conditions on the cited target law, requires discussion anchors, uses element-conditioned MiniLM queries plus lexical cues, screens selected procedural/headnote contexts, caps source windows at1,200 characters, permits abstention, and emits no exact variants. `DESIGN.md` and `config.json` specify all rules and limitations. Three development diagnostic snapshots are preserved. No per-case names, IDs, gold phrases or offsets are hardcoded into the proposer. ## Freeze and held-out generation Freeze SHA256: `8460b86e79fbea0787351c3e4332736c39534c4d03000df01401c0db35db0c1b`. `freeze.json` pins the implementation, configuration, catalogue, relevant runtime source files, model identity, sample metadata, development reports and original QA1 development annotations. The commit introducing `freeze.json` contains no held-out packets. Local pre-rebase commit `e1f1ebbd7bbdbfe1d0e4ec510de7e8cb8b4ffd35` was created before generation; the identical freeze was then rebased and verified. `generation.json` records the original local commit reference, publication note and every output packet hash. Only after freezing and committing were cases021–040 read by the new generator. All20 packets were generated with this unchanged freeze:75 proposals,2 empty packets. Their source hashes and every quotation slice, law/element prefix, variant abstention and generation timestamp passed verification. No source quotes, QA labels or original proposal packets were changed. **No held-out precision, recall, negative-discrimination or effort gain is claimed.** The held-out group contains4 drug,8 §1983 and8 Title VII cases; development had no §1983/Title VII labels and its4 drug cases were empty-gold. This statute-ordered split has substantial domain shift. The procedural opener gate can reject mixed merits cases, and excluded exception/defense/context elements can reduce recall. Anchor and application cues do not establish a legally applied holding. ## Handoff and verification Lead's dispatch request is `../REVIEW_REQUEST2.md`. QA1 must inventory full held-out sources blind to both generations of proposals, then adjudicate the new packet IDs. There are no accepted annotations in this Round4 directory. Six synthetic tests cover abstention, source fidelity, target/variant restrictions, missing/tampered freezes and repeated-freeze refusal. Ruff, Pyright and syntax checks pass. Development gold/source offsets and all75 held-out source slices were verified. The code refuses altered frozen inputs and packet overwrites. ```bash /tmp/neov2-search-venv/bin/python -m unittest discover -s neov2/tools/tests -p test_propose_r4.py -v # Inspect the already-created freeze; rerunning heldout refuses existing packets. cat neov2/assist/r4/freeze.sha256 ``` No new packages, corpus edits, site deployment or independent QA sign-off. Effort savings remain unestablished; shorter displayed passages are not measured human time savings. A fresh experiment requires a separate versioned output scope rather than modifying this frozen design after viewing held-out results.