Source: assist/REPORT.md
# Round 3: assisted extraction — independent review pending **Precision, recall and annotator effort savings are not established.** Forty fresh cases (eight per pilot statute) have 200 machine proposals, but zero independent annotation files have arrived. All proposals remain `unreviewed`. The proposer has created no accepted ground truth. ## Delivered - `sample.json`: frozen case selection, canonical source pointers and hashes. - `proposals/*.json`: five element/variant/passage proposals per case, with exact source offsets and quotes. Model ranks/scores and neighbor annotations are hidden; opaque proposal IDs and shuffled order avoid encoding rank in IDs. - `ANNOTATION_PROTOCOL.md`: blind full-source inventory first, then separate element/passage and exact-variant judgments, with reviewer identity and times. - `tools/score_assist.py`: rejects self-review, source mismatches, malformed judgments and invalid spans; calculates only independently supported metrics. - `metrics.json`: explicit null precision/recall and zero reviewed/timed cases. The matcher uses the unchanged frozen Round 2 annotated facts database; these 40 cases are absent from it and from authored CASE files at sampling. A final check against 144 currently authored shared-tree case IDs also found no overlap. All 200 quotation slices were checked verbatim against canonical IR text. No proposals were inserted into the annotation database or accepted CASE files. ## Independent dispatch blocker The task explicitly requires Lead to dispatch another annotator seat. Three `lmctl prompt ... Lead` attempts returned `receiver_busy` because Lead is held by an active terminal. No delivery/queue or reviewer assignment is claimed. A direct asynchronous user/Lead request was also submitted. Shared packets and the request are available at `/home/mma/repos/neochart/neov2/assist/`. See `REVIEW_REQUEST.md`. Astra3 will not substitute its own labels for this step. Independent reviewers should write `annotations/<sample_id>.json` per the protocol. Then run `python3 neov2/tools/score_assist.py`. Precision counts judged proposals only. Recall requires an exhaustive inventory produced before seeing proposals and reports reviewed-case coverage. Accept/reject-only review cannot supply a missed-evidence denominator. Variant correctness is scored separately from element/passage relevance; neither establishes a holding or polarity. ## Effort: descriptive display reduction only The 40 inputs contain 1,469,713 Unicode characters. Deduplicating repeated proposed spans exposes 286,442 characters (19.5% of the source text). Generation took 121.4 seconds in this run. These are interface/computation measurements, **not recall or annotator time savings**: useful evidence may lie outside those passages, and proposed variants may be wrong. No inventory or candidate-review times were measured because there is no independent reviewer yet. Even when supplied, Phase B follows Phase A and is confounded by prior reading. A causal effort-saving claim requires an additional randomized assisted-versus-unassisted study with independently fixed gold. It is honest to report this as an unfinished validation dependency, not a positive result or zero-precision result. ## Reproduce proposal generation ```bash /tmp/neov2-search-venv/bin/python neov2/tools/assist_extract.py python3 neov2/tools/score_assist.py ``` The generator resumes existing packets only when source/model/snapshot pins agree; do not overwrite independent annotation files. Rebuilding the sample against evolving annotations can change membership. The committed sample and packets are the concrete review batch. No source law/case text was altered.