assist/r6/REPORT.md

Source: assist/r6/REPORT.md

Exact source text

# Round 6: variant proposals and scaled recall search

**The frozen experiment ran on 30 cases. It produced 64 mechanically verified candidates, all `NEW:` labels and zero existing-label matches. Variant reuse has not been demonstrated. Independent precision, recall, novelty and reviewer agreement remain unknown.**

## Observed outputs, not legal quality

| Measure | Result |
|---|---:|
| Completed cases / model calls |30 /97|
| Retained candidates |64|
| Existing-label candidates |0|
| NEW candidates / distinct proposed labels |64 /64|
| Recall-only candidates / newly covered case–element slots |9 /5|
| Empty case packets |15|
| Incomplete executions / unknown-usage calls |0 /0|
| Mechanical quote/variant rejections |0|
| Exact duplicate candidates removed |0|

All64 candidates have exact unique source quotes and grammar-valid NEW labels. That does not make the labels necessary, useful or legally correct. No candidate or NEW label was inserted into CASE files or VARIANTS.md. Existing-label precision is undefined because none were proposed. QA must particularly check whether proposed novelty is actually a synonym of an eligible existing label or an over-specific one-off pattern.

Only2/6 controls previously annotated background/comparator received empty packets;4/6 received candidates. These are prior role annotations, not a fresh adjudication, but the nonempty controls are a warning against claiming success. Among24 fresh citing cases,13 received empty packets. Citing cases need not apply the cited statute. Do not equate any model abstention with a proven negative.

QA1 reported36/37 precision and12/23 strict recall for Round5. This round uses a different case mix, more statutes, variants, and extra recall calls. No matched-case causal comparison or accuracy improvement is claimed.

## Sample and catalogue isolation

The sample contains24 fresh citing cases (3 perlaw) plus6 independently authored negative-role controls, all outside the earlier60 assist IDs and exact source hashes. Fresh cases also exclude the authored selection snapshot. Controls are deliberately previously authored and may have appeared in earlier non-LLM research; they are not novel corpus cases. Their prior roles/provenance are withheld from model inputs and blind review views.

| Statute ID | Checklist elements | Eligible supported variants |
|---|---:|---:|
| 15usc1 | 18 | 27 |
| 18usc1962 | 30 | 36 |
| 21usc841 | 22 | 20 |
| 42usc1983 | 32 | 33 |
| 42usc2000e-2 | 37 | 60 |
| 18usc1341 | 7 | 23 |
| 18usc1343 | 8 | 20 |
| 15usc78j | 14 | 17 |

All30 test IDs and exact-source-hash aliases were removed globally before any variant support, description or example was built. Six authored CASE files were excluded. The catalogue was rebuilt from eligible merits-instance CASE blocks with factual evidence, corroborated under the correct element in VARIANTS.md. All411 supplied example quotes were mechanically verified; full authored quotes replaced the invalid idea of arbitrary stripped text slices. The final catalogue retains236 supported labels. Labels without surviving eligible support are omitted, not silently taught to the model.

The LAW checklist includes defenses/exceptions and elements with no existing variant, permitting justified NEW proposals. It supplies names/kinds/statutory text but no case-specific LAW rule quotes or glosses. No unfiltered markdown or test CASE content is supplied. Audit hashes, supporting case IDs, excluded files and source pins are retained in catalogue_audit.json/catalogue.json, while opaque provenance metadata is stripped from model prompts. This is record/source exclusion, not citation-network isolation or a guarantee against model pretraining overlap.

Nine selected IR texts are exactly60,000 characters and may be capped; the total provided text is1,257,884 characters. The runner did not truncate opinions, but it cannot infer missing official text. Both extraction and independent inventories must be interpreted relative to the provided canonical IR.

## Frozen inference and recall policy

Freeze SHA256: `bc283efcd46bbe30a220af9f911af8c0838e962e05a34000645fb4062ee062f7`.

The commit introducing freeze.json precedes real-case inference. Its hashes cover the prompt/protocol, runner/runtime reader, preparation code, complete30-case sample, filtered catalogue/audit, response schema and reviewer assignments. The same freeze passed verification after generation. No prompt or method was tuned on new outputs or reviewer labels.

Pass1 reads the full provided opinion and all target checklist elements/eligible variants, asking for ALL distinct applications and parties/counts rather than a single best result. It offers exact existing labels or `NEW:<label>` with justification. Pass2 fixes the missing-element list after mechanical Pass1 validation and scans every26k-character chunk with3k overlap. Opening/closing3k-character text is context only; evidence must be inside the current chunk. There is no presumption missing IDs have applications. This does not guarantee discovery of additional applications for IDs already covered by Pass1.

Quote checks locate unique exact substrings without repairs; existing membership is checked under the same element; NEW syntax, justification and shadow-existing checks are enforced. Exact element/variant/span/party tuples are deduplicated; conflicting labels/polarity remain explicit review issues. The source text and previous labels are unchanged.

## Measured machine cost

Provider-reported totals are3,449,366 input tokens,873,856 cached input tokens,26,001 output tokens and4,290 reasoning-output tokens (cache-write input0). Fields are reported separately and are not added twice. CLI/system context is included; monetary cost is not inferred. All97 calls reported usage and no model tool events were observed.

| Stage | Calls | Input tokens | Output tokens | Sum agent seconds |
|---|---:|---:|---:|---:|
| pass1 | 30 | 1,156,771 | 15,686 | 514.713 |
| recall | 67 | 2,292,595 | 10,315 | 708.608 |

Recall accounts for67 extra calls and2,292,595 input tokens, returning9 additional candidates across5 case–element slots. This is candidate yield, not gold recall improvement. Median per-case wall time is37.101s; batch wall time is441.002s at3 workers. Concurrent per-case/call latency sums differ from elapsed batch time. Per-case timing starts after its initial freeze check. See costs.csv, costs.json and stage_costs.json for per-case/call accounting; usage_events.jsonl retains completed-turn usage only, not private reasoning. Raw local traces remain ignored.

## Independent review handoff

REVIEW_REQUEST4.md asks Lead to dispatch QA1 and QA2. Each has15 primary cases plus3 cross-reviews:18 each,30 covered,6 shared, including2 controls. Phase A must be independent and blind to proposals, strata/prior labels and the other reviewer. Phase B assesses the application and existing/NEW fit/novelty; preserve disagreements before adjudication. Primary judgments form the non-duplicated30-case performance result; shared judgments support separate agreement estimates.

The review helper reports pending/null with no annotation files. It validates assignments and source/reviewer/proposal identities, scores each assigned18, separates existing/new acceptance, and calculates raw agreement/Cohen kappa for paired shared judgments and gold application presence, with undefined cases retained. No agreement value or accepted gold has been invented.

```bash
# Audit published outputs without new model calls:
python3 neov2/tools/summarize_r6.py
python3 neov2/tools/review_r6.py
python3 -m unittest discover -s neov2/tools/tests -p 'test_*r6.py' -v
```

For a new execution, use a fresh checkout at the pre-generation freeze commit with the read-only corpus and matching model/CLI configuration, then `python3 neov2/tools/llm_propose_r6.py run --workers 3`. The runner rejects completed-batch reruns and preserves invocation timing history for partial resumes; incomplete packets are not automatically resampled. Model outputs need not be deterministic on another execution.

Validation:15 synthetic tests,30 source fingerprints,411 catalogue examples,64 retained quotes, global ID/hash exclusions,18/18 assignments with6 shared, synthetic CLI schema preflight, Ruff/Pyright and syntax checks. No corpus edits, package additions, automatic catalogue updates or site deployment. Legal quality, useful novelty, recall gains and causal human effort savings remain unmeasured pending independent reviews.