assist/r5/REPORT.md

Source: assist/r5/REPORT.md
Exact source text

# Round 5: frozen whole-opinion LLM-agent proposer

**20 new cases processed; 37 mechanically verified candidates; 7 model
abstentions. Independent semantic precision, recall and effort savings remain
unknown pending QA1.** No accuracy improvement over embeddings is claimed.

## What changed and why

The one-page [POSTMORTEM.md](POSTMORTEM.md) explains the held-out embedding
failure:3/75 precision and0/20 strict recall. Similarity favored doctrinal
boilerplate; window boundaries did not represent court/party/element relations;
application, comparator facts and announced rules shared vocabulary.

This experiment instead supplied a fresh LLM-agent invocation with each complete
provided canonical IR text and the target law's checklist. It used the existing
configured **gpt-6-astra / medium**, codex-cli0.166.0, with no model tools or external
research, one ephemeral session percase, and no retriever. The agent could return
up to6 candidates or abstain. It gave an element, applied role, polarity, exact
quote and concise reason. No exact variants were requested.

The selected20 cases contain4 per pilot statute. All authored/prior case IDs at
selection (404) and the earlier40 assist cases were excluded; prior calibration
IDs were excluded too. Exact source hashes are unique and do not overlap earlier
assist/calibration sources. Source lengths are1,200–50,000 characters; the runner
never truncated them. Provided IR completeness relative to an original official
opinion is not guaranteed. One mechanically selected negative-candidate stratum
perlaw supplies an intended mix, but **true negative status is unlabelled**.
These strata were hidden from the proposer and are hidden from QA1's blind view.

## Freeze and execution evidence

Freeze: `6dd6b364ae02a550f8143354ea44b481b43d7ce3cf41b5ba347aa446bafd69c1`.

`PROMPT.txt`, `PROTOCOL.md`, schema, source reader/runner, sampler, checklists,
20-case manifest and post-mortem were hashed and committed before generation.
The commit introducing freeze.json contains no real-case packets. A synthetic
preflight tested the CLI/schema only; its tokens are excluded from this batch.
The runner rechecked the frozen model configuration, CLI version and file hashes
for each call. No prompt/protocol change, quote repair, semantic relabeling, or
resampling followed outputs. A later test-fixture correction only added the
sampler path to its dummy freeze tree; the actual frozen code was untouched.

All20 calls completed with token usage available and **no model tool calls**.
All37 raw candidates were unique exact substrings of their canonical sources;
zero required rejection or repair. Offsets were derived mechanically, not
model-guessed. All retained candidates still say `unreviewed`. Execution success
and quote fidelity do not prove an applied legal holding or correct polarity.

Raw model responses and event logs are retained in ignored `raw/`, fingerprinted
in the packets. `usage_events.jsonl` contains only provider completed-turn usage
records, no private reasoning text. `generation.json` hashes the final packets.
`review_packets/` removes model reasons/cost/role predictions for independent
candidate adjudication. The audit tool `summarize_r5.py` adds no legal labels.

## Tokens and wall time

Provider-reported fields, summed independently: input **667,191**;
cached input **96,000**; output **7,434**;
reasoning output **995**; cache-write input0. Do not
add detailed/cache fields again to a total. These counts include CLI system
context, not just opinion/checklist tokens. The synthetic preflight showed about
22k input-token overhead. Monetary cost is not inferred from unknown billing.

Batch wall time **90.94s** at concurrency3.
Median agent-call wall time **11.668s**; sum of per-case
agent latency **271.150s**. Per-case sums overlap in time
and are not batch duration. Latency includes CLI startup/communication; these
are machine costs, not human review times. Per-case total timing starts after the freeze precheck; batch timing includes scheduling/prechecks. `costs.json` and `costs.csv` retain
per-case source/prompt character counts and timing fields.

| Case | Input tokens | Cached input | Output tokens | Reasoning output | Agent seconds | Candidates |
|---|---:|---:|---:|---:|---:|---:|
| r5-001 | 38,102 | 0 | 83 | 0 | 7.924 | 0 |
| r5-002 | 32,566 | 14,848 | 1,036 | 206 | 28.734 | 5 |
| r5-003 | 35,576 | 0 | 489 | 98 | 17.208 | 2 |
| r5-004 | 32,321 | 12,288 | 853 | 127 | 22.820 | 4 |
| r5-005 | 38,197 | 0 | 86 | 0 | 7.637 | 0 |
| r5-006 | 34,678 | 0 | 273 | 49 | 11.243 | 1 |
| r5-007 | 35,518 | 0 | 310 | 41 | 10.893 | 2 |
| r5-008 | 37,489 | 0 | 816 | 119 | 21.048 | 4 |
| r5-009 | 34,947 | 0 | 265 | 60 | 11.595 | 1 |
| r5-010 | 31,889 | 0 | 505 | 68 | 14.749 | 2 |
| r5-011 | 33,754 | 0 | 529 | 0 | 15.299 | 5 |
| r5-012 | 29,661 | 0 | 635 | 32 | 17.261 | 4 |
| r5-013 | 31,554 | 14,848 | 72 | 0 | 6.881 | 0 |
| r5-014 | 30,000 | 0 | 94 | 0 | 8.134 | 0 |
| r5-015 | 34,403 | 0 | 75 | 0 | 6.635 | 0 |
| r5-016 | 31,387 | 14,592 | 513 | 37 | 15.052 | 3 |
| r5-017 | 33,175 | 12,288 | 112 | 27 | 9.241 | 0 |
| r5-018 | 30,734 | 12,288 | 301 | 131 | 20.573 | 1 |
| r5-019 | 33,007 | 14,848 | 325 | 0 | 11.740 | 3 |
| r5-020 | 28,233 | 0 | 62 | 0 | 6.483 | 0 |

## Review and limits

[REVIEW_REQUEST3.md](REVIEW_REQUEST3.md) asks Lead to dispatch QA1. Phase A uses
`review_sample.json` without output or sampling-stratum exposure. Phase B uses
`review_packets/`. There are no developer-authored accepted annotations.
The complete opinions were provided, but prompt compliance does not certify
that every relevant passage was understood. Public-case pretraining overlap is
unknown. Model-reported abstentions are not confirmed negatives. This is a fresh
batch, not a matched-case causal comparison to R4. No new QA1 labels have been
used for tuning, and no human effort-saving claim is supported.

Seven synthetic tests exercise source/Unicode fidelity, rejection without repair,
role/element/cap constraints, usage accounting, tool quarantine and freeze guards.
Ruff, Pyright, syntax checks and all20 source hashes /37 retained spans pass.
No corpus or prior labels changed, no packages were added and no site was deployed.

```bash
# Audit published outputs; does not invoke an LLM:
python3 neov2/tools/summarize_r5.py
python3 -m unittest discover -s neov2/tools/tests -p test_llm_propose_r5.py -v
```

For a new execution, use a fresh worktree at the commit introducing the freeze,
with the read-only corpus accessible and matching CLI/model settings, then run
`python3 neov2/tools/llm_propose_r5.py run --workers 3`. Do not rerun merely to audit:
resume skips existing pinned packets and records timing for that invocation,
not the original batch. Model outputs/timing may vary on repeated calls; the
published packets and usage are the measured experiment, not a determinism claim.