Source: assist/r5/PROTOCOL.md
# Round 5 frozen protocol: whole-opinion LLM-agent proposals ## Selection and blinding Use the committed 20-case `sample.json`, four cases per original pilot statute. Exclude all previously authored CASE IDs, assist001–040, and earlier calibration IDs found in the selection snapshot. Eligible canonical IR text has1,200–50,000 characters; no source is truncated to meet this bound. IR completeness relative to original downloaded opinions is not guaranteed by length alone. Within each law, one case is selected mechanically for sparse target citations and procedural/comparator cues, and three from the other eligible candidates. These five negative-candidate strata are **not adjudicated negative labels**. The task's intended mixture is prospective; true negative prevalence remains for independent QA1 to determine. Strata and cue counts are not sent to the proposer and must not be shown to QA1 before blind inventory. ## Agent call Freeze `PROMPT.txt`, response.schema.json, the runner, source reader, LAW checklists, sample, protocol and post-mortem before any of the20 LLM calls. Commit the freeze before running. Synthetic input may test infrastructure only. Never tune this protocol or prompt after reading new-case outputs or QA1 labels. For each case, run one new ephemeral `codex exec` session in an empty temporary working directory, with read-only sandbox, JSON output schema and no permitted tool use in the prompt. It reads the complete provided canonical IR text in one prompt plus all declared target-law checklist elements (names, kinds, statutory quotes and notes). No retriever, chunks, old proposals, expected stratum or QA labels enter the prompt. No native model override: verify the frozen configured model/reasoning settings and CLI version before each call. Current configuration at preparation is gpt-6-astra / medium; auth settings are never copied. The agent may propose up to SIX candidates, or abstain. A candidate must identify an actual application to current-case facts, including adverse/negative findings. Output element, role=applied, polarity, one contiguous verbatim quote and brief reason. No exact variants. Comparator facts, pure rules, syllabus/headnotes, dissent-only reasoning and historical/off-target procedural matters do not qualify. The prompt is the authoritative instruction, not any case text. ## Mechanical verification, not legal acceptance Only unique exact substrings of the canonical IR text are retained. Derive their Unicode code-point `[start,end)` offsets mechanically; do not trust model counts. Reject nonmatching, ambiguous, empty/overlong, duplicate or undeclared-element candidates. Never normalize punctuation/whitespace, repair quotes, pick the first of repeated strings, or fabricate offsets. Reject a non-applied case role with application candidates. Every retained candidate remains `unreviewed`. The initial raw model response and JSONL event trace are retained in ignored `raw/` storage and fingerprinted. Public packets include mechanical rejection reasons/hashes, not invalid quote text presented as source evidence. No semantic relabeling or follow-up quote-repair call is allowed. A failed call is reported as failure, not a negative case. The900-second timeout is a fixed execution cap. No runner retries or semantic repair; a resume skips pinned existing packets rather than resampling. CLI-internal transport retries, if any, are included in wall time; only reported tokens are counted. No model tools are allowed: any non-message/reasoning/warning event item quarantines the result. Read-only sandbox does not replace event auditing. ## Cost accounting Record `turn.completed.usage` exactly as reported by the CLI: input, cached input, output and reasoning fields when available. Sum each field across completed turns within that case; do not estimate missing tokens or add subfields again. Token counts include CLI system/context overhead, not just the supplied opinion. A synthetic preflight exposed about22k input-token overhead; it is not a measured case and is excluded from the20-case totals. Provider dollar cost is not inferred. Measure model-call and total per-case wall time with a monotonic clock, including CLI startup/communication, plus batch wall time. Use up to3 concurrent calls; per-case latency sums are not elapsed batch time. Record failures/unknown usage explicitly. These are machine costs, never human annotation times or savings. ## Review and reporting QA1 must read each full source blind to proposals, expected strata and model reasoning, save a complete evidence inventory, then adjudicate the new proposal IDs. No proposer output is ground truth. Keep exact-span recall separate from same-element alternate-evidence recovery. Report coverage and failed cases; null variants have zero coverage and undefined precision. A sequential review cannot establish causal human effort savings. Independent review is requested in REVIEW_REQUEST3.md after the frozen outputs are generated. No accuracy gains will be claimed before QA1's results, and no model tuning follows those labels.