Source: assist/r6/PROTOCOL.md
# Round 6 freeze: variant-aware LLM proposer and recall pass ## Sample and test isolation Use exactly30 unique canonical case IDs/source hashes:24 fresh citing cases (three per law) plus6 controls previously annotated comparator/background with no applied block. The8 laws are the five pilots plus18usc1341,18usc1343,15usc78j. All30 exclude the previous assist001–040 and r5-001–020 IDs and exact source hashes. The24 fresh cases exclude authored cases in the selection snapshot. The6 controls are deliberately previously authored, not novel corpus cases; existing role annotations supply the negative stratum, not new QA gold. Controls may have appeared in earlier non-LLM research. QA must reassess them. No expected stratum, prior role label or test CASE annotation reaches the agent. Source lengths are1,200–100,000 characters; no text is truncated by the runner. Canonical IR may itself be capped at60,000 characters. The manifest flags those records: extraction and independent inventory are limited to the provided text, not claimed exhaustive coverage of an original official opinion. ## Catalogue construction and global exclusion Read each selected statute's VARIANTS.md AND CASE files, with exact source-pin and quote verification. Read the LAW checklist through the existing parser, including defenses/exceptions and elements without known variants. LAW fields sent to the agent are element ID/name/kind and statutory quotes; no case-rule quotes or LAW glosses derived from test cases are supplied. Before any counts, examples or descriptions are assembled, remove every selected test case from ALL statute CASE collections, plus any exact-source-hash aliases. This is global case exclusion, stronger than omitting only the current pair. Only surviving CASE merits-instance blocks with factual evidence support an existing label. Corroborate each label under its own element in VARIANTS.md. Labels solely supported by excluded/unsupported records vanish. Preserve each variant's supporting case provenance and at most2 complete, verified authored fact quotes; never take arbitrary text slices or normalize quotes. Full audit hashes and excluded files are in catalogue_audit.json. The frozen eligible catalogue is narrower than raw markdown, which also lists non-instances and not-reached blocks. NEW therefore means a proposed addition relative to this filtered catalogue, not globally novel law. The three wave-2 catalogues retain23,20,17 supported variants respectively. No unfiltered markdown or test CASE content is inserted into a model prompt. Opaque provenance hashes and paths are also omitted from prompts; they remain in the audit artifact. No citation-network isolation or immunity from model pretraining is claimed. ## Frozen inference policy Freeze/commit this protocol, PROMPT.txt, schema, runner, preparation code, required runtime sources, all30-case inputs, filtered catalogue/audit and reviewer assignment before any real-case LLM call. Synthetic tests only before freeze. Use the installed configured model/reasoning settings (gpt-6-astra / medium at preparation), CLI version pinned, no model tools, fresh ephemeral read-only sessions. Up to3 case workers; calls inside a case are sequential. Pass1: give the complete provided canonical opinion and the full target-law checklist/eligible variant catalogue. Ask for ALL distinct applied elements, parties/counts/theories, not a single best result. Return element, role=applied, polarity, party/count, unique contiguous verbatim quote, brief application reason, and existing label or NEW:<generalizable-label> with a variant justification. Adverse findings can qualify. An assumed/uncontested element used only to decide another matter does not. No proposal quota; abstention remains valid. Pass2: identify element IDs without any mechanically retained Pass1 candidate. If none remain, skip recall. Otherwise independently scan every26,000-character chunk with3,000-character overlap, with the same fixed missing-ID set for every chunk. Include the first/last3,000 characters as context only. Quotes must lie inside the current evidence chunk. The overlap exceeds the2,500-character quote cap, so each eligible short span can be wholly present in a chunk. The opening and closing context cannot guarantee controlling-opinion attribution mid-dissent; uncertainty requires abstention. A missing-ID list is not evidence that something was missed. All applications within those missing elements are requested. This does not guarantee recovery of extra applications of already-covered IDs. No external text or fresh labels, no prompt revisions after outputs, no semantic repair retries. A900-second per-call timeout and64-candidate transport ceiling are safeguards, not extraction targets. Log failures, truncation/cap signals and unknown usage rather than hiding them. A failed Pass1 prevents recall; failed recall calls leave an explicitly incomplete packet. The batch records uncaught case failures without discarding completed results. Resume skips only packets with matching frozen/source pins, without resampling. ## Quote and variant checks Require exact unique substrings of the full source, valid declared element IDs, applied role and exact existing-variant membership under that same element. NEW labels must use the existing lowercase slash-label grammar, have at most180 characters after NEW:, differ from existing exact labels, and include a material justification. No source edits, whitespace/punctuation repair or first-occurrence selection for ambiguous quotes. Examples from the catalogue are not current case evidence. Quote/label validity is not legal or novelty acceptance. Deduplicate only exact element/variant/span/party tuples across passes. Preserve supporting call IDs, report differing labels on the same element/span and any polarity disagreement. Keep the first exact-duplicate representation; do not choose a supposedly better answer after inspecting outputs. All outputs remain UNREVIEWED. No NEW label is automatically added to CASE files or VARIANTS.md. ## Cost and independent evaluation Capture provider usage fields per call and sum each field per case, including all recall calls and invalid responses. Cached/reasoning fields are details, not extra tokens to add again. Record wall time per call/case and batch duration; concurrent latency sums are not elapsed batch time. No inferred dollar cost or human effort saving. CLI-internal transport retry time, if any, is included. QA1 has15 primary cases, QA2 the other15. Three cases from each half are shared: 18 cases per reviewer,30 covered,6 shared (including2 controls). Each writes a blind inventory before seeing outputs or the other reviewer. Both assess element application and existing/NEW variant fit/novelty; preserve disagreements before adjudication. Report agreement on the shared cases with explicit denominators, not agreement invented for unreviewed cases. NEW labels may be synonymous or unsupported and require independent evaluation. Precision/recall/variant quality and causal effort savings remain unknown until the reviews land.