Source: assist/ANNOTATION_PROTOCOL.md
# Independent review request — Round 3
These 40 cases have no authored CASE annotation in the selection snapshot. Five
machine proposals per case are **unreviewed**, not gold. Do not import them as
facts/instances or treat their element/variant assignments as correct.
Lead must dispatch an independent annotator seat. The proposer (Astra3) must not
create accepted labels. Reviewers should record their seat identity, timestamp,
source hash and annotation time. Model scores, rank order and neighbor labels are
withheld from proposal packets; packet order is deterministically shuffled.
## Phase A: blind evidence inventory (required for recall)
Before opening `proposals/*.json`, read the full canonical source from each case's
pointer in `sample.json`. Use the existing LAW catalogue as a vocabulary, but do
not read matcher scores or predictions. Record each supported element and the
smallest sufficient verbatim factual passage, exact Unicode `[start,end)` offsets,
polarity, and variant if the existing variant actually fits. New/unrepresented
variants must remain explicit; do not force a nearest-neighbor variant.
Distinguish facts from rules, holdings, comparator discussion and mere citation.
Mark exhausted review versus partial reading. Only an exhaustive blind inventory
supports a recall denominator. Include negative cases or no applicable elements
rather than fabricating evidence to fill a quota.
Write `assist/annotations/<sample_id>.json`:
```
{
"sample_id": "assist-001", "reviewer": "independent seat name",
"source_sha256": "...", "blind_before_proposals": true,
"review_complete": true, "inventory_seconds": 123,
"gold": [{"law_id": "...", "element_id": "...", "variant_id": null,
"start": 123, "end": 456, "quote": "verbatim source passage"}]
}
```
Use actual IDs from `sample.json`. A reviewer may decline a case or mark it partial;
that is a coverage gap, not an incorrect model prediction.
## Phase B: candidate adjudication
After saving Phase A, inspect the five shuffled proposals. For each, return
`accept` or `reject` for (a) element/passage support and (b) exact existing variant
fit. Record rationale, review_seconds, and any needed narrower evidence slice.
Accepting a passage does not mean accepting its proposed variant. Append:
```
"candidate_review_seconds": 45,
"judgments": [{"proposal_id": "assist-001-p1", "element_accept": false,
"variant_accept": false, "reason": "..."}]
```
## Metrics and effort limits
Precision = accepted element/passage proposals / independently judged proposals;
variant precision uses separate variant judgments. Recall = distinct blind gold
items recovered by an accepted same-element proposal whose source slice contains
the gold span / all exhaustive blind gold items. Report conditional coverage,
unreviewed cases and unsupported/new variants separately. No self-labels or
pending proposals enter metrics.
Record observed annotation/review time and number of passages inspected. Phase B
follows Phase A and therefore benefits from prior familiarity: its time cannot
establish causal effort savings. An unbiased savings estimate needs a separate,
randomized assisted-versus-unassisted timing study (with independently fixed gold).
Without that trial, report effort savings as **not established**, even if candidate
review is faster. No invented times, inferred acceptance, or claimed recall from
accept/reject-only reviews.