genshin-assistant/docs/ocr-eval.md

# OCR Eval Harness

Field-level accuracy measurement for the artifact OCR parser. This is the gate
every OCR, crop, layout, or parser change runs against (see ADR-007).
It is necessary but not sufficient for live scanner proof: scan speed and
review/miss rates are measured by `npm run scan:iterate:validated` for short
iteration and `npm run scan:goal:validated` for the final 100-artifact proof.
Use the `:wait` variants directly after UAC startup.

## Run it

```powershell
npm run eval     # full accuracy report for the seed corpus
npm run eval:review-candidates  # export unconfirmed review samples for human labeling
npm test         # runs the eval gate alongside the rest of the suite
npm run scan:assessment:test  # verifies quality-first scan ranking logic
npm run scan:iterate:validated:wait          # 20-artifact live scan
npm run scan:repeatability:wait              # 20/45/100 current-engine repeatability
npm run scan:goal:validated:wait             # final 100-artifact live scan
```

The report prints exact-match rate, overall field accuracy, a per-field
breakdown (critical fields marked with `*`), and every failing case with an
`expected "..." got "..."` diff.

Repeatability runs prove whether the visible-inventory scanner stays stable
across later sessions. They are current-path evidence and should be reported
with review/miss rates plus timing, not just click count.

## How it works

- `src/eval/ocrEvalHarness.ts` - pure metric functions. `runOcrEval(cases)`
  feeds each case's OCR text through the real `parseArtifactCandidate` and scores
  the produced fields against the labels. Order-independent for substats.
- `src/eval/corpus/seedCorpus.ts` - the seed corpus, transcribed from the
  verified assertions in `src/lib/artifactOcrParser.test.ts`. Must stay at 100%.
- `src/eval/ocrEval.test.ts` - the gate: seed field accuracy, critical-field
  accuracy, and exact-match rate must all be 1.0.

## Growing the corpus from review samples

The review queue is the corpus source. A saved review sample carries the OCR
text plus the parser's *guess* - `reviewSampleToEvalCase` extracts both.
For the local Electron queue, run:

```powershell
npm run eval:review-candidates -- --limit=80
```

This writes:

- `outputs/review-eval-candidates/review-eval-candidates.json`
- `outputs/review-eval-candidates/review-eval-candidates.md`

The exporter deduplicates samples, puts candidates with retrievable local visual
evidence first, then preserves the existing complete-OCR/staleness ordering
within each evidence group. It marks missing fast-profile fields so
stale/partial captures do not crowd out useful cases, and surfaces
ownership/lock evidence (`artifact-footer`, `equipped`, and `locked=true/false`)
for the next validation pass.

Each candidate records a safe `captureId`, `capturedAt`, and
`visualEvidence.status`. `available` means the referenced local PNG existed at
export time; `unavailable` is an OCR-only legacy/no-path case. The export never
serializes the local PNG path. Only label an `available` candidate after opening
that retained crop and checking the real artifact. Re-export if the crop was
removed; the preparer deliberately refuses `unavailable` candidates.

Current local snapshot re-exported on 2026-07-11:

- 177 records read, 0 invalid
- 138 unique candidates
- 80 exported candidates
- 36 candidates with complete fast-profile fields
- 41 likely stale or partial captures
- 14 equipped-footer candidates
- 3 candidates with retrievable visual evidence, all
  `native-review-approved` cases already represented in
  `confirmedReviewCorpus.ts`
- 77 OCR-only/unavailable exported cases, which must not be prepared as
  confirmed corpus labels until a retained crop is available

These counts describe the current local queue and may grow after later live
sessions. Do not treat the 36 complete-field candidates as automatically
correct; complete OCR is still only a review candidate until visually checked.

After manually checking one `visualEvidence: available` candidate against the
real artifact, create a confirmed corpus snippet with explicit expected labels:

```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
```

The script reads the latest
`outputs/review-eval-candidates/review-eval-candidates.json` by default and
writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`.
It refuses to run without explicit labels or retrievable visual evidence, so
parser guesses and OCR-only legacy records are not silently promoted to ground
truth. The generated provenance note includes the safe capture/run reference
and timestamp, never an absolute local crop path. Review that snippet, then paste the object into
`src/eval/corpus/confirmedReviewCorpus.ts`.

The parser's guess is a label **candidate, not ground truth** (using it directly
would be the parser grading itself). To add a real case:

1. Export review samples and choose a candidate marked
   `visualEvidence: available`.
2. Open its retained crop and confirm or correct the `expect` values against
   what the artifact actually is in-game. Set `confirmed: true`.
3. Move the corrected case into
   `src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports
   `src/eval/corpus/index.ts`, which combines the seed corpus with confirmed
   review cases.

`src/eval/corpus/confirmedReviewCorpus.test.ts` rejects common corpus mistakes:
duplicate case ids, missing OCR text, empty labels, missing source notes, or a
case that was copied in without `confirmed: true`.

Prefer cases that cover new failure modes: unseen resolutions, new sets or
characters, equipped footer noise, and OCR noise the current corpus does not
exercise. `locked` is a capture-side visual flag rather than a text parser field;
validate it from review-export metadata and live screenshots instead of adding
it to the OCR eval labels.

## When a change moves a number

- Accuracy **drops**: a regression. Read the printed failures; fix the parser or
  revert. Do not lower the threshold to make it pass.
- A change **intentionally** alters a previously-correct output: update the
  corpus label in the same commit. The label is the source of truth, not the code.