genshin-assistant/docs/ocr-eval.md

OCR Eval Harness

Field-level accuracy measurement for the artifact OCR parser. This is the gate every OCR, crop, layout, or parser change runs against (see ADR-007). It is necessary but not sufficient for live scanner proof: scan speed and review/miss rates are measured by npm run scan:iterate:validated for short iteration and npm run scan:goal:validated for the final 100-artifact proof. Use the :wait variants directly after UAC startup.

Run it

npm run eval     # full accuracy report for the seed corpus
npm run eval:review-candidates  # export unconfirmed review samples for human labeling
npm test         # runs the eval gate alongside the rest of the suite
npm run scan:assessment:test  # verifies quality-first scan ranking logic
npm run scan:iterate:validated:wait          # 20-artifact live scan
npm run scan:repeatability:wait              # 20/45/100 current-engine repeatability
npm run scan:goal:validated:wait             # final 100-artifact live scan

The report prints exact-match rate, overall field accuracy, a per-field breakdown (critical fields marked with *), and every failing case with an expected "..." got "..." diff.

Repeatability runs prove whether the visible-inventory scanner stays stable across later sessions. They are current-path evidence and should be reported with review/miss rates plus timing, not just click count.

How it works

• src/eval/ocrEvalHarness.ts - pure metric functions. runOcrEval(cases) feeds each case's OCR text through the real parseArtifactCandidate and scores the produced fields against the labels. Order-independent for substats.
• src/eval/corpus/seedCorpus.ts - the seed corpus, transcribed from the verified assertions in src/lib/artifactOcrParser.test.ts. Must stay at 100%.
• src/eval/ocrEval.test.ts - the gate: seed field accuracy, critical-field accuracy, and exact-match rate must all be 1.0.

Growing the corpus from review samples

The review queue is the corpus source. A saved review sample carries the OCR text plus the parser's guess - reviewSampleToEvalCase extracts both. For the local Electron queue, run:

npm run eval:review-candidates -- --limit=80

This writes:

• outputs/review-eval-candidates/review-eval-candidates.json
• outputs/review-eval-candidates/review-eval-candidates.md

The exporter deduplicates samples, puts candidates with retrievable local visual evidence first, then preserves the existing complete-OCR/staleness ordering within each evidence group. It marks missing fast-profile fields so stale/partial captures do not crowd out useful cases, and surfaces ownership/lock evidence (artifact-footer, equipped, and locked=true/false) for the next validation pass.

Each candidate records a safe captureId, capturedAt, and visualEvidence.status. available means the referenced local PNG existed at export time; unavailable is an OCR-only legacy/no-path case. The export never serializes the local PNG path. Only label an available candidate after opening that retained crop and checking the real artifact. Re-export if the crop was removed; the preparer deliberately refuses unavailable candidates.

Current local snapshot re-exported on 2026-07-11:

• 177 records read, 0 invalid
• 138 unique candidates
• 80 exported candidates
• 36 candidates with complete fast-profile fields
• 41 likely stale or partial captures
• 14 equipped-footer candidates
• 3 candidates with retrievable visual evidence, all native-review-approved cases already represented in confirmedReviewCorpus.ts
• 77 OCR-only/unavailable exported cases, which must not be prepared as confirmed corpus labels until a retained crop is available

These counts describe the current local queue and may grow after later live sessions. Do not treat the 36 complete-field candidates as automatically correct; complete OCR is still only a review candidate until visually checked.

After manually checking one visualEvidence: available candidate against the real artifact, create a confirmed corpus snippet with explicit expected labels:

npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json

The script reads the latest outputs/review-eval-candidates/review-eval-candidates.json by default and writes a .confirmed.ts snippet under outputs/review-eval-candidates/. It refuses to run without explicit labels or retrievable visual evidence, so parser guesses and OCR-only legacy records are not silently promoted to ground truth. The generated provenance note includes the safe capture/run reference and timestamp, never an absolute local crop path. Review that snippet, then paste the object into src/eval/corpus/confirmedReviewCorpus.ts.

The parser's guess is a label candidate, not ground truth (using it directly would be the parser grading itself). To add a real case:

1. Export review samples and choose a candidate marked visualEvidence: available.
2. Open its retained crop and confirm or correct the expect values against what the artifact actually is in-game. Set confirmed: true.
3. Move the corrected case into src/eval/corpus/confirmedReviewCorpus.ts. The main eval gate imports src/eval/corpus/index.ts, which combines the seed corpus with confirmed review cases.

src/eval/corpus/confirmedReviewCorpus.test.ts rejects common corpus mistakes: duplicate case ids, missing OCR text, empty labels, missing source notes, or a case that was copied in without confirmed: true.

Prefer cases that cover new failure modes: unseen resolutions, new sets or characters, equipped footer noise, and OCR noise the current corpus does not exercise. locked is a capture-side visual flag rather than a text parser field; validate it from review-export metadata and live screenshots instead of adding it to the OCR eval labels.

When a change moves a number

• Accuracy drops: a regression. Read the printed failures; fix the parser or revert. Do not lower the threshold to make it pass.
• A change intentionally alters a previously-correct output: update the corpus label in the same commit. The label is the source of truth, not the code.