genshin-assistant/docs/ocr-eval.md

OCR Eval Harness

Field-level accuracy measurement for the artifact OCR parser. This is the gate every OCR, crop, layout, or parser change runs against (see ADR-007). It is necessary but not sufficient for the IK target: live scan speed and review/miss rates are measured by npm run scan:goal:compare.

Run it

npm run eval     # full accuracy report for the seed corpus
npm test         # runs the eval gate alongside the rest of the suite
npm run scan:assessment:test  # verifies quality-first scan ranking logic

The report prints exact-match rate, overall field accuracy, a per-field breakdown (critical fields marked with *), and every failing case with an expected "..." got "..." diff.

How it works

• src/eval/ocrEvalHarness.ts - pure metric functions. runOcrEval(cases) feeds each case's OCR text through the real parseArtifactCandidate and scores the produced fields against the labels. Order-independent for substats.
• src/eval/corpus/seedCorpus.ts - the seed corpus, transcribed from the verified assertions in src/lib/artifactOcrParser.test.ts. Must stay at 100%.
• src/eval/ocrEval.test.ts - the gate: seed field accuracy, critical-field accuracy, and exact-match rate must all be 1.0.

Growing the corpus from review samples

The review queue is the corpus source. A saved review sample carries the OCR text plus the parser's guess - reviewSampleToEvalCase extracts both.

The parser's guess is a label candidate, not ground truth (using it directly would be the parser grading itself). To add a real case:

1. Convert review samples with reviewSamplesToEvalCases(records).
2. Open each produced case and confirm or correct the expect values against what the artifact actually is in-game. Set confirmed: true.
3. Move the corrected case into a file under src/eval/corpus/ and add it to the corpus array.

Prefer cases that cover new failure modes: unseen resolutions, new sets or characters, and OCR noise the current corpus does not exercise.

When a change moves a number

• Accuracy drops: a regression. Read the printed failures; fix the parser or revert. Do not lower the threshold to make it pass.
• A change intentionally alters a previously-correct output: update the corpus label in the same commit. The label is the source of truth, not the code.