genshin-assistant/docs/ocr-eval.md

OCR Eval Harness

Field-level accuracy measurement for the artifact OCR parser. This is the gate every OCR, crop, layout, or parser change runs against (see ADR-007). It is necessary but not sufficient for the IK target: live scan speed and review/miss rates are measured by npm run scan:iterate:compare:validated for short iteration and npm run scan:goal:compare:validated for the final 100-artifact proof. Use the :wait variants directly after UAC startup.

Run it

npm run eval     # full accuracy report for the seed corpus
npm run eval:review-candidates  # export unconfirmed review samples for human labeling
npm test         # runs the eval gate alongside the rest of the suite
npm run scan:assessment:test  # verifies quality-first scan ranking logic
npm run scan:iterate:compare:validated:wait  # 20-artifact live comparison
npm run scan:repeatability:wait              # 20/45/100 current-engine repeatability
npm run scan:goal:compare:validated:wait     # final 100-artifact live comparison

The report prints exact-match rate, overall field accuracy, a per-field breakdown (critical fields marked with *), and every failing case with an expected "..." got "..." diff.

Repeatability runs are single-engine evidence. Use them to prove that the visible-inventory scanner stays stable across later sessions, but keep current-vs-IK claims on scan:goal:compare:validated:*.

How it works

• src/eval/ocrEvalHarness.ts - pure metric functions. runOcrEval(cases) feeds each case's OCR text through the real parseArtifactCandidate and scores the produced fields against the labels. Order-independent for substats.
• src/eval/corpus/seedCorpus.ts - the seed corpus, transcribed from the verified assertions in src/lib/artifactOcrParser.test.ts. Must stay at 100%.
• src/eval/ocrEval.test.ts - the gate: seed field accuracy, critical-field accuracy, and exact-match rate must all be 1.0.

Growing the corpus from review samples

The review queue is the corpus source. A saved review sample carries the OCR text plus the parser's guess - reviewSampleToEvalCase extracts both. For the local Electron queue, run:

npm run eval:review-candidates -- --limit=80

This writes:

• outputs/review-eval-candidates/review-eval-candidates.json
• outputs/review-eval-candidates/review-eval-candidates.md

The exporter deduplicates samples, puts complete modern OCR captures first, marks missing fast-profile fields so stale/partial captures do not crowd out useful cases, and surfaces ownership/lock evidence (artifact-footer, equipped, and locked=true/false) for the next validation pass.

After manually checking one candidate against the real artifact, create a confirmed corpus snippet with explicit expected labels:

npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json

The script reads the latest outputs/review-eval-candidates/review-eval-candidates.json by default and writes a .confirmed.ts snippet under outputs/review-eval-candidates/. It refuses to run without explicit labels, so parser guesses are not silently promoted to ground truth. Review that snippet, then paste the object into src/eval/corpus/confirmedReviewCorpus.ts.

The parser's guess is a label candidate, not ground truth (using it directly would be the parser grading itself). To add a real case:

1. Convert review samples with reviewSamplesToEvalCases(records).
2. Open each produced case and confirm or correct the expect values against what the artifact actually is in-game. Set confirmed: true.
3. Move the corrected case into src/eval/corpus/confirmedReviewCorpus.ts. The main eval gate imports src/eval/corpus/index.ts, which combines the seed corpus with confirmed review cases.

src/eval/corpus/confirmedReviewCorpus.test.ts rejects common corpus mistakes: duplicate case ids, missing OCR text, empty labels, missing source notes, or a case that was copied in without confirmed: true.

Prefer cases that cover new failure modes: unseen resolutions, new sets or characters, equipped footer noise, and OCR noise the current corpus does not exercise. locked is a capture-side visual flag rather than a text parser field; validate it from review-export metadata and live screenshots instead of adding it to the OCR eval labels.

When a change moves a number

• Accuracy drops: a regression. Read the printed failures; fix the parser or revert. Do not lower the threshold to make it pass.
• A change intentionally alters a previously-correct output: update the corpus label in the same commit. The label is the source of truth, not the code.