# OCR Eval Harness Field-level accuracy measurement for the artifact OCR parser. This is the gate every OCR, crop, layout, or parser change runs against (see ADR-007). It is necessary but not sufficient for live scanner proof: scan speed and review/miss rates are measured by `npm run scan:iterate:validated` for short iteration and `npm run scan:goal:validated` for the final 100-artifact proof. Use the `:wait` variants directly after UAC startup. ## Run it ```powershell npm run eval # full accuracy report for the seed corpus npm run eval:review-candidates # export unconfirmed review samples for human labeling npm test # runs the eval gate alongside the rest of the suite npm run scan:assessment:test # verifies quality-first scan ranking logic npm run scan:iterate:validated:wait # 20-artifact live scan npm run scan:repeatability:wait # 20/45/100 current-engine repeatability npm run scan:goal:validated:wait # final 100-artifact live scan ``` The report prints exact-match rate, overall field accuracy, a per-field breakdown (critical fields marked with `*`), and every failing case with an `expected "..." got "..."` diff. Repeatability runs prove whether the visible-inventory scanner stays stable across later sessions. They are current-path evidence and should be reported with review/miss rates plus timing, not just click count. ## How it works - `src/eval/ocrEvalHarness.ts` - pure metric functions. `runOcrEval(cases)` feeds each case's OCR text through the real `parseArtifactCandidate` and scores the produced fields against the labels. Order-independent for substats. - `src/eval/corpus/seedCorpus.ts` - the seed corpus, transcribed from the verified assertions in `src/lib/artifactOcrParser.test.ts`. Must stay at 100%. - `src/eval/ocrEval.test.ts` - the gate: seed field accuracy, critical-field accuracy, and exact-match rate must all be 1.0. ## Growing the corpus from review samples The review queue is the corpus source. A saved review sample carries the OCR text plus the parser's *guess* - `reviewSampleToEvalCase` extracts both. For the local Electron queue, run: ```powershell npm run eval:review-candidates -- --limit=80 ``` This writes: - `outputs/review-eval-candidates/review-eval-candidates.json` - `outputs/review-eval-candidates/review-eval-candidates.md` The exporter deduplicates samples, puts candidates with retrievable local visual evidence first, then preserves the existing complete-OCR/staleness ordering within each evidence group. It marks missing fast-profile fields so stale/partial captures do not crowd out useful cases, and surfaces ownership/lock evidence (`artifact-footer`, `equipped`, and `locked=true/false`) for the next validation pass. Each candidate records a safe `captureId`, `capturedAt`, and `visualEvidence.status`. `available` means the referenced local PNG existed at export time; `unavailable` is an OCR-only legacy/no-path case. The export never serializes the local PNG path. Only label an `available` candidate after opening that retained crop and checking the real artifact. Re-export if the crop was removed; the preparer deliberately refuses `unavailable` candidates. Current local snapshot re-exported on 2026-07-11: - 177 records read, 0 invalid - 138 unique candidates - 80 exported candidates - 36 candidates with complete fast-profile fields - 41 likely stale or partial captures - 14 equipped-footer candidates - 3 candidates with retrievable visual evidence, all `native-review-approved` cases already represented in `confirmedReviewCorpus.ts` - 77 OCR-only/unavailable exported cases, which must not be prepared as confirmed corpus labels until a retained crop is available These counts describe the current local queue and may grow after later live sessions. Do not treat the 36 complete-field candidates as automatically correct; complete OCR is still only a review candidate until visually checked. After manually checking one `visualEvidence: available` candidate against the real artifact, create a confirmed corpus snippet with explicit expected labels: ```powershell npm run eval:prepare-confirmed -- --candidate= --expect-file=.\path\to\expect.json ``` The script reads the latest `outputs/review-eval-candidates/review-eval-candidates.json` by default and writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`. It refuses to run without explicit labels or retrievable visual evidence, so parser guesses and OCR-only legacy records are not silently promoted to ground truth. The generated provenance note includes the safe capture/run reference and timestamp, never an absolute local crop path. Review that snippet, then paste the object into `src/eval/corpus/confirmedReviewCorpus.ts`. The parser's guess is a label **candidate, not ground truth** (using it directly would be the parser grading itself). To add a real case: 1. Export review samples and choose a candidate marked `visualEvidence: available`. 2. Open its retained crop and confirm or correct the `expect` values against what the artifact actually is in-game. Set `confirmed: true`. 3. Move the corrected case into `src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports `src/eval/corpus/index.ts`, which combines the seed corpus with confirmed review cases. `src/eval/corpus/confirmedReviewCorpus.test.ts` rejects common corpus mistakes: duplicate case ids, missing OCR text, empty labels, missing source notes, or a case that was copied in without `confirmed: true`. Prefer cases that cover new failure modes: unseen resolutions, new sets or characters, equipped footer noise, and OCR noise the current corpus does not exercise. `locked` is a capture-side visual flag rather than a text parser field; validate it from review-export metadata and live screenshots instead of adding it to the OCR eval labels. ## When a change moves a number - Accuracy **drops**: a regression. Read the printed failures; fix the parser or revert. Do not lower the threshold to make it pass. - A change **intentionally** alters a previously-correct output: update the corpus label in the same commit. The label is the source of truth, not the code.