Files

T

AzuTear b7dbc618b3 feat(eval): add field-level OCR accuracy harness + seed corpus

Adds a measurement gate for the artifact OCR parser (ADR-007), the prerequisite
for the layout-profile and preprocessing rework. runOcrEval feeds labeled OCR
text through the real parseArtifactCandidate and scores per-field / per-case
accuracy.

- src/eval/ocrEvalHarness.ts: pure metrics (per-field, critical-field, exact).
- src/eval/corpus/seedCorpus.ts: 23 cases transcribed from the verified parser
  test assertions; runs at 100%.
- src/eval/reviewSampleCorpus.ts: converts review samples into label *candidates*
  (never ground truth) so the review queue can grow the corpus.
- src/eval/ocrEval.test.ts + reviewSampleCorpus.test.ts: gate (must stay 1.0) and
  converter unit tests.
- npm run eval script; docs/ocr-eval.md; ADR-007/008/009.

Also records the agreed rework direction: C# input/capture sidecar (ADR-008) and
resolution-anchored layout profiles + OCR preprocessing (ADR-009).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-05 20:41:30 +02:00

2.1 KiB

Raw Permalink Blame History

OCR Eval Harness

Field-level accuracy measurement for the artifact OCR parser. This is the gate every OCR, crop, layout, or parser change runs against (see ADR-007).

Run it

npm run eval     # full accuracy report for the seed corpus
npm test         # runs the eval gate alongside the rest of the suite

The report prints exact-match rate, overall field accuracy, a per-field breakdown (critical fields marked with *), and every failing case with an expected "..." got "..." diff.

How it works

src/eval/ocrEvalHarness.ts - pure metric functions. runOcrEval(cases) feeds each case's OCR text through the real parseArtifactCandidate and scores the produced fields against the labels. Order-independent for substats.
src/eval/corpus/seedCorpus.ts - the seed corpus, transcribed from the verified assertions in src/lib/artifactOcrParser.test.ts. Must stay at 100%.
src/eval/ocrEval.test.ts - the gate: seed field accuracy, critical-field accuracy, and exact-match rate must all be 1.0.

Growing the corpus from review samples

The review queue is the corpus source. A saved review sample carries the OCR text plus the parser's guess - reviewSampleToEvalCase extracts both.

The parser's guess is a label candidate, not ground truth (using it directly would be the parser grading itself). To add a real case:

Convert review samples with reviewSamplesToEvalCases(records).
Open each produced case and confirm or correct the expect values against what the artifact actually is in-game. Set confirmed: true.
Move the corrected case into a file under src/eval/corpus/ and add it to the corpus array.

Prefer cases that cover new failure modes: unseen resolutions, new sets or characters, and OCR noise the current corpus does not exercise.

When a change moves a number

Accuracy drops: a regression. Read the printed failures; fix the parser or revert. Do not lower the threshold to make it pass.
A change intentionally alters a previously-correct output: update the corpus label in the same commit. The label is the source of truth, not the code.

2.1 KiB Raw Permalink Blame History