AzuTear
b7dbc618b3
Adds a measurement gate for the artifact OCR parser (ADR-007), the prerequisite for the layout-profile and preprocessing rework. runOcrEval feeds labeled OCR text through the real parseArtifactCandidate and scores per-field / per-case accuracy. - src/eval/ocrEvalHarness.ts: pure metrics (per-field, critical-field, exact). - src/eval/corpus/seedCorpus.ts: 23 cases transcribed from the verified parser test assertions; runs at 100%. - src/eval/reviewSampleCorpus.ts: converts review samples into label *candidates* (never ground truth) so the review queue can grow the corpus. - src/eval/ocrEval.test.ts + reviewSampleCorpus.test.ts: gate (must stay 1.0) and converter unit tests. - npm run eval script; docs/ocr-eval.md; ADR-007/008/009. Also records the agreed rework direction: C# input/capture sidecar (ADR-008) and resolution-anchored layout profiles + OCR preprocessing (ADR-009). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
50 lines
2.1 KiB
Markdown
50 lines
2.1 KiB
Markdown
# OCR Eval Harness
|
|
|
|
Field-level accuracy measurement for the artifact OCR parser. This is the gate
|
|
every OCR, crop, layout, or parser change runs against (see ADR-007).
|
|
|
|
## Run it
|
|
|
|
```powershell
|
|
npm run eval # full accuracy report for the seed corpus
|
|
npm test # runs the eval gate alongside the rest of the suite
|
|
```
|
|
|
|
The report prints exact-match rate, overall field accuracy, a per-field
|
|
breakdown (critical fields marked with `*`), and every failing case with an
|
|
`expected "..." got "..."` diff.
|
|
|
|
## How it works
|
|
|
|
- `src/eval/ocrEvalHarness.ts` - pure metric functions. `runOcrEval(cases)`
|
|
feeds each case's OCR text through the real `parseArtifactCandidate` and scores
|
|
the produced fields against the labels. Order-independent for substats.
|
|
- `src/eval/corpus/seedCorpus.ts` - the seed corpus, transcribed from the
|
|
verified assertions in `src/lib/artifactOcrParser.test.ts`. Must stay at 100%.
|
|
- `src/eval/ocrEval.test.ts` - the gate: seed field accuracy, critical-field
|
|
accuracy, and exact-match rate must all be 1.0.
|
|
|
|
## Growing the corpus from review samples
|
|
|
|
The review queue is the corpus source. A saved review sample carries the OCR
|
|
text plus the parser's *guess* - `reviewSampleToEvalCase` extracts both.
|
|
|
|
The parser's guess is a label **candidate, not ground truth** (using it directly
|
|
would be the parser grading itself). To add a real case:
|
|
|
|
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
|
|
2. Open each produced case and confirm or correct the `expect` values against
|
|
what the artifact actually is in-game. Set `confirmed: true`.
|
|
3. Move the corrected case into a file under `src/eval/corpus/` and add it to the
|
|
corpus array.
|
|
|
|
Prefer cases that cover new failure modes: unseen resolutions, new sets or
|
|
characters, and OCR noise the current corpus does not exercise.
|
|
|
|
## When a change moves a number
|
|
|
|
- Accuracy **drops**: a regression. Read the printed failures; fix the parser or
|
|
revert. Do not lower the threshold to make it pass.
|
|
- A change **intentionally** alters a previously-correct output: update the
|
|
corpus label in the same commit. The label is the source of truth, not the code.
|