feat(eval): add field-level OCR accuracy harness + seed corpus
Adds a measurement gate for the artifact OCR parser (ADR-007), the prerequisite for the layout-profile and preprocessing rework. runOcrEval feeds labeled OCR text through the real parseArtifactCandidate and scores per-field / per-case accuracy. - src/eval/ocrEvalHarness.ts: pure metrics (per-field, critical-field, exact). - src/eval/corpus/seedCorpus.ts: 23 cases transcribed from the verified parser test assertions; runs at 100%. - src/eval/reviewSampleCorpus.ts: converts review samples into label *candidates* (never ground truth) so the review queue can grow the corpus. - src/eval/ocrEval.test.ts + reviewSampleCorpus.test.ts: gate (must stay 1.0) and converter unit tests. - npm run eval script; docs/ocr-eval.md; ADR-007/008/009. Also records the agreed rework direction: C# input/capture sidecar (ADR-008) and resolution-anchored layout profiles + OCR preprocessing (ADR-009). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
AzuTear
@@ -12,6 +12,9 @@ This document contains Architecture Decision Records.
|
||||
| ADR-004 | Treat in-game marking as a later opt-in feature | Accepted | 2026-07-04 |
|
||||
| ADR-005 | Use a generated Genshin data package for OCR matching | Accepted | 2026-07-04 |
|
||||
| ADR-006 | Persistent input helper and JSON artifact store before SQLite | Accepted | 2026-07-04 |
|
||||
| ADR-007 | Measure OCR accuracy with a labeled eval harness before reworking the scanner | Accepted | 2026-07-05 |
|
||||
| ADR-008 | Replace the PowerShell input/capture helper with a C# sidecar | Accepted | 2026-07-05 |
|
||||
| ADR-009 | Resolution-anchored layout profiles and OCR preprocessing over color detection | Accepted | 2026-07-05 |
|
||||
|
||||
## ADR-001: Build A Local Electron App First
|
||||
|
||||
@@ -133,3 +136,102 @@ Run one persistent PowerShell helper process (compiled once, JSON protocol over
|
||||
- Automated clicks are verified by checking that the parsed detail signature changed; repeated failures abort with a diagnosis hint instead of clicking blindly.
|
||||
- Leveling an artifact changes its signature and creates a new record; rescan-merge is an open follow-up.
|
||||
- If the helper process dies it is respawned on the next request; pending requests fail loudly instead of hanging.
|
||||
|
||||
## ADR-007: Measure OCR Accuracy With A Labeled Eval Harness Before Reworking The Scanner
|
||||
|
||||
### Status
|
||||
|
||||
Accepted
|
||||
|
||||
### Context
|
||||
|
||||
OCR and crop reliability are the core product risk (ADR-002), but there was no
|
||||
way to measure field-level accuracy. Every OCR, crop, or parser change was a
|
||||
blind change - regressions could only be caught by a human re-testing against the
|
||||
live game, and there was no baseline to compare a new engine or preprocessing
|
||||
step against.
|
||||
|
||||
### Decision
|
||||
|
||||
Add a field-level eval harness (`src/eval/`) that runs the real
|
||||
`parseArtifactCandidate` over a labeled corpus and reports per-field and
|
||||
per-case accuracy. Seed the corpus from the existing parser test cases (verified
|
||||
labels), and grow it from human-confirmed review samples via
|
||||
`reviewSampleToEvalCase`. Run it as a gate under `npm test` (must stay 100% on
|
||||
the verified seed) and as a full report via `npm run eval`.
|
||||
|
||||
### Consequences
|
||||
|
||||
- OCR/layout/preprocessing changes are measured, not guessed at; a new engine or
|
||||
a preprocessing step has to beat a recorded baseline.
|
||||
- The review queue does double duty: it flags artifacts for human correction and
|
||||
feeds the eval corpus. Review-sample `parsed` blocks are label *candidates*,
|
||||
never ground truth, to avoid the parser grading itself.
|
||||
- The corpus label is the source of truth. If a code change intentionally alters
|
||||
a correct output, the label is updated in the same commit.
|
||||
|
||||
## ADR-008: Replace The PowerShell Input/Capture Helper With A C# Sidecar
|
||||
|
||||
### Status
|
||||
|
||||
Accepted
|
||||
|
||||
### Context
|
||||
|
||||
The persistent PowerShell helper (ADR-006) still carries Windows PowerShell 5.1
|
||||
quirks (the `Marshal::SizeOf` interop bug), compiles Win32 interop at startup,
|
||||
and captures each frame by writing a PNG to the temp directory and reading it
|
||||
back. Inventory Kamera - the proven reference for automated Genshin scanning -
|
||||
uses a C#/.NET stack with InputSimulator (SendInput) and direct GDI/BitBlt
|
||||
capture.
|
||||
|
||||
### Decision
|
||||
|
||||
Replace the PowerShell helper with a self-contained .NET (C#) sidecar that speaks
|
||||
the same JSON-over-stdin/stdout protocol, so the Electron-side `InputHelperService`
|
||||
interface stays stable. The sidecar does per-monitor DPI-aware SendInput
|
||||
click/scroll, BitBlt client-rect capture returning bytes without a temp file, and
|
||||
elevation detection.
|
||||
|
||||
### Consequences
|
||||
|
||||
- No PS 5.1 marshalling bugs, no per-call interop compile, no temp-PNG churn;
|
||||
lower latency makes batch scans and the ESC/mouse failsafe polling cheaper.
|
||||
- Adds a .NET build/publish step and ships a compiled exe with the app.
|
||||
- The migration is behind the existing service interface, so the renderer and
|
||||
scan loop do not change.
|
||||
|
||||
## ADR-009: Resolution-Anchored Layout Profiles And OCR Preprocessing Over Color Detection
|
||||
|
||||
### Status
|
||||
|
||||
Accepted
|
||||
|
||||
### Context
|
||||
|
||||
The current pipeline finds the artifact detail panel with hardcoded orange/green
|
||||
color thresholds (`inferDetailRect`) and then crops fixed percentages of that
|
||||
guessed rectangle. This is brittle against HDR, color profiles, UI scale, aspect
|
||||
ratio, and game UI updates. Inventory Kamera instead requires borderless 16:9 and
|
||||
scales fixed crop coordinates from a reference resolution, then feeds Tesseract
|
||||
preprocessed (grayscale, upscaled, thresholded) crops - which is why general
|
||||
Tesseract is accurate enough for them.
|
||||
|
||||
### Decision
|
||||
|
||||
Adopt the same approach: require borderless 16:9, drive crops from
|
||||
resolution-anchored layout profiles scaled from a reference resolution (color
|
||||
detection only as a fallback), and add a per-region preprocessing pass
|
||||
(grayscale, upscale, threshold, invert) plus a digit-whitelist mode for numeric
|
||||
fields. The set name stays derived from the static piece-to-set data package (no
|
||||
dedicated set-effect crop). English-only OCR is accepted. Every change is
|
||||
validated against the ADR-007 eval harness.
|
||||
|
||||
### Consequences
|
||||
|
||||
- Crop positions become deterministic per resolution instead of per-frame guesses.
|
||||
- Preprocessing is expected to lift accuracy enough that a custom OCR engine is
|
||||
only pursued if the eval harness shows Tesseract-plus-preprocessing is
|
||||
insufficient.
|
||||
- Non-16:9 or non-borderless setups are explicitly unsupported for the auto
|
||||
scanner; the app should detect and warn rather than silently misread.
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# OCR Eval Harness
|
||||
|
||||
Field-level accuracy measurement for the artifact OCR parser. This is the gate
|
||||
every OCR, crop, layout, or parser change runs against (see ADR-007).
|
||||
|
||||
## Run it
|
||||
|
||||
```powershell
|
||||
npm run eval # full accuracy report for the seed corpus
|
||||
npm test # runs the eval gate alongside the rest of the suite
|
||||
```
|
||||
|
||||
The report prints exact-match rate, overall field accuracy, a per-field
|
||||
breakdown (critical fields marked with `*`), and every failing case with an
|
||||
`expected "..." got "..."` diff.
|
||||
|
||||
## How it works
|
||||
|
||||
- `src/eval/ocrEvalHarness.ts` - pure metric functions. `runOcrEval(cases)`
|
||||
feeds each case's OCR text through the real `parseArtifactCandidate` and scores
|
||||
the produced fields against the labels. Order-independent for substats.
|
||||
- `src/eval/corpus/seedCorpus.ts` - the seed corpus, transcribed from the
|
||||
verified assertions in `src/lib/artifactOcrParser.test.ts`. Must stay at 100%.
|
||||
- `src/eval/ocrEval.test.ts` - the gate: seed field accuracy, critical-field
|
||||
accuracy, and exact-match rate must all be 1.0.
|
||||
|
||||
## Growing the corpus from review samples
|
||||
|
||||
The review queue is the corpus source. A saved review sample carries the OCR
|
||||
text plus the parser's *guess* - `reviewSampleToEvalCase` extracts both.
|
||||
|
||||
The parser's guess is a label **candidate, not ground truth** (using it directly
|
||||
would be the parser grading itself). To add a real case:
|
||||
|
||||
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
|
||||
2. Open each produced case and confirm or correct the `expect` values against
|
||||
what the artifact actually is in-game. Set `confirmed: true`.
|
||||
3. Move the corrected case into a file under `src/eval/corpus/` and add it to the
|
||||
corpus array.
|
||||
|
||||
Prefer cases that cover new failure modes: unseen resolutions, new sets or
|
||||
characters, and OCR noise the current corpus does not exercise.
|
||||
|
||||
## When a change moves a number
|
||||
|
||||
- Accuracy **drops**: a regression. Read the printed failures; fix the parser or
|
||||
revert. Do not lower the threshold to make it pass.
|
||||
- A change **intentionally** alters a previously-correct output: update the
|
||||
corpus label in the same commit. The label is the source of truth, not the code.
|
||||
Reference in New Issue
Block a user