Prepare scanner branch for merge
This commit is contained in:
+47
-4
@@ -3,14 +3,19 @@
|
||||
Field-level accuracy measurement for the artifact OCR parser. This is the gate
|
||||
every OCR, crop, layout, or parser change runs against (see ADR-007).
|
||||
It is necessary but not sufficient for the IK target: live scan speed and
|
||||
review/miss rates are measured by `npm run scan:goal:compare`.
|
||||
review/miss rates are measured by `npm run scan:iterate:compare:validated` for
|
||||
short iteration and `npm run scan:goal:compare:validated` for the final
|
||||
100-artifact proof. Use the `:wait` variants directly after UAC startup.
|
||||
|
||||
## Run it
|
||||
|
||||
```powershell
|
||||
npm run eval # full accuracy report for the seed corpus
|
||||
npm run eval:review-candidates # export unconfirmed review samples for human labeling
|
||||
npm test # runs the eval gate alongside the rest of the suite
|
||||
npm run scan:assessment:test # verifies quality-first scan ranking logic
|
||||
npm run scan:iterate:compare:validated:wait # 20-artifact live comparison
|
||||
npm run scan:goal:compare:validated:wait # final 100-artifact live comparison
|
||||
```
|
||||
|
||||
The report prints exact-match rate, overall field accuracy, a per-field
|
||||
@@ -31,6 +36,35 @@ breakdown (critical fields marked with `*`), and every failing case with an
|
||||
|
||||
The review queue is the corpus source. A saved review sample carries the OCR
|
||||
text plus the parser's *guess* - `reviewSampleToEvalCase` extracts both.
|
||||
For the local Electron queue, run:
|
||||
|
||||
```powershell
|
||||
npm run eval:review-candidates -- --limit=80
|
||||
```
|
||||
|
||||
This writes:
|
||||
|
||||
- `outputs/review-eval-candidates/review-eval-candidates.json`
|
||||
- `outputs/review-eval-candidates/review-eval-candidates.md`
|
||||
|
||||
The exporter deduplicates samples, puts complete modern OCR captures first,
|
||||
marks missing fast-profile fields so stale/partial captures do not crowd out
|
||||
useful cases, and surfaces ownership/lock evidence (`artifact-footer`,
|
||||
`equipped`, and `locked=true/false`) for the next validation pass.
|
||||
|
||||
After manually checking one candidate against the real artifact, create a
|
||||
confirmed corpus snippet with explicit expected labels:
|
||||
|
||||
```powershell
|
||||
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
|
||||
```
|
||||
|
||||
The script reads the latest
|
||||
`outputs/review-eval-candidates/review-eval-candidates.json` by default and
|
||||
writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`.
|
||||
It refuses to run without explicit labels, so parser guesses are not silently
|
||||
promoted to ground truth. Review that snippet, then paste the object into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
The parser's guess is a label **candidate, not ground truth** (using it directly
|
||||
would be the parser grading itself). To add a real case:
|
||||
@@ -38,11 +72,20 @@ would be the parser grading itself). To add a real case:
|
||||
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
|
||||
2. Open each produced case and confirm or correct the `expect` values against
|
||||
what the artifact actually is in-game. Set `confirmed: true`.
|
||||
3. Move the corrected case into a file under `src/eval/corpus/` and add it to the
|
||||
corpus array.
|
||||
3. Move the corrected case into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports
|
||||
`src/eval/corpus/index.ts`, which combines the seed corpus with confirmed
|
||||
review cases.
|
||||
|
||||
`src/eval/corpus/confirmedReviewCorpus.test.ts` rejects common corpus mistakes:
|
||||
duplicate case ids, missing OCR text, empty labels, missing source notes, or a
|
||||
case that was copied in without `confirmed: true`.
|
||||
|
||||
Prefer cases that cover new failure modes: unseen resolutions, new sets or
|
||||
characters, and OCR noise the current corpus does not exercise.
|
||||
characters, equipped footer noise, and OCR noise the current corpus does not
|
||||
exercise. `locked` is a capture-side visual flag rather than a text parser field;
|
||||
validate it from review-export metadata and live screenshots instead of adding
|
||||
it to the OCR eval labels.
|
||||
|
||||
## When a change moves a number
|
||||
|
||||
|
||||
Reference in New Issue
Block a user