bao/genshin-assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(corpus): require retrievable visual review evidence

Browse Source
This commit is contained in:
AzuTear
2026-07-11 16:13:02 +02:00
parent 8b9f948c6b
commit 3e2ab61c97
14 changed files with 325 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ocr-eval.md
+28 -14
View File
@@ -52,12 +52,21 @@ This writes:
- `outputs/review-eval-candidates/review-eval-candidates.json`
- `outputs/review-eval-candidates/review-eval-candidates.md`
The exporter deduplicates samples, puts complete modern OCR captures first,
marks missing fast-profile fields so stale/partial captures do not crowd out
useful cases, and surfaces ownership/lock evidence (`artifact-footer`,
`equipped`, and `locked=true/false`) for the next validation pass.
The exporter deduplicates samples, puts candidates with retrievable local visual
evidence first, then preserves the existing complete-OCR/staleness ordering
within each evidence group. It marks missing fast-profile fields so
stale/partial captures do not crowd out useful cases, and surfaces
ownership/lock evidence (`artifact-footer`, `equipped`, and `locked=true/false`)
for the next validation pass.
Current local snapshot from 2026-07-10:
Each candidate records a safe `captureId`, `capturedAt`, and
`visualEvidence.status`. `available` means the referenced local PNG existed at
export time; `unavailable` is an OCR-only legacy/no-path case. The export never
serializes the local PNG path. Only label an `available` candidate after opening
that retained crop and checking the real artifact. Re-export if the crop was
removed; the preparer deliberately refuses `unavailable` candidates.
Current local snapshot re-exported on 2026-07-11:
- 177 records read, 0 invalid
- 138 unique candidates
@@ -65,16 +74,18 @@ Current local snapshot from 2026-07-10:
- 36 candidates with complete fast-profile fields
- 41 likely stale or partial captures
- 14 equipped-footer candidates
- 3 `native-review-approved` cases already represented in
`confirmedReviewCorpus.ts`; the remaining 77 exported cases still require
explicit human labels
- 3 candidates with retrievable visual evidence, all
`native-review-approved` cases already represented in
`confirmedReviewCorpus.ts`
- 77 OCR-only/unavailable exported cases, which must not be prepared as
confirmed corpus labels until a retained crop is available
These counts describe the current local queue and may grow after later live
sessions. Do not treat the 36 complete-field candidates as automatically
correct; complete OCR is still only a review candidate until visually checked.
After manually checking one candidate against the real artifact, create a
confirmed corpus snippet with explicit expected labels:
After manually checking one `visualEvidence: available` candidate against the
real artifact, create a confirmed corpus snippet with explicit expected labels:
```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
@@ -83,15 +94,18 @@ npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\pat
The script reads the latest
`outputs/review-eval-candidates/review-eval-candidates.json` by default and
writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`.
It refuses to run without explicit labels, so parser guesses are not silently
promoted to ground truth. Review that snippet, then paste the object into
It refuses to run without explicit labels or retrievable visual evidence, so
parser guesses and OCR-only legacy records are not silently promoted to ground
truth. The generated provenance note includes the safe capture/run reference
and timestamp, never an absolute local crop path. Review that snippet, then paste the object into
`src/eval/corpus/confirmedReviewCorpus.ts`.
The parser's guess is a label **candidate, not ground truth** (using it directly
would be the parser grading itself). To add a real case:
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
2. Open each produced case and confirm or correct the `expect` values against
1. Export review samples and choose a candidate marked
`visualEvidence: available`.
2. Open its retained crop and confirm or correct the `expect` values against
what the artifact actually is in-game. Set `confirmed: true`.
3. Move the corrected case into
`src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports