AzuTear
149 lines
8.9 KiB
Markdown
149 lines
8.9 KiB
Markdown
# Scanner rework status
|
|
|
|
Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in
|
|
[DECISIONS.md](DECISIONS.md) for the decisions behind these. For the current
|
|
live automation runbook, see
|
|
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
|
|
|
## Current IK-Speed Target Status
|
|
|
|
See [scanner-ik-progress-report.md](scanner-ik-progress-report.md) for the full
|
|
report.
|
|
|
|
Current status:
|
|
|
|
- The scanner architecture now follows the relevant Inventory Kamera model:
|
|
32 artifact targets per page, lookup-derived fields, fast artifact OCR profile,
|
|
short readiness gates, page-overlap planning, and queued OCR/store work.
|
|
- The live runner can compare `current` and `ik-traineddata` engines and rejects
|
|
runs that are fast but fail miss/review quality thresholds.
|
|
- The final 100-artifact IK target is not proven yet. The dev-control port is
|
|
currently owned by an older elevated Electron build, and the runner correctly
|
|
refuses stale timing evidence until the app is restarted with UAC approval.
|
|
|
|
## Done (implemented, unit-tested, build green)
|
|
|
|
- **OCR eval harness** — `src/eval/`, `npm run eval`, gate in `npm test`. See
|
|
[ocr-eval.md](ocr-eval.md).
|
|
- **C# input/capture sidecar** — `native/input-helper/`, `npm run helper:build`.
|
|
Replaces the PowerShell helper on the same JSON protocol; PowerShell remains a
|
|
fallback. Verified end-to-end (spawn, runtime, base64 capture).
|
|
- **Layout profiles + OCR preprocessing** — `src/lib/layoutProfile.ts` (pure
|
|
geometry, 16:9 detection), `src/lib/ocrPreprocess.ts` (grayscale + Otsu
|
|
binarize). main.ts now uses calibrated 16:9 detail/count/grid coordinates
|
|
first and OCRs an upscaled + binarized copy.
|
|
- **Card-ready gating** — `src/lib/cardReadyGate.ts` replaces the fixed 280 ms
|
|
settle with change+stability polling; robust to animation.
|
|
- **GOOD interop** — `src/lib/goodInterop.ts` (export + best-effort import for
|
|
scanned records), Electron file-picker import/export, and store merge.
|
|
- **Rescan-merge** — `src/lib/artifactMerge.ts` collapses leveled re-scan
|
|
duplicates by a level-independent identity.
|
|
- **Data staleness warning** — `src/lib/dataPackageStatus.ts`, surfaced in the
|
|
Scanner Diagnose data-package line.
|
|
- **Lock detection (experimental)** — `src/lib/lockDetection.ts`, wired into
|
|
live capture as a read-only `locked` flag and persisted with scanned records.
|
|
- **Elevated live automation path** — `npm run dev:admin` now starts through
|
|
`scripts/dev-admin.ps1` and logs to `outputs/admin-start/admin-dev.log`.
|
|
Live status confirmed `isElevated: true`, `genshinFound: true`, and
|
|
`targetProcess: "GenshinImpact"`.
|
|
- **Read-only click probe** — `/automation/probe-click?index=1` verified that
|
|
the app can focus Genshin, move to a visible inventory tile, click it, and
|
|
observe a changed detail panel fingerprint (`clicked: true`,
|
|
`inputBlocked: false`, `changed: true`).
|
|
- **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live
|
|
with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records,
|
|
2 review samples, and 0 misses.
|
|
- **Auto-scan OCR performance pass** - auto-scan captures now use an artifact
|
|
OCR mode that skips inventory-count OCR on each tile, keeps equipped-character
|
|
OCR, raises the substat crop to catch artifact level, stores automatic review
|
|
samples without full-screen/inventory screenshots, reads only the tail of large
|
|
JSONL files, avoids review noise when only level/equipped is missing, starts
|
|
the scan with an OCR-free preflight capture, skips exact visual duplicates
|
|
before OCR, prevents repeated startup review reprocessing, omits full-frame
|
|
and inventory-preview Base64 payloads from tile captures, and applies
|
|
crop-specific Tesseract page-segmentation/whitelist parameters.
|
|
- **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the
|
|
dev-control health/status, smart-capture, probe-click, bounded scan, and
|
|
review-tail endpoints and writes evidence to `outputs/live-soak/`. On
|
|
2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10,
|
|
and 20 against the elevated running app. The limit 20 run finished `done` with
|
|
20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1
|
|
miss, and 1 page.
|
|
- **Scroll/page-transition live soak** - after the helper and loop fixes,
|
|
`scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture`
|
|
completed `done` on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35
|
|
stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the
|
|
scanner can cross from the first visible page into a scrolled page in the live
|
|
1920x1080 setup.
|
|
- **Lookup package layer** - `scripts/generate-genshin-data.cjs` now emits
|
|
normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source
|
|
version metadata, generated time, and validation summary. `src/lib/genshinLookup.ts`
|
|
provides pure matching and validation APIs, and the scanner status/dev-control
|
|
path exposes lookup validity. Auto-scan preflight blocks when the lookup package
|
|
is invalid.
|
|
- **Inventory-Kamera-style field split** - artifact detail crops now separate
|
|
name, slot, main-stat label, main-stat value, level, substats, set effects, and
|
|
footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives
|
|
slot/set/main-stat through lookup constraints before falling back to review.
|
|
- **Paimon-menu auto-entry scaffold** - auto-scan supports
|
|
`scanEntryMode: "paimon-menu"` and `/scanner/start?entry=paimon-menu&limit=N`.
|
|
The entry sends only read-only navigation (`ESC`, `B`, artifact-tab click),
|
|
then requires a valid lookup, supported layout, and detected artifact grid
|
|
before the scan loop starts. The existing visible-inventory start remains the
|
|
fallback/debug path.
|
|
- **OCR benchmark endpoint scaffold** - `/scanner/benchmark-ocr?limit=N` captures
|
|
identical artifact crops with the current engine and returns timing/field counts.
|
|
`/scanner/benchmark-ocr?engine=compare` can also compare the local
|
|
Inventory-Kamera-traineddata Tesseract.js path when
|
|
`genshin_fast_09_04_21.traineddata` is present in `data/tessdata`, `work/`, or
|
|
`IK_TESSDATA_DIR`. The OCR worker pool defaults to four workers and can be
|
|
tuned with `GAA_OCR_WORKERS=1..8`. Native Tesseract is still not the default
|
|
and should only replace `tesseract.js` after the benchmark proves it faster
|
|
and more accurate on the same crops.
|
|
- **Quality-gated live comparison** - `scripts/live-soak.ps1` now supports
|
|
goal runs for `current`, `ik-traineddata`, and `compare`, writes CSV/JSON
|
|
summaries, groups results by limit, identifies timing bottlenecks, and rejects
|
|
winners that miss the requested count, exceed 2% misses, or exceed 15% review.
|
|
`npm run scan:assessment:test` verifies this ranking logic without Genshin.
|
|
- **State-polled guided entry** - the guided auto-entry waits for Inventory,
|
|
artifact grid, and first detail card evidence instead of sleeping the full
|
|
fixed delay every time. OCR/review/store work still starts only after artifact
|
|
detail preflight passes.
|
|
|
|
## Remaining — needs the live environment or a UI pass
|
|
|
|
These cannot be finished/validated without Genshin running at the user's
|
|
resolution or without UI work best tested live:
|
|
|
|
1. **Validate/tune OCR preprocessing** on more real captures — confirm invert +
|
|
threshold + upscale factor help (not hurt) actual Tesseract reads. The
|
|
text-level eval harness cannot measure image preprocessing.
|
|
2. **Wire and benchmark native IK-traineddata OCR** against the same crop set.
|
|
The current benchmark can use IK-traineddata through Tesseract.js; native
|
|
Tesseract integration remains the next implementation step before any engine
|
|
default changes.
|
|
3. **Validate guided entry live** from world, visible inventory, and Paimon/menu
|
|
states with limits 2, 20, and 45. Confirm the artifact-tab coordinate in the
|
|
user's current 16:9 layout and keep `visible-inventory` as fallback if the
|
|
menu path is blocked.
|
|
4. **Validate locked=true** against a known locked artifact — unlocked/grey lock
|
|
was live-checked; a gold locked icon still needs a positive sample.
|
|
|
|
5. **Broader scan soak test** — after the bounded two-item live scan passed,
|
|
the next automation validation should increase the limit gradually and watch
|
|
for repeated pages, scroll behavior, duplicate handling, and OCR review rate.
|
|
6. **100-artifact IK comparison** — after `/health.appBuild.signature` matches
|
|
current source, run `npm run scan:goal:compare` and compare qualified
|
|
100-artifact results.
|
|
|
|
Visible-page limits up to 20 and a scroll/page-transition limit of 45 have
|
|
passed. The remaining soak work is now OCR accuracy, review-rate reduction, and
|
|
larger runs after the review corpus has grown.
|
|
|
|
## Grow the eval corpus
|
|
|
|
Every low-confidence review sample already stores its crops + OCR. Confirm/correct
|
|
those via `reviewSampleToEvalCase` and commit them into `src/eval/corpus/` so the
|
|
harness keeps measuring real-world accuracy across patches. See
|
|
[ocr-eval.md](ocr-eval.md).
|