# Scanner rework status Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in [DECISIONS.md](DECISIONS.md) for the decisions behind these. For the current live automation runbook, see [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md). ## Current IK-Speed Target Status See [scanner-ik-progress-report.md](scanner-ik-progress-report.md) for the full report. Current status: - The scanner architecture now follows the relevant Inventory Kamera model: 32 artifact targets per page, lookup-derived fields, fast artifact OCR profile, short readiness gates, page-overlap planning, and queued OCR/store work. - The live runner can compare `current` and `ik-traineddata` engines and rejects runs that are fast but fail miss/review quality thresholds. - The final 100-artifact IK target is not proven yet. The dev-control port is currently owned by an older elevated Electron build, and the runner correctly refuses stale timing evidence until the app is restarted with UAC approval. ## Done (implemented, unit-tested, build green) - **OCR eval harness** — `src/eval/`, `npm run eval`, gate in `npm test`. See [ocr-eval.md](ocr-eval.md). - **C# input/capture sidecar** — `native/input-helper/`, `npm run helper:build`. Replaces the PowerShell helper on the same JSON protocol; PowerShell remains a fallback. Verified end-to-end (spawn, runtime, base64 capture). - **Layout profiles + OCR preprocessing** — `src/lib/layoutProfile.ts` (pure geometry, 16:9 detection), `src/lib/ocrPreprocess.ts` (grayscale + Otsu binarize). main.ts now uses calibrated 16:9 detail/count/grid coordinates first and OCRs an upscaled + binarized copy. - **Card-ready gating** — `src/lib/cardReadyGate.ts` replaces the fixed 280 ms settle with change+stability polling; robust to animation. - **GOOD interop** — `src/lib/goodInterop.ts` (export + best-effort import for scanned records), Electron file-picker import/export, and store merge. - **Rescan-merge** — `src/lib/artifactMerge.ts` collapses leveled re-scan duplicates by a level-independent identity. - **Data staleness warning** — `src/lib/dataPackageStatus.ts`, surfaced in the Scanner Diagnose data-package line. - **Lock detection (experimental)** — `src/lib/lockDetection.ts`, wired into live capture as a read-only `locked` flag and persisted with scanned records. - **Elevated live automation path** — `npm run dev:admin` now starts through `scripts/dev-admin.ps1` and logs to `outputs/admin-start/admin-dev.log`. Live status confirmed `isElevated: true`, `genshinFound: true`, and `targetProcess: "GenshinImpact"`. - **Read-only click probe** — `/automation/probe-click?index=1` verified that the app can focus Genshin, move to a visible inventory tile, click it, and observe a changed detail panel fingerprint (`clicked: true`, `inputBlocked: false`, `changed: true`). - **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records, 2 review samples, and 0 misses. - **Auto-scan OCR performance pass** - auto-scan captures now use an artifact OCR mode that skips inventory-count OCR on each tile, keeps equipped-character OCR, raises the substat crop to catch artifact level, stores automatic review samples without full-screen/inventory screenshots, reads only the tail of large JSONL files, avoids review noise when only level/equipped is missing, starts the scan with an OCR-free preflight capture, skips exact visual duplicates before OCR, prevents repeated startup review reprocessing, omits full-frame and inventory-preview Base64 payloads from tile captures, and applies crop-specific Tesseract page-segmentation/whitelist parameters. - **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the dev-control health/status, smart-capture, probe-click, bounded scan, and review-tail endpoints and writes evidence to `outputs/live-soak/`. On 2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10, and 20 against the elevated running app. The limit 20 run finished `done` with 20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1 miss, and 1 page. - **Scroll/page-transition live soak** - after the helper and loop fixes, `scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture` completed `done` on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35 stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the scanner can cross from the first visible page into a scrolled page in the live 1920x1080 setup. - **Lookup package layer** - `scripts/generate-genshin-data.cjs` now emits normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source version metadata, generated time, and validation summary. `src/lib/genshinLookup.ts` provides pure matching and validation APIs, and the scanner status/dev-control path exposes lookup validity. Auto-scan preflight blocks when the lookup package is invalid. - **Inventory-Kamera-style field split** - artifact detail crops now separate name, slot, main-stat label, main-stat value, level, substats, set effects, and footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives slot/set/main-stat through lookup constraints before falling back to review. - **Paimon-menu auto-entry scaffold** - auto-scan supports `scanEntryMode: "paimon-menu"` and `/scanner/start?entry=paimon-menu&limit=N`. The entry sends only read-only navigation (`ESC`, `B`, artifact-tab click), then requires a valid lookup, supported layout, and detected artifact grid before the scan loop starts. The existing visible-inventory start remains the fallback/debug path. - **OCR benchmark endpoint scaffold** - `/scanner/benchmark-ocr?limit=N` captures identical artifact crops with the current engine and returns timing/field counts. `/scanner/benchmark-ocr?engine=compare` can also compare the local Inventory-Kamera-traineddata Tesseract.js path when `genshin_fast_09_04_21.traineddata` is present in `data/tessdata`, `work/`, or `IK_TESSDATA_DIR`. The OCR worker pool defaults to four workers and can be tuned with `GAA_OCR_WORKERS=1..8`. Native Tesseract is still not the default and should only replace `tesseract.js` after the benchmark proves it faster and more accurate on the same crops. - **Quality-gated live comparison** - `scripts/live-soak.ps1` now supports goal runs for `current`, `ik-traineddata`, and `compare`, writes CSV/JSON summaries, groups results by limit, identifies timing bottlenecks, and rejects winners that miss the requested count, exceed 2% misses, or exceed 15% review. `npm run scan:assessment:test` verifies this ranking logic without Genshin. - **State-polled guided entry** - the guided auto-entry waits for Inventory, artifact grid, and first detail card evidence instead of sleeping the full fixed delay every time. OCR/review/store work still starts only after artifact detail preflight passes. ## Remaining — needs the live environment or a UI pass These cannot be finished/validated without Genshin running at the user's resolution or without UI work best tested live: 1. **Validate/tune OCR preprocessing** on more real captures — confirm invert + threshold + upscale factor help (not hurt) actual Tesseract reads. The text-level eval harness cannot measure image preprocessing. 2. **Wire and benchmark native IK-traineddata OCR** against the same crop set. The current benchmark can use IK-traineddata through Tesseract.js; native Tesseract integration remains the next implementation step before any engine default changes. 3. **Validate guided entry live** from world, visible inventory, and Paimon/menu states with limits 2, 20, and 45. Confirm the artifact-tab coordinate in the user's current 16:9 layout and keep `visible-inventory` as fallback if the menu path is blocked. 4. **Validate locked=true** against a known locked artifact — unlocked/grey lock was live-checked; a gold locked icon still needs a positive sample. 5. **Broader scan soak test** — after the bounded two-item live scan passed, the next automation validation should increase the limit gradually and watch for repeated pages, scroll behavior, duplicate handling, and OCR review rate. 6. **100-artifact IK comparison** — after `/health.appBuild.signature` matches current source, run `npm run scan:goal:compare` and compare qualified 100-artifact results. Visible-page limits up to 20 and a scroll/page-transition limit of 45 have passed. The remaining soak work is now OCR accuracy, review-rate reduction, and larger runs after the review corpus has grown. ## Grow the eval corpus Every low-confidence review sample already stores its crops + OCR. Confirm/correct those via `reviewSampleToEvalCase` and commit them into `src/eval/corpus/` so the harness keeps measuring real-world accuracy across patches. See [ocr-eval.md](ocr-eval.md).