genshin-assistant/docs/scanner-rework-status.md

Scanner rework status

Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in DECISIONS.md for the decisions behind these. For the current live automation runbook, see AUTOMATION_LIVE_SCAN.md.

Current IK-Speed Target Status

See scanner-ik-progress-report.md for the full report.

Current status:

• The scanner architecture now follows the relevant Inventory Kamera model: 32 artifact targets per page, lookup-derived fields, fast artifact OCR profile, short readiness gates, page-overlap planning, and queued OCR/store work.
• The live runner can compare current and ik-traineddata engines and rejects runs that are fast but fail miss/review quality thresholds.
• The final 100-artifact IK target is not proven yet. The dev-control port is currently owned by an older elevated Electron build, and the runner correctly refuses stale timing evidence until the app is restarted with UAC approval.

Done (implemented, unit-tested, build green)

• OCR eval harness — src/eval/, npm run eval, gate in npm test. See ocr-eval.md.
• C# input/capture sidecar — native/input-helper/, npm run helper:build. Replaces the PowerShell helper on the same JSON protocol; PowerShell remains a fallback. Verified end-to-end (spawn, runtime, base64 capture).
• Layout profiles + OCR preprocessing — src/lib/layoutProfile.ts (pure geometry, 16:9 detection), src/lib/ocrPreprocess.ts (grayscale + Otsu binarize). main.ts now uses calibrated 16:9 detail/count/grid coordinates first and OCRs an upscaled + binarized copy.
• Card-ready gating — src/lib/cardReadyGate.ts replaces the fixed 280 ms settle with change+stability polling; robust to animation.
• GOOD interop — src/lib/goodInterop.ts (export + best-effort import for scanned records), Electron file-picker import/export, and store merge.
• Rescan-merge — src/lib/artifactMerge.ts collapses leveled re-scan duplicates by a level-independent identity.
• Data staleness warning — src/lib/dataPackageStatus.ts, surfaced in the Scanner Diagnose data-package line.
• Lock detection (experimental) — src/lib/lockDetection.ts, wired into live capture as a read-only locked flag and persisted with scanned records.
• Elevated live automation path — npm run dev:admin now starts through scripts/dev-admin.ps1 and logs to outputs/admin-start/admin-dev.log. Live status confirmed isElevated: true, genshinFound: true, and targetProcess: "GenshinImpact".
• Read-only click probe — /automation/probe-click?index=1 verified that the app can focus Genshin, move to a visible inventory tile, click it, and observe a changed detail panel fingerprint (clicked: true, inputBlocked: false, changed: true).
• Bounded auto-scan validation — /scanner/start?limit=2 completed live with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records, 2 review samples, and 0 misses.
• Auto-scan OCR performance pass - auto-scan captures now use an artifact OCR mode that skips inventory-count OCR on each tile, keeps equipped-character OCR, raises the substat crop to catch artifact level, stores automatic review samples without full-screen/inventory screenshots, reads only the tail of large JSONL files, avoids review noise when only level/equipped is missing, starts the scan with an OCR-free preflight capture, skips exact visual duplicates before OCR, prevents repeated startup review reprocessing, omits full-frame and inventory-preview Base64 payloads from tile captures, and applies crop-specific Tesseract page-segmentation/whitelist parameters.
• Visible-page live soak helper - scripts/live-soak.ps1 now drives the dev-control health/status, smart-capture, probe-click, bounded scan, and review-tail endpoints and writes evidence to outputs/live-soak/. On 2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10, and 20 against the elevated running app. The limit 20 run finished done with 20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1 miss, and 1 page.
• Scroll/page-transition live soak - after the helper and loop fixes, scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture completed done on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35 stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the scanner can cross from the first visible page into a scrolled page in the live 1920x1080 setup.
• Lookup package layer - scripts/generate-genshin-data.cjs now emits normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source version metadata, generated time, and validation summary. src/lib/genshinLookup.ts provides pure matching and validation APIs, and the scanner status/dev-control path exposes lookup validity. Auto-scan preflight blocks when the lookup package is invalid.
• Inventory-Kamera-style field split - artifact detail crops now separate name, slot, main-stat label, main-stat value, level, substats, set effects, and footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives slot/set/main-stat through lookup constraints before falling back to review.
• Paimon-menu auto-entry scaffold - auto-scan supports scanEntryMode: "paimon-menu" and /scanner/start?entry=paimon-menu&limit=N. The entry sends only read-only navigation (ESC, B, artifact-tab click), then requires a valid lookup, supported layout, and detected artifact grid before the scan loop starts. The existing visible-inventory start remains the fallback/debug path.
• OCR benchmark endpoint scaffold - /scanner/benchmark-ocr?limit=N captures identical artifact crops with the current engine and returns timing/field counts. /scanner/benchmark-ocr?engine=compare can also compare the local Inventory-Kamera-traineddata Tesseract.js path when genshin_fast_09_04_21.traineddata is present in data/tessdata, work/, or IK_TESSDATA_DIR. The OCR worker pool defaults to four workers and can be tuned with GAA_OCR_WORKERS=1..8. Native Tesseract is still not the default and should only replace tesseract.js after the benchmark proves it faster and more accurate on the same crops.
• Quality-gated live comparison - scripts/live-soak.ps1 now supports goal runs for current, ik-traineddata, and compare, writes CSV/JSON summaries, groups results by limit, identifies timing bottlenecks, and rejects winners that miss the requested count, exceed 2% misses, or exceed 15% review. npm run scan:assessment:test verifies this ranking logic without Genshin.
• State-polled guided entry - the guided auto-entry waits for Inventory, artifact grid, and first detail card evidence instead of sleeping the full fixed delay every time. OCR/review/store work still starts only after artifact detail preflight passes.

Remaining — needs the live environment or a UI pass

These cannot be finished/validated without Genshin running at the user's resolution or without UI work best tested live:

1. Validate/tune OCR preprocessing on more real captures — confirm invert + threshold + upscale factor help (not hurt) actual Tesseract reads. The text-level eval harness cannot measure image preprocessing.

2. Wire and benchmark native IK-traineddata OCR against the same crop set. The current benchmark can use IK-traineddata through Tesseract.js; native Tesseract integration remains the next implementation step before any engine default changes.

3. Validate guided entry live from world, visible inventory, and Paimon/menu states with limits 2, 20, and 45. Confirm the artifact-tab coordinate in the user's current 16:9 layout and keep visible-inventory as fallback if the menu path is blocked.

4. Validate locked=true against a known locked artifact — unlocked/grey lock was live-checked; a gold locked icon still needs a positive sample.

5. Broader scan soak test — after the bounded two-item live scan passed, the next automation validation should increase the limit gradually and watch for repeated pages, scroll behavior, duplicate handling, and OCR review rate.

6. 100-artifact IK comparison — after /health.appBuild.signature matches current source, run npm run scan:goal:compare and compare qualified 100-artifact results.

Visible-page limits up to 20 and a scroll/page-transition limit of 45 have passed. The remaining soak work is now OCR accuracy, review-rate reduction, and larger runs after the review corpus has grown.

Grow the eval corpus

Every low-confidence review sample already stores its crops + OCR. Confirm/correct those via reviewSampleToEvalCase and commit them into src/eval/corpus/ so the harness keeps measuring real-world accuracy across patches. See ocr-eval.md.