# Scanner rework status Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in [DECISIONS.md](DECISIONS.md) for the decisions behind these. For the current live automation runbook, see [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md). ## Current IK-Speed Target Status See [scanner-ik-progress-report.md](scanner-ik-progress-report.md) for the full report. Current status: - The scanner architecture now follows the relevant Inventory Kamera model: 32 artifact targets per page, lookup-derived fields, fast artifact OCR profile, direct detail-fingerprint verification from the OCR capture, page-overlap planning, and batched store work. - The live runner can compare `current` and `ik-traineddata` engines and rejects runs that are fast but fail miss/review quality thresholds. - The final current-vs-IK-traineddata 100-artifact comparison is now proven for the current live environment. On 2026-07-08, `npm run scan:goal:compare:validated` passed with evidence at `outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`. `current` won with `100/100` parsed, `0` review, `0` misses, and `378 ms/artifact` active average. `ik-traineddata` was rejected at 100 because it parsed `97/100`, had `5` review and `3` misses. The next optional speed target remains `3 artifacts/second`, which means `333 ms/artifact` or faster on clean 20-artifact iterations. ## Done (implemented, unit-tested, build green) - **OCR eval harness** — `src/eval/`, `npm run eval`, gate in `npm test`. See [ocr-eval.md](ocr-eval.md). - **C# input/capture sidecar** — `native/input-helper/`, `npm run helper:build`. Replaces the PowerShell helper on the same JSON protocol; PowerShell remains a fallback. Verified end-to-end (spawn, runtime, base64 capture). - **Layout profiles + OCR preprocessing** — `src/lib/layoutProfile.ts` (pure geometry, 16:9 detection), `src/lib/ocrPreprocess.ts` (grayscale + Otsu binarize). main.ts now uses calibrated 16:9 detail/count/grid coordinates first and OCRs an upscaled + binarized copy. - **Card-ready gating** — `src/lib/cardReadyGate.ts` replaces the fixed 280 ms settle with change+stability polling; robust to animation. - **GOOD interop** — `src/lib/goodInterop.ts` (export + best-effort import for scanned records), Electron file-picker import/export, and store merge. - **Rescan-merge** — `src/lib/artifactMerge.ts` collapses leveled re-scan duplicates by a level-independent identity. - **Data staleness warning** — `src/lib/dataPackageStatus.ts`, surfaced in the Scanner Diagnose data-package line. - **Lock detection (experimental)** — `src/lib/lockDetection.ts`, wired into live capture as a read-only `locked` flag and persisted with scanned records. - **Elevated live automation path** — `npm run dev:admin` now starts through `scripts/dev-admin.ps1` and logs to `outputs/admin-start/admin-dev.log`. Live status confirmed `isElevated: true`, `genshinFound: true`, and `targetProcess: "GenshinImpact"`. - **Read-only click probe** — `/automation/probe-click?index=1` verified that the app can focus Genshin, move to a visible inventory tile, click it, and observe a changed detail panel fingerprint (`clicked: true`, `inputBlocked: false`, `changed: true`). - **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records, 2 review samples, and 0 misses. - **Auto-scan OCR performance pass** - auto-scan captures now use an artifact OCR mode that skips inventory-count OCR on each tile, keeps equipped-character OCR on the real artifact-read captures, raises the substat crop to catch artifact level, stores automatic review samples without full-screen/inventory screenshots, reads only the tail of large JSONL files, avoids review noise when only level/equipped is missing, starts the scan with an OCR-free preflight capture, prevents repeated startup review reprocessing, omits full-frame and inventory-preview Base64 payloads from tile captures, and applies crop-specific Tesseract page-segmentation/whitelist parameters. - **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the dev-control health/status, smart-capture, probe-click, bounded scan, and review-tail endpoints and writes evidence to `outputs/live-soak/`. On 2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10, and 20 against the elevated running app. The limit 20 run finished `done` with 20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1 miss, and 1 page. - **Scroll/page-transition live soak** - after the helper and loop fixes, `scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture` completed `done` on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35 stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the scanner can cross from the first visible page into a scrolled page in the live 1920x1080 setup. - **Lookup package layer** - `scripts/generate-genshin-data.cjs` now emits normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source version metadata, generated time, and validation summary. `src/lib/genshinLookup.ts` provides pure matching and validation APIs, and the scanner status/dev-control path exposes lookup validity. Auto-scan preflight blocks when the lookup package is invalid. - **Inventory-Kamera-style field split** - artifact detail crops now separate name, slot, main-stat label, main-stat value, level, substats, set effects, and footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives slot/set/main-stat through lookup constraints before falling back to review. - **Paimon-menu auto-entry scaffold** - auto-scan supports `scanEntryMode: "paimon-menu"` and `/scanner/start?entry=paimon-menu&limit=N`. The entry sends only read-only navigation (`ESC`, `B`, artifact-tab click), then requires a valid lookup, supported layout, and detected artifact grid before the scan loop starts. The existing visible-inventory start remains the fallback/debug path. - **OCR benchmark endpoint scaffold** - `/scanner/benchmark-ocr?limit=N` captures identical artifact crops with the current engine and returns timing/field counts. `/scanner/benchmark-ocr?engine=compare` can also compare the local Inventory-Kamera-traineddata Tesseract.js path when `genshin_fast_09_04_21.traineddata` is present in `data/tessdata`, `work/`, or `IK_TESSDATA_DIR`. The OCR worker pool defaults to four workers and can be tuned with `GAA_OCR_WORKERS=1..8`. Native Tesseract is still not the default and should only replace `tesseract.js` after the benchmark proves it faster and more accurate on the same crops. - **Quality-gated live comparison** - `scripts/live-soak.ps1` now supports goal runs for `current`, `ik-traineddata`, and `compare`, writes CSV/JSON summaries, groups results by limit, identifies timing bottlenecks, and rejects winners that miss the requested count, exceed 2% misses, or exceed 15% review. `npm run scan:assessment:test` verifies this ranking logic without Genshin. The assessment also reports `goal100Decision` and `goal100.comparisonComplete`, so a single-engine 100-artifact run cannot be misread as the final IK comparison. Use `npm run scan:iterate:compare:validated:wait` for the 20-artifact live iteration and `npm run scan:goal:compare:validated:wait` for the final proof when starting directly after UAC. The validator `--summary` output includes the assessment path and timestamp for reporting. - **State-polled guided entry** - the guided auto-entry waits for Inventory, artifact grid, and first detail card evidence instead of sleeping the full fixed delay every time. OCR/review/store work still starts only after artifact detail preflight passes. - **Hot-loop speed pass (2026-07-08)** - the scan loop no longer performs a separate card-ready capture before OCR; the artifact OCR capture itself verifies detail-fingerprint change. Routine click diagnostics and scan stat publishes are throttled. Auto-scan artifact captures no longer update the full preview/topbar UI on every tile. Store writes can be batched so the scan path avoids per-artifact save/reload churn. Auto-scan artifact captures now use a direct GDI hot path and skip Electron `desktopCapturer.getSources()` in the per-artifact loop. - **3/s instrumentation pass (2026-07-08)** - artifact hot-path captures omit the detail-preview payload, and scan stats now split inner capture time from end-to-end capture roundtrip time. Use `averageCaptureRoundTripMs` and `averageCaptureRoundTripOverheadMs` in the next `limit=20` live iteration to decide whether the next cut belongs in native capture transport or OCR. - **3/s live attempt (2026-07-08)** - the missing-detail-preview review trigger was fixed and tested. The best clean 20-artifact run reached `7285 ms` (`364 ms/artifact`, about `2.75 artifacts/second`) with 0 review and 0 misses. The final stable run on `2026-07-08-direct-gdi-reviewfix` completed `20/20` with 0 review, 0 misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail region capture, 5 OCR workers, DataURL buffer decode, and substat `PSM.SINGLE_COLUMN` were tested and rejected as slower. - **Review-to-eval loop (2026-07-08)** - `npm run eval:review-candidates` exports the local review queue into `outputs/review-eval-candidates/` as a human-labeling worklist. The exporter deduplicates samples, surfaces complete fast-field captures first, marks stale captures, and now surfaces equipped footer OCR plus `locked=true/false` payload counts for the next ownership/lock validation pass. Its output is deliberately ignored by Git and must not be treated as ground truth until fields are confirmed against the real artifact. Confirmed review labels now have a dedicated corpus file, `src/eval/corpus/confirmedReviewCorpus.ts`, with tests that reject duplicate ids, empty labels, and unconfirmed entries. The helper `npm run eval:prepare-confirmed` generates a paste-ready confirmed-case snippet only when explicit expected labels are provided. - **Prepared ownership/learning loop (2026-07-08)** - fast auto-scan no longer drops the artifact footer by profile alone; it omits footer OCR only when the capture option explicitly requests that or when the footer marker is absent. Parser tests cover noisy equipped names, split `Equipped:`/name footers, and one-letter OCR fragments that must stay `Not detected`. Scanner learning now persists text replacements, field aliases, constrained fixes, crop adjustment proposals, and UI-profile adjustment proposals instead of truncating everything back to text replacements. - **Visible-inventory merge guard (2026-07-09)** - the normal guided Auto-Scan start no longer falls back into `auto-entry` when the artifact detail card is missing. It now blocks and asks the operator to open the Artifact inventory with a visible detail card. The explicit `auto-entry`, `direct-inventory`, and `paimon-menu` Dev-Control modes remain available for targeted experiments, but they are not the merge-ready default path. - **Ownership live smoke (2026-07-09)** - live artifact detail capture parsed and stored an equipped footer as `equipped: "Citlali"` and the grey lock state as `locked: false`. A same-session visible-inventory run with `/scanner/start?entry=visible-inventory&limit=20&engine=current` completed `20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and `0` misses in `8047 ms` elapsed (`402 ms/artifact`). - **Locked artifact live proof (2026-07-09)** - a visibly locked artifact was selected through a read-only inventory tile click. Smart Capture reported `locked: true` with `lockSignal.ratio: 0.14797913950456323` over threshold `0.06`, and `/scanner/start?entry=visible-inventory&limit=1&engine=current` persisted the same artifact with `equipped: "Citlali"` and `locked: true`. Lock detection now decodes the lock crop PNG before measuring active lock pixels because Electron's native bitmap channel order was ambiguous in live captures. ## Remaining — needs the live environment or a UI pass These cannot be finished/validated without Genshin running at the user's resolution or without UI work best tested live: 1. **Validate/tune OCR preprocessing** on more real captures — confirm invert + threshold + upscale factor help (not hurt) actual Tesseract reads. The text-level eval harness cannot measure image preprocessing. 2. **Wire and benchmark native IK-traineddata OCR** against the same crop set. The current benchmark can use IK-traineddata through Tesseract.js; native Tesseract integration remains the next implementation step before any engine default changes. 3. **Validate explicit entry modes separately** from world, direct inventory, and Paimon/menu states with low limits only. These are now Dev-Control experiments, not the normal merge path; the normal Auto-Scan button blocks unless the visible artifact detail card is already present. 4. **Repeat locked=true on another page/session** if lock behavior changes. The first positive live proof passed on 2026-07-09, including store persistence. Further repeats are useful for confidence but no longer block the merge. 5. **3 artifacts/second iteration** - not reached yet. The next credible path is either native Tesseract/IK-traineddata integration that materially reduces substat OCR time, or a larger capture pipeline change that avoids full-frame PNG/Base64 transport without hurting safety checks. The target remains `<= 6667 ms` elapsed for 20 parsed artifacts with 0 misses and no silent OCR review regression. 6. **Broader scan soak test** — direct-GDI current-engine runs now passed at `20/20`, `45/45`, and `100/100` with 0 misses. Continue with repeat runs if duplicate rate needs tuning. 7. **Repeatability pass** — repeat the qualified current-vs-IK-traineddata run in a later live session before making major OCR-engine defaults or speed claims beyond this environment. Visible-page limits up to 20, scroll/page-transition limit 45, and the final 100-artifact current-vs-IK-traineddata comparison have passed for the current environment. Remaining soak work is repeatability, OCR corpus growth, equipped footer confirmation repeats, locked artifact repeats, and optional 3 artifacts/second speed work. ## Grow the eval corpus Every low-confidence review sample already stores its crops + OCR. Confirm/correct those via `reviewSampleToEvalCase` and commit them into `src/eval/corpus/` so the harness keeps measuring real-world accuracy across patches. See [ocr-eval.md](ocr-eval.md).