# Scanner IK Progress Report - 2026-07-07 This report summarizes the scanner/OCR work toward the current target: scan the first 100 artifacts with accuracy at least as good as Inventory Kamera and speed equal to or better than Inventory Kamera, without memory reads, hooks, injection, game-file modification, or unsafe in-game actions. ## Executive Summary The scanner has moved from a fragile OCR-first prototype toward an Inventory-Kamera-style artifact scanner: - Artifact scan is now the first-class path. - Auto-scan starts only after a validated artifact inventory/detail preflight. - Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and missing-detail states block before OCR/store/review work. - OCR uses a fast artifact profile that skips low-value fields and derives slot, set, and main-stat value through lookup constraints when safe. - The OCR worker pool, field crop split, page model, scroll model, and direct detail-change verification now mirror the relevant IK design choices more closely. - Diagnostics now preserve state evidence, timings, screenshots where useful, entry events, focus/input events, preflight failures, and scan-loop reasons. - A live soak runner now measures throughput and quality, compares current vs. IK-traineddata engines, and refuses to run against stale Electron builds. The current-vs-IK-traineddata comparison proof is now captured. On 2026-07-08, `npm run scan:goal:compare:validated` passed with `outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`. At `limit=100`, `current` won the qualified comparison with `100/100` parsed, `0` review, `0` misses, `378 ms/artifact` active average, and `37800 ms` projected time for 100 artifacts. `ik-traineddata` was not qualified at `limit=100` because it parsed `97/100`, had `5` review and `3` misses. The separate 3 artifacts/second target is still not proven. ## What Changed ### Lookup and validation - `scripts/generate-genshin-data.cjs` was extended into a stricter lookup package generator. - `src/lib/genshinLookup.ts` provides pure matching and validation for sets, pieces, slots, stats, characters, aliases, GOOD keys, source version, and validation summaries. - Auto-scan preflight blocks if the lookup package is invalid. Why this matters: IK succeeds partly because raw OCR is not trusted by itself. The app now follows the same principle: OCR text is normalized, matched, constrained, and derived against a canonical package before it is accepted. ### OCR and parser pipeline - Artifact detail crops are split into field-specific regions: name, slot, main-stat label, main-stat value, level, substats, set effects, equipped/footer, lock, and rarity. - Fast auto-scan profile skips lower-value OCR work: set effects, main-stat value crop, crop images, full-frame payloads, and inventory preview payloads. The equipped footer remains in real artifact-read captures when its marker is visible, because ownership now matters for phase 1 validation. Preflight and poll captures still skip OCR/crops/lock-state work. The slot crop remains in the fast path because it materially improved real-read quality. - Slot, set, and main-stat value are derived when lookup, slot rules, and level constraints make that safe. - Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used. - OCR crops are passed as PNG buffers internally instead of Base64 DataURLs. - Exact visual duplicate skipping is disabled in the hottest path; duplicate handling now primarily uses parsed artifact signatures so OCR is not skipped solely from a crop fingerprint collision. Why this matters: The fast path spends OCR only on fields that materially change the artifact identity or review decision. That is closer to IK's queued crop model than a manual-debug capture that OCRs every visible thing. ### Engine comparison and benchmark path - `/scanner/ocr/warmup?engine=current|ik-traineddata` warms OCR workers. - `/scanner/benchmark-ocr?engine=current|ik-traineddata|compare` benchmarks the same visible artifact crops. - Auto-scan accepts `ocrEngine: "current" | "ik-traineddata"`. - `scripts/live-soak.ps1` supports: - `npm run scan:goal` - `npm run scan:goal:current` - `npm run scan:goal:ik` - `npm run scan:goal:compare` - `npm run scan:goal:compare:validated` - `npm run scan:goal:compare:validated:wait` - `npm run scan:iterate:compare:validated` - `npm run scan:iterate:compare:validated:wait` - `npm run scan:live:preflight` - `npm run scan:live:preflight:wait` - `scan-performance-assessment.json` ranks runs by quality first and speed second, and records whether the 100-artifact result is a complete current-vs-IK comparison through `goal100Decision` and `goal100.comparisonComplete`. Important rule: A fast engine cannot win if it has too many misses or too much review. A qualified winner must finish cleanly, parse the requested count, keep miss rate at or below 2%, and keep review rate at or below 15%. ### Auto-scan entry and safety - The normal auto button runs a guided start: 1. focus Genshin, 2. run a lightweight no-OCR preflight, 3. if artifact detail is visible, use visible-inventory mode, 4. otherwise try direct `B -> artifact tab -> first artifact tile`, 5. if needed, fall back to the IK-style ESC/B inventory sequence, 6. start OCR only after artifact grid and detail card pass preflight. - Entry captures are state evidence only. They do not create review samples, store artifacts, or run artifact OCR before the detail preflight passes. - Entry waits now poll for state readiness instead of always sleeping the full fixed delay. - Scan loop also rechecks the same safety boundary after each click and scroll. Why this matters: The previous failure mode was dangerous from a product-quality point of view: when the game was not in artifact inventory, the scanner could still take screenshots and try to read artifacts. The current path is explicitly blocked outside the artifact inventory/detail state. ### Scan loop and speed - Grid model uses IK's 32-artifact visible page concept (`8 x 4`) instead of clicking the risky lower band. - Last/partial page planning bottom-aligns like IK, avoiding unnecessary duplicate reads after scroll. - Detail-change verification now uses the artifact OCR capture itself instead of a separate card-ready capture before OCR. - Scroll readiness uses inventory fingerprint polling: max 760 ms, 80 ms polls, changed pages may proceed after 100 ms. - Store/review writes are held out of the click/capture/OCR hot path. The latest source batches artifact store writes before the final summary instead of issuing one save/reload cycle per artifact. - Auto-scan artifact captures skip Electron source enumeration in the hot path and call the GDI capture helper directly. - Focus is done once at scan start; hot-loop captures do not refocus every tile. Why this matters: IK uses fixed waits around 200 ms after selecting inventory items and 100 ms after fast scrolls. The app now keeps those as safety ceilings/acceptance points while allowing earlier continuation when visual evidence is ready. ### Diagnostics and logging - Scanner diagnostics now capture timeline events for runtime, focus, keypress, entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid, counts, detail/page fingerprints, and failure reasons. - `/scanner/status` publishes recent diagnostic evidence. - Review sample output is compact by default so Vite does not watch large Base64 payloads during live soak runs. - The live runner writes timestamped JSON snapshots, CSV summaries, transcript, benchmark data, and performance assessment files under `outputs/live-soak/`. Why this matters: Future scanner bugs can be debugged from captured evidence instead of relying only on a human description of what appeared on screen. ## Vorgehensweise 1. Read the local Inventory Kamera reference under `work/Inventory_Kamera`. 2. Copy the proven concepts, not the entire implementation: 32 artifact targets per page, fixed coordinate ratios, queued OCR work, short item/scroll waits, read-only inventory navigation, and Tesseract traineddata comparison. 3. Harden the app's own architecture around those concepts: pure lookup API, parser derivation, renderer scan orchestration, Electron capture/OCR boundary, sidecar input helper, and diagnostics. 4. Add tests before trusting behavior: lookup validation, parser derivation, auto-entry planning/preflight, card-ready gates, page planning, scan-loop blocking, OCR eval corpus. 5. Add live tooling before claiming performance: bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON assessment, and quality-first comparison. ## Tests and Evidence Latest repo validation after the recent changes: | Check | Result | | --- | --- | | PowerShell parse for `scripts/live-soak.ps1` | Passed | | `npm run scan:assessment:test` | Passed | | `npm run lint` | Passed | | Focused scanner tests | Passed | | `npm test` | Passed, 171 tests | | OCR eval seed corpus | 100% exact match, 100% field accuracy, 100% critical fields | | `npm run build` | Passed | | `git diff --check` | Passed | Live evidence already collected: - Probe click changed artifact detail successfully. - Limit 2 live auto-scan completed with 2/2 parsed and 0 misses. - Limit 20 live soak completed on the first visible page. - Limit 45 live soak crossed into a scrolled page. - On 2026-07-08, `/scanner/start?entry=visible-inventory&limit=50&engine=current` completed `50/50` parsed and stored, `0` review, `0` duplicates, `0` misses, `2` pages, `61765 ms` elapsed, `1235 ms/artifact`, `averageCaptureMs: 186`, and `averageOcrMs: 162`. - A deferred single-write flush experiment also completed `50/50`, but regressed to `63616 ms` because `writeFlushMs` was `8163`; the source now uses batch persist instead, pending a fresh elevated live measurement. - After direct GDI hot-path optimization, a 20-artifact run completed `20/20` parsed, `19` stored, `0` review, `1` duplicate, `0` misses, `7966 ms` elapsed, `398 ms/artifact`, `averageCaptureMs: 193`, and `averageOcrMs: 167`. - A 45-artifact direct-GDI run completed `45/45` parsed, `42` stored, `0` review, `3` duplicates, `0` misses, `2` pages, `18625 ms` elapsed, `414 ms/artifact`, `averageCaptureMs: 187`, and `averageOcrMs: 162`. - A 100-artifact direct-GDI run on signature `2026-07-08-direct-gdi-hotpath` completed `100/100` parsed, `97` stored, `0` review, `3` duplicates, `0` misses, `4` pages, `42064 ms` elapsed, `421 ms/artifact`, `averageCaptureMs: 179`, and `averageOcrMs: 154`. - `npm run eval` passed after the speed work with `23/23` exact-match cases, `100%` field accuracy, and `100%` critical fields. - The 3 artifacts/second target is now prepared in code but not live-proven: artifact hot-path captures omit detail preview payloads, and stats expose capture roundtrip/overhead timing. A qualifying 20-artifact run must finish in `<= 6667 ms` with 0 misses and no silent OCR quality regression. - Follow-up 3/s attempts on 2026-07-08 fixed the false review trigger caused by omitted detail previews. The best clean `limit=20` run reached `7285 ms` (`364 ms/artifact`, about `2.75 artifacts/second`) with `20/20` parsed, `0` review, and `0` misses. The final stable run on `2026-07-08-direct-gdi-reviewfix` completed `20/20` with `0` review, `0` misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail-region capture, 5 OCR workers, DataURL buffer decode, and substat `PSM.SINGLE_COLUMN` were tested and rejected as slower than the direct-GDI baseline. - Final current-vs-IK-traineddata comparison on 2026-07-08: `npm run scan:goal:compare:validated` passed. Evidence file: `outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`, `createdAt: 2026-07-08T18:41:11.6120957+02:00`. `goal100Decision` was `qualified-comparison: winner=current`; `goal100.comparisonComplete` was true. `current` completed `100/100` parsed, `97` stored, `0` review, `0` misses, `4` pages, `378 ms/artifact` active average. `ik-traineddata` completed `97/100` parsed, `92` stored, `5` review, `3` misses and was rejected for parsing fewer artifacts than requested. - The review queue now has a bounded corpus-growth workflow: `npm run eval:review-candidates` writes a deduplicated, Git-ignored worklist to `outputs/review-eval-candidates/`. This separates complete modern OCR samples from stale captures and prevents the parser's own guess from being promoted to ground truth without human confirmation. After manual checking, `npm run eval:prepare-confirmed` turns one candidate plus explicit expected labels into a paste-ready confirmed corpus snippet. Current live limitation: - The fast path and current-vs-IK-traineddata 100-artifact comparison are proven in the current live environment. 3 artifacts/second is not proven; remaining speed work needs a larger OCR or capture-pipeline change, not more click tuning. ## Inventory Kamera Comparison | Area | Inventory Kamera | Current app status | | --- | --- | --- | | Safe scope | Reads inventory through screen/click automation | Same safety boundary: screen capture and read-only input only | | Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards | | Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented | | Last page | Bottom-aligned partial page after scroll | Implemented in page planner | | Item wait | About 200 ms fixed wait | No separate wait capture; OCR capture verifies changed detail | | Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms | | OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path | | Capture hot path | Direct window/screen capture without source-list scan per item | Direct GDI capture in auto-scan artifact loop | | Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints | | Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added | | Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle | | 100-artifact proof | Reference target | Qualified current-vs-IK comparison captured; `current` won with `100/100`, 0 review, 0 misses, 37.8s projected | What is theoretically better than before: - The app no longer spends OCR on invalid screens. - It no longer treats click count as scanner success. - It can prove whether `current` or `ik-traineddata` wins on the same capture set instead of changing engines blindly. - It can reject fast-but-wrong results automatically. - It can identify whether the bottleneck is OCR, capture, card readiness, or scroll readiness. What is not yet proven better than IK: - Native Tesseract speed is not integrated as the default. - 3 artifacts/second is not proven. - Native Inventory Kamera outside this app was not re-run in the same session; the completed comparison is against the bundled `ik-traineddata` scan engine. ## Theoretical Runtime Flow For short iteration while tuning: 1. Start current elevated app with `npm run dev:admin` and confirm UAC. 2. Run `npm run scan:iterate:compare:validated:wait` from a visible artifact inventory when starting directly after UAC, or `npm run scan:iterate:compare:validated` if preflight already passes. 3. Inspect `scan-performance-assessment.json`, review samples, and timings if the 20-artifact comparison fails quality gates. For the intended 100-artifact comparison: 1. Start current elevated app with `npm run dev:admin` and confirm UAC. 2. Verify `/health.appBuild.signature` matches `electron/main.ts`. 3. Warm current and IK-traineddata OCR workers. 4. Run a small bounded probe from the artifact inventory. 5. Run `npm run scan:goal:compare:validated:wait` directly after UAC, or `npm run scan:goal:compare:validated` if preflight already passes. 6. For each engine and limit (`2, 5, 20, 45, 100`): - focus Genshin once, - verify lookup and layout, - verify artifact grid and detail card, - click one safe grid target, - poll detail fingerprint, - skip duplicate visuals, - OCR only the fast artifact crop set, - parse through lookup constraints, - queue store/review writes, - scroll with inventory fingerprint polling, - stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or repeated misses. 7. Write CSV, JSON snapshots, transcript, benchmark report, and performance assessment. 8. Declare a winner only if the 100-artifact run is qualified by quality. Expected bottleneck sequence: - If OCR dominates, compare `current` vs `ik-traineddata`, crop count, and worker pool size. - If capture dominates, reduce payload construction and preview/crop image work. - If card-ready dominates, tune the detail fingerprint gate. - If scroll-ready dominates, tune page fingerprint polling and scroll notches. ## What Is Better Than Before - Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or menu screens. - Paimon/menu detection blocks before scan-loop OCR or writes. - The normal button is one coherent guided flow instead of a separate "get to inventory first, then scan" workflow. - The app has a real lookup layer instead of raw OCR plus scattered hardcoded assumptions. - The scanner can compare OCR engines without changing the default blindly. - Performance reports now include quality decisions, not just elapsed time. - Diagnostics are concrete enough for later self-troubleshooting. - Stale elevated runtime is detected before live soak, avoiding false evidence. ## Risks and Remaining Work 1. Keep `current` as the default OCR engine for now; it won the qualified current-vs-IK-traineddata live comparison. 2. If pursuing 3 artifacts/second, focus on capture/OCR pipeline changes rather than click timing. 3. Grow the eval corpus with confirmed real review samples before tightening parser thresholds further. 4. Validate a positive locked-artifact sample. 5. Keep recommendations secondary until scanner quality remains stable across repeated live sessions. ## Definition of Done for the IK Target The goal is complete only when current evidence proves all of these: - The app is running the latest runtime signature. - The 100-artifact scan finishes cleanly. - Parsed count is at least 100. - Miss rate is at or below 2%. - Review rate is at or below 15%. - The run is equal to or faster than the recorded IK reference or the selected IK-traineddata/native baseline on the same machine and inventory setup. - The evidence bundle is saved under `outputs/live-soak/`. - Any chosen default OCR engine is backed by the same-capture benchmark.