genshin-assistant/docs/scanner-ik-progress-report.md

Scanner IK Progress Report - 2026-07-07

This report summarizes the scanner/OCR work toward the current target: scan the first 100 artifacts with accuracy at least as good as Inventory Kamera and speed equal to or better than Inventory Kamera, without memory reads, hooks, injection, game-file modification, or unsafe in-game actions.

Executive Summary

The scanner has moved from a fragile OCR-first prototype toward an Inventory-Kamera-style artifact scanner:

• Artifact scan is now the first-class path.
• Auto-scan starts only after a validated artifact inventory/detail preflight.
• Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and missing-detail states block before OCR/store/review work.
• OCR uses a fast artifact profile that skips low-value fields and derives slot, set, and main-stat value through lookup constraints when safe.
• The OCR worker pool, field crop split, page model, scroll model, and direct detail-change verification now mirror the relevant IK design choices more closely.
• Diagnostics now preserve state evidence, timings, screenshots where useful, entry events, focus/input events, preflight failures, and scan-loop reasons.
• A live soak runner now measures throughput and quality, compares current vs. IK-traineddata engines, and refuses to run against stale Electron builds.

The current-vs-IK-traineddata comparison proof is now captured. On 2026-07-08, npm run scan:goal:compare:validated passed with outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json. At limit=100, current won the qualified comparison with 100/100 parsed, 0 review, 0 misses, 378 ms/artifact active average, and 37800 ms projected time for 100 artifacts. ik-traineddata was not qualified at limit=100 because it parsed 97/100, had 5 review and 3 misses. The separate 3 artifacts/second target is still not proven.

What Changed

Lookup and validation

• scripts/generate-genshin-data.cjs was extended into a stricter lookup package generator.
• src/lib/genshinLookup.ts provides pure matching and validation for sets, pieces, slots, stats, characters, aliases, GOOD keys, source version, and validation summaries.
• Auto-scan preflight blocks if the lookup package is invalid.

Why this matters:

IK succeeds partly because raw OCR is not trusted by itself. The app now follows the same principle: OCR text is normalized, matched, constrained, and derived against a canonical package before it is accepted.

OCR and parser pipeline

• Artifact detail crops are split into field-specific regions: name, slot, main-stat label, main-stat value, level, substats, set effects, equipped/footer, lock, and rarity.
• Fast auto-scan profile skips lower-value OCR work: set effects, main-stat value crop, crop images, full-frame payloads, and inventory preview payloads. The equipped footer remains in real artifact-read captures when its marker is visible, because ownership now matters for phase 1 validation. Preflight and poll captures still skip OCR/crops/lock-state work. The slot crop remains in the fast path because it materially improved real-read quality.
• Slot, set, and main-stat value are derived when lookup, slot rules, and level constraints make that safe.
• Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
• OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
• Exact visual duplicate skipping is disabled in the hottest path; duplicate handling now primarily uses parsed artifact signatures so OCR is not skipped solely from a crop fingerprint collision.

Why this matters:

The fast path spends OCR only on fields that materially change the artifact identity or review decision. That is closer to IK's queued crop model than a manual-debug capture that OCRs every visible thing.

Engine comparison and benchmark path

• /scanner/ocr/warmup?engine=current|ik-traineddata warms OCR workers.
• /scanner/benchmark-ocr?engine=current|ik-traineddata|compare benchmarks the same visible artifact crops.
• Auto-scan accepts ocrEngine: "current" | "ik-traineddata".
• scripts/live-soak.ps1 supports:
  • npm run scan:goal
  • npm run scan:goal:current
  • npm run scan:goal:ik
  • npm run scan:goal:compare
  • npm run scan:goal:compare:validated
  • npm run scan:goal:compare:validated:wait
  • npm run scan:iterate:compare:validated
  • npm run scan:iterate:compare:validated:wait
  • npm run scan:live:preflight
  • npm run scan:live:preflight:wait
• scan-performance-assessment.json ranks runs by quality first and speed second, and records whether the 100-artifact result is a complete current-vs-IK comparison through goal100Decision and goal100.comparisonComplete.

Important rule:

A fast engine cannot win if it has too many misses or too much review. A qualified winner must finish cleanly, parse the requested count, keep miss rate at or below 2%, and keep review rate at or below 15%.

Auto-scan entry and safety

• The normal auto button runs a guided start:
  1. focus Genshin,
  2. run a lightweight no-OCR preflight,
  3. if artifact detail is visible, use visible-inventory mode,
  4. otherwise try direct B -> artifact tab -> first artifact tile,
  5. if needed, fall back to the IK-style ESC/B inventory sequence,
  6. start OCR only after artifact grid and detail card pass preflight.
• Entry captures are state evidence only. They do not create review samples, store artifacts, or run artifact OCR before the detail preflight passes.
• Entry waits now poll for state readiness instead of always sleeping the full fixed delay.
• Scan loop also rechecks the same safety boundary after each click and scroll.

Why this matters:

The previous failure mode was dangerous from a product-quality point of view: when the game was not in artifact inventory, the scanner could still take screenshots and try to read artifacts. The current path is explicitly blocked outside the artifact inventory/detail state.

Scan loop and speed

• Grid model uses IK's 32-artifact visible page concept (8 x 4) instead of clicking the risky lower band.
• Last/partial page planning bottom-aligns like IK, avoiding unnecessary duplicate reads after scroll.
• Detail-change verification now uses the artifact OCR capture itself instead of a separate card-ready capture before OCR.
• Scroll readiness uses inventory fingerprint polling: max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
• Store/review writes are held out of the click/capture/OCR hot path. The latest source batches artifact store writes before the final summary instead of issuing one save/reload cycle per artifact.
• Auto-scan artifact captures skip Electron source enumeration in the hot path and call the GDI capture helper directly.
• Focus is done once at scan start; hot-loop captures do not refocus every tile.

Why this matters:

IK uses fixed waits around 200 ms after selecting inventory items and 100 ms after fast scrolls. The app now keeps those as safety ceilings/acceptance points while allowing earlier continuation when visual evidence is ready.

Diagnostics and logging

• Scanner diagnostics now capture timeline events for runtime, focus, keypress, entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid, counts, detail/page fingerprints, and failure reasons.
• /scanner/status publishes recent diagnostic evidence.
• Review sample output is compact by default so Vite does not watch large Base64 payloads during live soak runs.
• The live runner writes timestamped JSON snapshots, CSV summaries, transcript, benchmark data, and performance assessment files under outputs/live-soak/.

Why this matters:

Future scanner bugs can be debugged from captured evidence instead of relying only on a human description of what appeared on screen.

Vorgehensweise

1. Read the local Inventory Kamera reference under work/Inventory_Kamera.
2. Copy the proven concepts, not the entire implementation: 32 artifact targets per page, fixed coordinate ratios, queued OCR work, short item/scroll waits, read-only inventory navigation, and Tesseract traineddata comparison.
3. Harden the app's own architecture around those concepts: pure lookup API, parser derivation, renderer scan orchestration, Electron capture/OCR boundary, sidecar input helper, and diagnostics.
4. Add tests before trusting behavior: lookup validation, parser derivation, auto-entry planning/preflight, card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
5. Add live tooling before claiming performance: bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON assessment, and quality-first comparison.

Tests and Evidence

Latest repo validation after the recent changes:

Check	Result
PowerShell parse for scripts/live-soak.ps1	Passed
npm run scan:assessment:test	Passed
npm run lint	Passed
Focused scanner tests	Passed
npm test	Passed, 171 tests
OCR eval seed corpus	100% exact match, 100% field accuracy, 100% critical fields
npm run build	Passed
git diff --check	Passed

Live evidence already collected:

• Probe click changed artifact detail successfully.
• Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
• Limit 20 live soak completed on the first visible page.
• Limit 45 live soak crossed into a scrolled page.
• On 2026-07-08, /scanner/start?entry=visible-inventory&limit=50&engine=current completed 50/50 parsed and stored, 0 review, 0 duplicates, 0 misses, 2 pages, 61765 ms elapsed, 1235 ms/artifact, averageCaptureMs: 186, and averageOcrMs: 162.
• A deferred single-write flush experiment also completed 50/50, but regressed to 63616 ms because writeFlushMs was 8163; the source now uses batch persist instead, pending a fresh elevated live measurement.
• After direct GDI hot-path optimization, a 20-artifact run completed 20/20 parsed, 19 stored, 0 review, 1 duplicate, 0 misses, 7966 ms elapsed, 398 ms/artifact, averageCaptureMs: 193, and averageOcrMs: 167.
• A 45-artifact direct-GDI run completed 45/45 parsed, 42 stored, 0 review, 3 duplicates, 0 misses, 2 pages, 18625 ms elapsed, 414 ms/artifact, averageCaptureMs: 187, and averageOcrMs: 162.
• A 100-artifact direct-GDI run on signature 2026-07-08-direct-gdi-hotpath completed 100/100 parsed, 97 stored, 0 review, 3 duplicates, 0 misses, 4 pages, 42064 ms elapsed, 421 ms/artifact, averageCaptureMs: 179, and averageOcrMs: 154.
• npm run eval passed after the speed work with 23/23 exact-match cases, 100% field accuracy, and 100% critical fields.
• The 3 artifacts/second target is now prepared in code but not live-proven: artifact hot-path captures omit detail preview payloads, and stats expose capture roundtrip/overhead timing. A qualifying 20-artifact run must finish in <= 6667 ms with 0 misses and no silent OCR quality regression.
• Follow-up 3/s attempts on 2026-07-08 fixed the false review trigger caused by omitted detail previews. The best clean limit=20 run reached 7285 ms (364 ms/artifact, about 2.75 artifacts/second) with 20/20 parsed, 0 review, and 0 misses. The final stable run on 2026-07-08-direct-gdi-reviewfix completed 20/20 with 0 review, 0 misses, and 7973 ms elapsed (399 ms/artifact). Detail-region capture, 5 OCR workers, DataURL buffer decode, and substat PSM.SINGLE_COLUMN were tested and rejected as slower than the direct-GDI baseline.
• Final current-vs-IK-traineddata comparison on 2026-07-08: npm run scan:goal:compare:validated passed. Evidence file: outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json, createdAt: 2026-07-08T18:41:11.6120957+02:00. goal100Decision was qualified-comparison: winner=current; goal100.comparisonComplete was true. current completed 100/100 parsed, 97 stored, 0 review, 0 misses, 4 pages, 378 ms/artifact active average. ik-traineddata completed 97/100 parsed, 92 stored, 5 review, 3 misses and was rejected for parsing fewer artifacts than requested.
• The review queue now has a bounded corpus-growth workflow: npm run eval:review-candidates writes a deduplicated, Git-ignored worklist to outputs/review-eval-candidates/. This separates complete modern OCR samples from stale captures and prevents the parser's own guess from being promoted to ground truth without human confirmation. After manual checking, npm run eval:prepare-confirmed turns one candidate plus explicit expected labels into a paste-ready confirmed corpus snippet.

Current live limitation:

• The fast path and current-vs-IK-traineddata 100-artifact comparison are proven in the current live environment. 3 artifacts/second is not proven; remaining speed work needs a larger OCR or capture-pipeline change, not more click tuning.
• The visible-inventory ownership/lock extension is also live-proven in this environment: equipped footer reads for Citlali and Linnea, unlocked locked: false, positive locked locked: true, and locked/equipped persistence all passed on 2026-07-09.

Inventory Kamera Comparison

Area	Inventory Kamera	Current app status
Safe scope	Reads inventory through screen/click automation	Same safety boundary: screen capture and read-only input only
Entry	ESC/B inventory navigation and tab click	Direct B path plus IK-style fallback, with preflight guards
Page model	32 artifact items per page	32 safe targets (8 x 4) implemented
Last page	Bottom-aligned partial page after scroll	Implemented in page planner
Item wait	About 200 ms fixed wait	No separate wait capture; OCR capture verifies changed detail
Scroll wait	About 100 ms fast wait after scroll	Fingerprint polling, accepts changed page after 100 ms
OCR model	Native Tesseract worker queue and custom traineddata	Tesseract.js pool with current and IK-traineddata comparison path
Capture hot path	Direct window/screen capture without source-list scan per item	Direct GDI capture in auto-scan artifact loop
Field parsing	OCR plus game-data lookup	OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints
Quality gate	Mature behavior by design and user history	Explicit benchmark/soak quality gates added
Diagnostics	Logs/screenshots in IK flow	Diagnostics timeline plus JSON evidence bundle
100-artifact proof	Reference target	Qualified current-vs-IK comparison captured; current won with 100/100, 0 review, 0 misses, 37.8s projected

What is theoretically better than before:

• The app no longer spends OCR on invalid screens.
• It no longer treats click count as scanner success.
• It can prove whether current or ik-traineddata wins on the same capture set instead of changing engines blindly.
• It can reject fast-but-wrong results automatically.
• It can identify whether the bottleneck is OCR, capture, card readiness, or scroll readiness.

What is not yet proven better than IK:

• Native Tesseract speed is not integrated as the default.
• 3 artifacts/second is not proven.
• Native Inventory Kamera outside this app was not re-run in the same session; the completed comparison is against the bundled ik-traineddata scan engine.

Theoretical Runtime Flow

For short iteration while tuning:

1. Start current elevated app with npm run dev:admin and confirm UAC.
2. Run npm run scan:iterate:compare:validated:wait from a visible artifact inventory when starting directly after UAC, or npm run scan:iterate:compare:validated if preflight already passes.
3. Inspect scan-performance-assessment.json, review samples, and timings if the 20-artifact comparison fails quality gates.

For the intended 100-artifact comparison:

1. Start current elevated app with npm run dev:admin and confirm UAC.
2. Verify /health.appBuild.signature matches electron/main.ts.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run npm run scan:goal:compare:validated:wait directly after UAC, or npm run scan:goal:compare:validated if preflight already passes.
6. For each engine and limit (2, 5, 20, 45, 100):
   • focus Genshin once,
   • verify lookup and layout,
   • verify artifact grid and detail card,
   • click one safe grid target,
   • poll detail fingerprint,
   • skip duplicate visuals,
   • OCR only the fast artifact crop set,
   • parse through lookup constraints,
   • queue store/review writes,
   • scroll with inventory fingerprint polling,
   • stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or repeated misses.
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance assessment.
8. Declare a winner only if the 100-artifact run is qualified by quality.

Expected bottleneck sequence:

• If OCR dominates, compare current vs ik-traineddata, crop count, and worker pool size.
• If capture dominates, reduce payload construction and preview/crop image work.
• If card-ready dominates, tune the detail fingerprint gate.
• If scroll-ready dominates, tune page fingerprint polling and scroll notches.

What Is Better Than Before

• Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or menu screens.
• Paimon/menu detection blocks before scan-loop OCR or writes.
• The normal button is one coherent guided flow instead of a separate "get to inventory first, then scan" workflow.
• The app has a real lookup layer instead of raw OCR plus scattered hardcoded assumptions.
• The scanner can compare OCR engines without changing the default blindly.
• Performance reports now include quality decisions, not just elapsed time.
• Diagnostics are concrete enough for later self-troubleshooting.
• Stale elevated runtime is detected before live soak, avoiding false evidence.

Risks and Remaining Work

1. Keep current as the default OCR engine for now; it won the qualified current-vs-IK-traineddata live comparison.
2. If pursuing 3 artifacts/second, focus on capture/OCR pipeline changes rather than click timing.
3. Grow the eval corpus with confirmed real review samples before tightening parser thresholds further.
4. Repeat equipped/locked live samples in later sessions if confidence or UI behavior changes, but the first positive locked proof has passed.
5. Keep recommendations secondary until scanner quality remains stable across repeated live sessions and the result rail, artifact inventory, and detail evaluation flow are implemented.
6. Treat current speed as acceptable for the next product phase; prioritize artifact content extraction and review-safe value scoring before another broad speed pass.

Definition of Done for the IK Target

The goal is complete only when current evidence proves all of these:

• The app is running the latest runtime signature.
• The 100-artifact scan finishes cleanly.
• Parsed count is at least 100.
• Miss rate is at or below 2%.
• Review rate is at or below 15%.
• The run is equal to or faster than the recorded IK reference or the selected IK-traineddata/native baseline on the same machine and inventory setup.
• The evidence bundle is saved under outputs/live-soak/.
• Any chosen default OCR engine is backed by the same-capture benchmark.