genshin-assistant/docs/scanner-ik-progress-report.md

Scanner IK Progress Report - 2026-07-07

This report summarizes the scanner/OCR work toward the current target: scan the first 100 artifacts with accuracy at least as good as Inventory Kamera and speed equal to or better than Inventory Kamera, without memory reads, hooks, injection, game-file modification, or unsafe in-game actions.

Executive Summary

The scanner has moved from a fragile OCR-first prototype toward an Inventory-Kamera-style artifact scanner:

• Artifact scan is now the first-class path.
• Auto-scan starts only after a validated artifact inventory/detail preflight.
• Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and missing-detail states block before OCR/store/review work.
• OCR uses a fast artifact profile that skips low-value fields and derives slot, set, and main-stat value through lookup constraints when safe.
• The OCR worker pool, field crop split, page model, scroll model, and readiness waits now mirror the relevant IK design choices more closely.
• Diagnostics now preserve state evidence, timings, screenshots where useful, entry events, focus/input events, preflight failures, and scan-loop reasons.
• A live soak runner now measures throughput and quality, compares current vs. IK-traineddata engines, and refuses to run against stale Electron builds.

The requested final goal is not proven complete yet. The current live dev port is still owned by an older elevated Electron instance, so the latest code cannot be truthfully benchmarked against IK until the app is restarted with UAC approval and npm run scan:goal:compare completes a qualified 100-artifact run.

What Changed

Lookup and validation

• scripts/generate-genshin-data.cjs was extended into a stricter lookup package generator.
• src/lib/genshinLookup.ts provides pure matching and validation for sets, pieces, slots, stats, characters, aliases, GOOD keys, source version, and validation summaries.
• Auto-scan preflight blocks if the lookup package is invalid.

Why this matters:

IK succeeds partly because raw OCR is not trusted by itself. The app now follows the same principle: OCR text is normalized, matched, constrained, and derived against a canonical package before it is accepted.

OCR and parser pipeline

• Artifact detail crops are split into field-specific regions: name, slot, main-stat label, main-stat value, level, substats, set effects, equipped/footer, lock, and rarity.
• Fast auto-scan profile skips lower-value OCR work: set effects, slot crop, main-stat value crop, equipped footer, crop images, full-frame payloads, and inventory preview payloads.
• Slot, set, and main-stat value are derived when lookup, slot rules, and level constraints make that safe.
• Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
• OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
• Exact visual duplicates are skipped before OCR.

Why this matters:

The fast path spends OCR only on fields that materially change the artifact identity or review decision. That is closer to IK's queued crop model than a manual-debug capture that OCRs every visible thing.

Engine comparison and benchmark path

• /scanner/ocr/warmup?engine=current|ik-traineddata warms OCR workers.
• /scanner/benchmark-ocr?engine=current|ik-traineddata|compare benchmarks the same visible artifact crops.
• Auto-scan accepts ocrEngine: "current" | "ik-traineddata".
• scripts/live-soak.ps1 supports:
  • npm run scan:goal
  • npm run scan:goal:current
  • npm run scan:goal:ik
  • npm run scan:goal:compare
• scan-performance-assessment.json ranks runs by quality first and speed second.

Important rule:

A fast engine cannot win if it has too many misses or too much review. A qualified winner must finish cleanly, parse the requested count, keep miss rate at or below 2%, and keep review rate at or below 15%.

Auto-scan entry and safety

• The normal auto button runs a guided start:
  1. focus Genshin,
  2. run a lightweight no-OCR preflight,
  3. if artifact detail is visible, use visible-inventory mode,
  4. otherwise try direct B -> artifact tab -> first artifact tile,
  5. if needed, fall back to the IK-style ESC/B inventory sequence,
  6. start OCR only after artifact grid and detail card pass preflight.
• Entry captures are state evidence only. They do not create review samples, store artifacts, or run artifact OCR before the detail preflight passes.
• Entry waits now poll for state readiness instead of always sleeping the full fixed delay.
• Scan loop also rechecks the same safety boundary after each click and scroll.

Why this matters:

The previous failure mode was dangerous from a product-quality point of view: when the game was not in artifact inventory, the scanner could still take screenshots and try to read artifacts. The current path is explicitly blocked outside the artifact inventory/detail state.

Scan loop and speed

• Grid model uses IK's 32-artifact visible page concept (8 x 4) instead of clicking the risky lower band.
• Last/partial page planning bottom-aligns like IK, avoiding unnecessary duplicate reads after scroll.
• Card readiness uses detail fingerprint polling: max 420 ms, 60 ms polls, changed cards may proceed after 200 ms.
• Scroll readiness uses inventory fingerprint polling: max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
• Store/review writes are queued so the next tile can be clicked before disk writes finish. The queue is still flushed before final summary.
• Focus is done once at scan start; hot-loop captures do not refocus every tile.

Why this matters:

IK uses fixed waits around 200 ms after selecting inventory items and 100 ms after fast scrolls. The app now keeps those as safety ceilings/acceptance points while allowing earlier continuation when visual evidence is ready.

Diagnostics and logging

• Scanner diagnostics now capture timeline events for runtime, focus, keypress, entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid, counts, detail/page fingerprints, and failure reasons.
• /scanner/status publishes recent diagnostic evidence.
• Review sample output is compact by default so Vite does not watch large Base64 payloads during live soak runs.
• The live runner writes timestamped JSON snapshots, CSV summaries, transcript, benchmark data, and performance assessment files under outputs/live-soak/.

Why this matters:

Future scanner bugs can be debugged from captured evidence instead of relying only on a human description of what appeared on screen.

Vorgehensweise

1. Read the local Inventory Kamera reference under work/Inventory_Kamera.
2. Copy the proven concepts, not the entire implementation: 32 artifact targets per page, fixed coordinate ratios, queued OCR work, short item/scroll waits, read-only inventory navigation, and Tesseract traineddata comparison.
3. Harden the app's own architecture around those concepts: pure lookup API, parser derivation, renderer scan orchestration, Electron capture/OCR boundary, sidecar input helper, and diagnostics.
4. Add tests before trusting behavior: lookup validation, parser derivation, auto-entry planning/preflight, card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
5. Add live tooling before claiming performance: bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON assessment, and quality-first comparison.

Tests and Evidence

Latest repo validation after the recent changes:

Check	Result
PowerShell parse for scripts/live-soak.ps1	Passed
npm run scan:assessment:test	Passed
npm run lint	Passed
Focused scanner tests	Passed
npm test	Passed, 171 tests
OCR eval seed corpus	100% exact match, 100% field accuracy, 100% critical fields
npm run build	Passed
git diff --check	Passed

Live evidence already collected earlier on 2026-07-07:

• Probe click changed artifact detail successfully.
• Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
• Limit 20 live soak completed on the first visible page.
• Limit 45 live soak crossed into a scrolled page.

Current live limitation:

• /health still reports an older elevated build: 2026-07-07-ocr-pool4-hotloop-no-refocus.
• Current source expects: 2026-07-07-ik32-fastsubstats-active-timing.
• The live soak runner correctly refuses to benchmark the stale runtime.
• A UAC restart attempt was canceled, so the latest code is not yet live.

Inventory Kamera Comparison

Area	Inventory Kamera	Current app status
Safe scope	Reads inventory through screen/click automation	Same safety boundary: screen capture and read-only input only
Entry	ESC/B inventory navigation and tab click	Direct B path plus IK-style fallback, with preflight guards
Page model	32 artifact items per page	32 safe targets (8 x 4) implemented
Last page	Bottom-aligned partial page after scroll	Implemented in page planner
Item wait	About 200 ms fixed wait	Fingerprint polling, accepts changed card after 200 ms
Scroll wait	About 100 ms fast wait after scroll	Fingerprint polling, accepts changed page after 100 ms
OCR model	Native Tesseract worker queue and custom traineddata	Tesseract.js pool with current and IK-traineddata comparison path
Field parsing	OCR plus game-data lookup	OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints
Quality gate	Mature behavior by design and user history	Explicit benchmark/soak quality gates added
Diagnostics	Logs/screenshots in IK flow	Diagnostics timeline plus JSON evidence bundle
100-artifact proof	Reference target	Not yet proven on latest app build

What is theoretically better than before:

• The app no longer spends OCR on invalid screens.
• It no longer treats click count as scanner success.
• It can prove whether current or ik-traineddata wins on the same capture set instead of changing engines blindly.
• It can reject fast-but-wrong results automatically.
• It can identify whether the bottleneck is OCR, capture, card readiness, or scroll readiness.

What is not yet proven better than IK:

• Native Tesseract speed is not integrated as the default.
• The latest code has not completed the 100-artifact live run.
• Review rate and miss rate on the user's real inventory still need the new live report.

Theoretical Runtime Flow

For the intended 100-artifact comparison:

1. Start current elevated app with npm run dev:admin and confirm UAC.
2. Verify /health.appBuild.signature matches electron/main.ts.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run npm run scan:goal:compare.
6. For each engine and limit (2, 5, 20, 45, 100):
   • focus Genshin once,
   • verify lookup and layout,
   • verify artifact grid and detail card,
   • click one safe grid target,
   • poll detail fingerprint,
   • skip duplicate visuals,
   • OCR only the fast artifact crop set,
   • parse through lookup constraints,
   • queue store/review writes,
   • scroll with inventory fingerprint polling,
   • stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or repeated misses.
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance assessment.
8. Declare a winner only if the 100-artifact run is qualified by quality.

Expected bottleneck sequence:

• If OCR dominates, compare current vs ik-traineddata, crop count, and worker pool size.
• If capture dominates, reduce payload construction and preview/crop image work.
• If card-ready dominates, tune the detail fingerprint gate.
• If scroll-ready dominates, tune page fingerprint polling and scroll notches.

What Is Better Than Before

• Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or menu screens.
• Paimon/menu detection blocks before scan-loop OCR or writes.
• The normal button is one coherent guided flow instead of a separate "get to inventory first, then scan" workflow.
• The app has a real lookup layer instead of raw OCR plus scattered hardcoded assumptions.
• The scanner can compare OCR engines without changing the default blindly.
• Performance reports now include quality decisions, not just elapsed time.
• Diagnostics are concrete enough for later self-troubleshooting.
• Stale elevated runtime is detected before live soak, avoiding false evidence.

Risks and Remaining Work

1. Restart with UAC and run the latest build live.
2. Run npm run scan:goal:compare from a visible artifact inventory.
3. If the 100-artifact winner is not qualified, inspect: scan-performance-assessment.json, review samples, diagnostic timeline, and field timings.
4. If ik-traineddata wins but Tesseract.js is still slow, evaluate native Tesseract integration.
5. Grow the eval corpus with confirmed real review samples before tightening parser thresholds further.
6. Validate a positive locked-artifact sample.
7. Keep recommendations secondary until scanner quality is proven.

Definition of Done for the IK Target

The goal is complete only when current evidence proves all of these:

• The app is running the latest runtime signature.
• The 100-artifact scan finishes cleanly.
• Parsed count is at least 100.
• Miss rate is at or below 2%.
• Review rate is at or below 15%.
• The run is equal to or faster than the recorded IK reference or the selected IK-traineddata/native baseline on the same machine and inventory setup.
• The evidence bundle is saved under outputs/live-soak/.
• Any chosen default OCR engine is backed by the same-capture benchmark.