genshin-assistant/docs/scanner-ik-progress-report.md

# Scanner IK Progress Report - 2026-07-07

This report summarizes the scanner/OCR work toward the current target:
scan the first 100 artifacts with accuracy at least as good as Inventory Kamera
and speed equal to or better than Inventory Kamera, without memory reads, hooks,
injection, game-file modification, or unsafe in-game actions.

## Executive Summary

The scanner has moved from a fragile OCR-first prototype toward an
Inventory-Kamera-style artifact scanner:

- Artifact scan is now the first-class path.
- Auto-scan starts only after a validated artifact inventory/detail preflight.
- Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and
  missing-detail states block before OCR/store/review work.
- OCR uses a fast artifact profile that skips low-value fields and derives
  slot, set, and main-stat value through lookup constraints when safe.
- The OCR worker pool, field crop split, page model, scroll model, and direct
  detail-change verification now mirror the relevant IK design choices more
  closely.
- Diagnostics now preserve state evidence, timings, screenshots where useful,
  entry events, focus/input events, preflight failures, and scan-loop reasons.
- A live soak runner now measures throughput and quality, compares current vs.
  IK-traineddata engines, and refuses to run against stale Electron builds.

The current-vs-IK-traineddata comparison proof is now captured. On 2026-07-08,
`npm run scan:goal:compare:validated` passed with
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`.
At `limit=100`, `current` won the qualified comparison with `100/100` parsed,
`0` review, `0` misses, `378 ms/artifact` active average, and `37800 ms`
projected time for 100 artifacts. `ik-traineddata` was not qualified at
`limit=100` because it parsed `97/100`, had `5` review and `3` misses. The
separate 3 artifacts/second target is still not proven.

## What Changed

### Lookup and validation

- `scripts/generate-genshin-data.cjs` was extended into a stricter lookup
  package generator.
- `src/lib/genshinLookup.ts` provides pure matching and validation for sets,
  pieces, slots, stats, characters, aliases, GOOD keys, source version, and
  validation summaries.
- Auto-scan preflight blocks if the lookup package is invalid.

Why this matters:

IK succeeds partly because raw OCR is not trusted by itself. The app now follows
the same principle: OCR text is normalized, matched, constrained, and derived
against a canonical package before it is accepted.

### OCR and parser pipeline

- Artifact detail crops are split into field-specific regions:
  name, slot, main-stat label, main-stat value, level, substats, set effects,
  equipped/footer, lock, and rarity.
- Fast auto-scan profile skips lower-value OCR work:
  set effects, main-stat value crop, crop images, full-frame payloads, and
  inventory preview payloads. The equipped footer remains in real artifact-read
  captures when its marker is visible, because ownership now matters for phase 1
  validation. Preflight and poll captures still skip OCR/crops/lock-state work.
  The slot crop remains in the fast path because it materially improved
  real-read quality.
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
  constraints make that safe.
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
- Exact visual duplicate skipping is disabled in the hottest path; duplicate
  handling now primarily uses parsed artifact signatures so OCR is not skipped
  solely from a crop fingerprint collision.

Why this matters:

The fast path spends OCR only on fields that materially change the artifact
identity or review decision. That is closer to IK's queued crop model than a
manual-debug capture that OCRs every visible thing.

### Engine comparison and benchmark path

- `/scanner/ocr/warmup?engine=current|ik-traineddata` warms OCR workers.
- `/scanner/benchmark-ocr?engine=current|ik-traineddata|compare` benchmarks the
  same visible artifact crops.
- Auto-scan accepts `ocrEngine: "current" | "ik-traineddata"`.
- `scripts/live-soak.ps1` supports:
  - `npm run scan:goal`
  - `npm run scan:goal:current`
  - `npm run scan:goal:ik`
  - `npm run scan:goal:compare`
  - `npm run scan:goal:compare:validated`
  - `npm run scan:goal:compare:validated:wait`
  - `npm run scan:iterate:compare:validated`
  - `npm run scan:iterate:compare:validated:wait`
  - `npm run scan:live:preflight`
  - `npm run scan:live:preflight:wait`
- `scan-performance-assessment.json` ranks runs by quality first and speed
  second, and records whether the 100-artifact result is a complete
  current-vs-IK comparison through `goal100Decision` and
  `goal100.comparisonComplete`.

Important rule:

A fast engine cannot win if it has too many misses or too much review. A
qualified winner must finish cleanly, parse the requested count, keep miss rate
at or below 2%, and keep review rate at or below 15%.

### Auto-scan entry and safety

- The normal auto button runs a guided start:
  1. focus Genshin,
  2. run a lightweight no-OCR preflight,
  3. if artifact detail is visible, use visible-inventory mode,
  4. otherwise try direct `B -> artifact tab -> first artifact tile`,
  5. if needed, fall back to the IK-style ESC/B inventory sequence,
  6. start OCR only after artifact grid and detail card pass preflight.
- Entry captures are state evidence only. They do not create review samples,
  store artifacts, or run artifact OCR before the detail preflight passes.
- Entry waits now poll for state readiness instead of always sleeping the full
  fixed delay.
- Scan loop also rechecks the same safety boundary after each click and scroll.

Why this matters:

The previous failure mode was dangerous from a product-quality point of view:
when the game was not in artifact inventory, the scanner could still take
screenshots and try to read artifacts. The current path is explicitly blocked
outside the artifact inventory/detail state.

### Scan loop and speed

- Grid model uses IK's 32-artifact visible page concept (`8 x 4`) instead of
  clicking the risky lower band.
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
  duplicate reads after scroll.
- Detail-change verification now uses the artifact OCR capture itself instead
  of a separate card-ready capture before OCR.
- Scroll readiness uses inventory fingerprint polling:
  max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
- Store/review writes are held out of the click/capture/OCR hot path. The
  latest source batches artifact store writes before the final summary instead
  of issuing one save/reload cycle per artifact.
- Auto-scan artifact captures skip Electron source enumeration in the hot path
  and call the GDI capture helper directly.
- Focus is done once at scan start; hot-loop captures do not refocus every tile.

Why this matters:

IK uses fixed waits around 200 ms after selecting inventory items and 100 ms
after fast scrolls. The app now keeps those as safety ceilings/acceptance points
while allowing earlier continuation when visual evidence is ready.

### Diagnostics and logging

- Scanner diagnostics now capture timeline events for runtime, focus, keypress,
  entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid,
  counts, detail/page fingerprints, and failure reasons.
- `/scanner/status` publishes recent diagnostic evidence.
- Review sample output is compact by default so Vite does not watch large Base64
  payloads during live soak runs.
- The live runner writes timestamped JSON snapshots, CSV summaries, transcript,
  benchmark data, and performance assessment files under `outputs/live-soak/`.

Why this matters:

Future scanner bugs can be debugged from captured evidence instead of relying
only on a human description of what appeared on screen.

## Vorgehensweise

1. Read the local Inventory Kamera reference under `work/Inventory_Kamera`.
2. Copy the proven concepts, not the entire implementation:
   32 artifact targets per page, fixed coordinate ratios, queued OCR work,
   short item/scroll waits, read-only inventory navigation, and Tesseract
   traineddata comparison.
3. Harden the app's own architecture around those concepts:
   pure lookup API, parser derivation, renderer scan orchestration,
   Electron capture/OCR boundary, sidecar input helper, and diagnostics.
4. Add tests before trusting behavior:
   lookup validation, parser derivation, auto-entry planning/preflight,
   card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
5. Add live tooling before claiming performance:
   bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON
   assessment, and quality-first comparison.

## Tests and Evidence

Latest repo validation after the recent changes:

| Check | Result |
| --- | --- |
| PowerShell parse for `scripts/live-soak.ps1` | Passed |
| `npm run scan:assessment:test` | Passed |
| `npm run lint` | Passed |
| Focused scanner tests | Passed |
| `npm test` | Passed, 171 tests |
| OCR eval seed corpus | 100% exact match, 100% field accuracy, 100% critical fields |
| `npm run build` | Passed |
| `git diff --check` | Passed |

Live evidence already collected:

- Probe click changed artifact detail successfully.
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
- Limit 20 live soak completed on the first visible page.
- Limit 45 live soak crossed into a scrolled page.
- On 2026-07-08, `/scanner/start?entry=visible-inventory&limit=50&engine=current`
  completed `50/50` parsed and stored, `0` review, `0` duplicates, `0` misses,
  `2` pages, `61765 ms` elapsed, `1235 ms/artifact`, `averageCaptureMs: 186`,
  and `averageOcrMs: 162`.
- A deferred single-write flush experiment also completed `50/50`, but regressed
  to `63616 ms` because `writeFlushMs` was `8163`; the source now uses batch
  persist instead, pending a fresh elevated live measurement.
- After direct GDI hot-path optimization, a 20-artifact run completed
  `20/20` parsed, `19` stored, `0` review, `1` duplicate, `0` misses,
  `7966 ms` elapsed, `398 ms/artifact`, `averageCaptureMs: 193`, and
  `averageOcrMs: 167`.
- A 45-artifact direct-GDI run completed `45/45` parsed, `42` stored,
  `0` review, `3` duplicates, `0` misses, `2` pages, `18625 ms` elapsed,
  `414 ms/artifact`, `averageCaptureMs: 187`, and `averageOcrMs: 162`.
- A 100-artifact direct-GDI run on signature
  `2026-07-08-direct-gdi-hotpath` completed `100/100` parsed, `97` stored,
  `0` review, `3` duplicates, `0` misses, `4` pages, `42064 ms` elapsed,
  `421 ms/artifact`, `averageCaptureMs: 179`, and `averageOcrMs: 154`.
- `npm run eval` passed after the speed work with `23/23` exact-match cases,
  `100%` field accuracy, and `100%` critical fields.
- The 3 artifacts/second target is now prepared in code but not live-proven:
  artifact hot-path captures omit detail preview payloads, and stats expose
  capture roundtrip/overhead timing. A qualifying 20-artifact run must finish in
  `<= 6667 ms` with 0 misses and no silent OCR quality regression.
- Follow-up 3/s attempts on 2026-07-08 fixed the false review trigger caused by
  omitted detail previews. The best clean `limit=20` run reached `7285 ms`
  (`364 ms/artifact`, about `2.75 artifacts/second`) with `20/20` parsed,
  `0` review, and `0` misses. The final stable run on
  `2026-07-08-direct-gdi-reviewfix` completed `20/20` with `0` review,
  `0` misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail-region capture,
  5 OCR workers, DataURL buffer decode, and substat `PSM.SINGLE_COLUMN` were
  tested and rejected as slower than the direct-GDI baseline.
- Final current-vs-IK-traineddata comparison on 2026-07-08:
  `npm run scan:goal:compare:validated` passed. Evidence file:
  `outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`,
  `createdAt: 2026-07-08T18:41:11.6120957+02:00`. `goal100Decision` was
  `qualified-comparison: winner=current`; `goal100.comparisonComplete` was true.
  `current` completed `100/100` parsed, `97` stored, `0` review, `0` misses,
  `4` pages, `378 ms/artifact` active average. `ik-traineddata` completed
  `97/100` parsed, `92` stored, `5` review, `3` misses and was rejected for
  parsing fewer artifacts than requested.
- The review queue now has a bounded corpus-growth workflow:
  `npm run eval:review-candidates` writes a deduplicated, Git-ignored worklist
  to `outputs/review-eval-candidates/`. This separates complete modern OCR
  samples from stale captures and prevents the parser's own guess from being
  promoted to ground truth without human confirmation. After manual checking,
  `npm run eval:prepare-confirmed` turns one candidate plus explicit expected
  labels into a paste-ready confirmed corpus snippet.

Current live limitation:

- The fast path and current-vs-IK-traineddata 100-artifact comparison are proven
  in the current live environment. 3 artifacts/second is not proven; remaining
  speed work needs a larger OCR or capture-pipeline change, not more click
  tuning.

## Inventory Kamera Comparison

| Area | Inventory Kamera | Current app status |
| --- | --- | --- |
| Safe scope | Reads inventory through screen/click automation | Same safety boundary: screen capture and read-only input only |
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
| Item wait | About 200 ms fixed wait | No separate wait capture; OCR capture verifies changed detail |
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
| Capture hot path | Direct window/screen capture without source-list scan per item | Direct GDI capture in auto-scan artifact loop |
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
| 100-artifact proof | Reference target | Qualified current-vs-IK comparison captured; `current` won with `100/100`, 0 review, 0 misses, 37.8s projected |

What is theoretically better than before:

- The app no longer spends OCR on invalid screens.
- It no longer treats click count as scanner success.
- It can prove whether `current` or `ik-traineddata` wins on the same capture
  set instead of changing engines blindly.
- It can reject fast-but-wrong results automatically.
- It can identify whether the bottleneck is OCR, capture, card readiness, or
  scroll readiness.

What is not yet proven better than IK:

- Native Tesseract speed is not integrated as the default.
- 3 artifacts/second is not proven.
- Native Inventory Kamera outside this app was not re-run in the same session;
  the completed comparison is against the bundled `ik-traineddata` scan engine.

## Theoretical Runtime Flow

For short iteration while tuning:

1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Run `npm run scan:iterate:compare:validated:wait` from a visible artifact inventory
   when starting directly after UAC, or `npm run scan:iterate:compare:validated`
   if preflight already passes.
3. Inspect `scan-performance-assessment.json`, review samples, and timings if the
   20-artifact comparison fails quality gates.

For the intended 100-artifact comparison:

1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run `npm run scan:goal:compare:validated:wait` directly after UAC, or
   `npm run scan:goal:compare:validated` if preflight already passes.
6. For each engine and limit (`2, 5, 20, 45, 100`):
   - focus Genshin once,
   - verify lookup and layout,
   - verify artifact grid and detail card,
   - click one safe grid target,
   - poll detail fingerprint,
   - skip duplicate visuals,
   - OCR only the fast artifact crop set,
   - parse through lookup constraints,
   - queue store/review writes,
   - scroll with inventory fingerprint polling,
   - stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or
     repeated misses.
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance
   assessment.
8. Declare a winner only if the 100-artifact run is qualified by quality.

Expected bottleneck sequence:

- If OCR dominates, compare `current` vs `ik-traineddata`, crop count, and
  worker pool size.
- If capture dominates, reduce payload construction and preview/crop image work.
- If card-ready dominates, tune the detail fingerprint gate.
- If scroll-ready dominates, tune page fingerprint polling and scroll notches.

## What Is Better Than Before

- Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or
  menu screens.
- Paimon/menu detection blocks before scan-loop OCR or writes.
- The normal button is one coherent guided flow instead of a separate "get to
  inventory first, then scan" workflow.
- The app has a real lookup layer instead of raw OCR plus scattered hardcoded
  assumptions.
- The scanner can compare OCR engines without changing the default blindly.
- Performance reports now include quality decisions, not just elapsed time.
- Diagnostics are concrete enough for later self-troubleshooting.
- Stale elevated runtime is detected before live soak, avoiding false evidence.

## Risks and Remaining Work

1. Keep `current` as the default OCR engine for now; it won the qualified
   current-vs-IK-traineddata live comparison.
2. If pursuing 3 artifacts/second, focus on capture/OCR pipeline changes rather
   than click timing.
3. Grow the eval corpus with confirmed real review samples before tightening
   parser thresholds further.
4. Validate a positive locked-artifact sample.
5. Keep recommendations secondary until scanner quality remains stable across
   repeated live sessions.

## Definition of Done for the IK Target

The goal is complete only when current evidence proves all of these:

- The app is running the latest runtime signature.
- The 100-artifact scan finishes cleanly.
- Parsed count is at least 100.
- Miss rate is at or below 2%.
- Review rate is at or below 15%.
- The run is equal to or faster than the recorded IK reference or the selected
  IK-traineddata/native baseline on the same machine and inventory setup.
- The evidence bundle is saved under `outputs/live-soak/`.
- Any chosen default OCR engine is backed by the same-capture benchmark.