genshin-assistant/docs/scanner-ik-progress-report.md

# Scanner IK Progress Report - 2026-07-07

This report summarizes the scanner/OCR work toward the current target:
scan the first 100 artifacts with accuracy at least as good as Inventory Kamera
and speed equal to or better than Inventory Kamera, without memory reads, hooks,
injection, game-file modification, or unsafe in-game actions.

## Executive Summary

The scanner has moved from a fragile OCR-first prototype toward an
Inventory-Kamera-style artifact scanner:

- Artifact scan is now the first-class path.
- Auto-scan starts only after a validated artifact inventory/detail preflight.
- Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and
  missing-detail states block before OCR/store/review work.
- OCR uses a fast artifact profile that skips low-value fields and derives
  slot, set, and main-stat value through lookup constraints when safe.
- The OCR worker pool, field crop split, page model, scroll model, and direct
  detail-change verification now mirror the relevant IK design choices more
  closely.
- Diagnostics now preserve state evidence, timings, screenshots where useful,
  entry events, focus/input events, preflight failures, and scan-loop reasons.
- A live soak runner now measures throughput and quality, compares current vs.
  IK-traineddata engines, and refuses to run against stale Electron builds.

The current-vs-IK-traineddata comparison proof is now captured. On 2026-07-08,
`npm run scan:goal:compare:validated` passed with
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`.
At `limit=100`, `current` won the qualified comparison with `100/100` parsed,
`0` review, `0` misses, `378 ms/artifact` active average, and `37800 ms`
projected time for 100 artifacts. `ik-traineddata` was not qualified at
`limit=100` because it parsed `97/100`, had `5` review and `3` misses. The
separate 3 artifacts/second target is still not proven.

## What Changed

### Lookup and validation

- `scripts/generate-genshin-data.cjs` was extended into a stricter lookup
  package generator.
- `src/lib/genshinLookup.ts` provides pure matching and validation for sets,
  pieces, slots, stats, characters, aliases, GOOD keys, source version, and
  validation summaries.
- Auto-scan preflight blocks if the lookup package is invalid.

Why this matters:

IK succeeds partly because raw OCR is not trusted by itself. The app now follows
the same principle: OCR text is normalized, matched, constrained, and derived
against a canonical package before it is accepted.

### OCR and parser pipeline

- Artifact detail crops are split into field-specific regions:
  name, slot, main-stat label, main-stat value, level, substats, set effects,
  equipped/footer, lock, and rarity.
- Fast auto-scan profile skips lower-value OCR work:
  set effects, main-stat value crop, crop images, full-frame payloads, and
  inventory preview payloads. The equipped footer remains in real artifact-read
  captures when its marker is visible, because ownership now matters for phase 1
  validation. Preflight and poll captures still skip OCR/crops/lock-state work.
  The slot crop remains in the fast path because it materially improved
  real-read quality.
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
  constraints make that safe.
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
- Exact visual duplicate skipping is disabled in the hottest path; duplicate
  handling now primarily uses parsed artifact signatures so OCR is not skipped
  solely from a crop fingerprint collision.

Why this matters:

The fast path spends OCR only on fields that materially change the artifact
identity or review decision. That is closer to IK's queued crop model than a
manual-debug capture that OCRs every visible thing.

### Engine comparison and benchmark path

- `/scanner/ocr/warmup?engine=current|ik-traineddata` warms OCR workers.
- `/scanner/benchmark-ocr?engine=current|ik-traineddata|compare` benchmarks the
  same visible artifact crops.
- Auto-scan accepts `ocrEngine: "current" | "ik-traineddata"`.
- `scripts/live-soak.ps1` supports:
  - `npm run scan:goal`
  - `npm run scan:goal:current`
  - `npm run scan:goal:ik`
  - `npm run scan:goal:compare`
  - `npm run scan:goal:compare:validated`
  - `npm run scan:goal:compare:validated:wait`
  - `npm run scan:iterate:compare:validated`
  - `npm run scan:iterate:compare:validated:wait`
  - `npm run scan:live:preflight`
  - `npm run scan:live:preflight:wait`
- `scan-performance-assessment.json` ranks runs by quality first and speed
  second, and records whether the 100-artifact result is a complete
  current-vs-IK comparison through `goal100Decision` and
  `goal100.comparisonComplete`.

Important rule:

A fast engine cannot win if it has too many misses or too much review. A
qualified winner must finish cleanly, parse the requested count, keep miss rate
at or below 2%, and keep review rate at or below 15%.

### Auto-scan entry and safety

- The normal auto button runs a guided start:
  1. focus Genshin,
  2. run a lightweight no-OCR preflight,
  3. if artifact detail is visible, use visible-inventory mode,
  4. otherwise try direct `B -> artifact tab -> first artifact tile`,
  5. if needed, fall back to the IK-style ESC/B inventory sequence,
  6. start OCR only after artifact grid and detail card pass preflight.
- Entry captures are state evidence only. They do not create review samples,
  store artifacts, or run artifact OCR before the detail preflight passes.
- Entry waits now poll for state readiness instead of always sleeping the full
  fixed delay.
- Scan loop also rechecks the same safety boundary after each click and scroll.

Why this matters:

The previous failure mode was dangerous from a product-quality point of view:
when the game was not in artifact inventory, the scanner could still take
screenshots and try to read artifacts. The current path is explicitly blocked
outside the artifact inventory/detail state.

### Scan loop and speed

- Grid model uses IK's 32-artifact visible page concept (`8 x 4`) instead of
  clicking the risky lower band.
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
  duplicate reads after scroll.
- Detail-change verification now uses the artifact OCR capture itself instead
  of a separate card-ready capture before OCR.
- Scroll readiness uses inventory fingerprint polling:
  max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
- Store/review writes are held out of the click/capture/OCR hot path. The
  latest source batches artifact store writes before the final summary instead
  of issuing one save/reload cycle per artifact.
- Auto-scan artifact captures skip Electron source enumeration in the hot path
  and call the GDI capture helper directly.
- Focus is done once at scan start; hot-loop captures do not refocus every tile.

Why this matters:

IK uses fixed waits around 200 ms after selecting inventory items and 100 ms
after fast scrolls. The app now keeps those as safety ceilings/acceptance points
while allowing earlier continuation when visual evidence is ready.

### Diagnostics and logging

- Scanner diagnostics now capture timeline events for runtime, focus, keypress,
  entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid,
  counts, detail/page fingerprints, and failure reasons.
- `/scanner/status` publishes recent diagnostic evidence.
- Review sample output is compact by default so Vite does not watch large Base64
  payloads during live soak runs.
- The live runner writes timestamped JSON snapshots, CSV summaries, transcript,
  benchmark data, and performance assessment files under `outputs/live-soak/`.

Why this matters:

Future scanner bugs can be debugged from captured evidence instead of relying
only on a human description of what appeared on screen.

## Vorgehensweise

1. Read the local Inventory Kamera reference under `work/Inventory_Kamera`.
2. Copy the proven concepts, not the entire implementation:
   32 artifact targets per page, fixed coordinate ratios, queued OCR work,
   short item/scroll waits, read-only inventory navigation, and Tesseract
   traineddata comparison.
3. Harden the app's own architecture around those concepts:
   pure lookup API, parser derivation, renderer scan orchestration,
   Electron capture/OCR boundary, sidecar input helper, and diagnostics.
4. Add tests before trusting behavior:
   lookup validation, parser derivation, auto-entry planning/preflight,
   card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
5. Add live tooling before claiming performance:
   bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON
   assessment, and quality-first comparison.

## Tests and Evidence

Latest repo validation after the recent changes:

| Check | Result |
| --- | --- |
| PowerShell parse for `scripts/live-soak.ps1` | Passed |
| `npm run scan:assessment:test` | Passed |
| `npm run lint` | Passed |
| Focused scanner tests | Passed |
| `npm test` | Passed, 171 tests |
| OCR eval seed corpus | 100% exact match, 100% field accuracy, 100% critical fields |
| `npm run build` | Passed |
| `git diff --check` | Passed |

Live evidence already collected:

- Probe click changed artifact detail successfully.
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
- Limit 20 live soak completed on the first visible page.
- Limit 45 live soak crossed into a scrolled page.
- On 2026-07-08, `/scanner/start?entry=visible-inventory&limit=50&engine=current`
  completed `50/50` parsed and stored, `0` review, `0` duplicates, `0` misses,
  `2` pages, `61765 ms` elapsed, `1235 ms/artifact`, `averageCaptureMs: 186`,
  and `averageOcrMs: 162`.
- A deferred single-write flush experiment also completed `50/50`, but regressed
  to `63616 ms` because `writeFlushMs` was `8163`; the source now uses batch
  persist instead, pending a fresh elevated live measurement.
- After direct GDI hot-path optimization, a 20-artifact run completed
  `20/20` parsed, `19` stored, `0` review, `1` duplicate, `0` misses,
  `7966 ms` elapsed, `398 ms/artifact`, `averageCaptureMs: 193`, and
  `averageOcrMs: 167`.
- A 45-artifact direct-GDI run completed `45/45` parsed, `42` stored,
  `0` review, `3` duplicates, `0` misses, `2` pages, `18625 ms` elapsed,
  `414 ms/artifact`, `averageCaptureMs: 187`, and `averageOcrMs: 162`.
- A 100-artifact direct-GDI run on signature
  `2026-07-08-direct-gdi-hotpath` completed `100/100` parsed, `97` stored,
  `0` review, `3` duplicates, `0` misses, `4` pages, `42064 ms` elapsed,
  `421 ms/artifact`, `averageCaptureMs: 179`, and `averageOcrMs: 154`.
- `npm run eval` passed after the speed work with `23/23` exact-match cases,
  `100%` field accuracy, and `100%` critical fields.
- The 3 artifacts/second target is now prepared in code but not live-proven:
  artifact hot-path captures omit detail preview payloads, and stats expose
  capture roundtrip/overhead timing. A qualifying 20-artifact run must finish in
  `<= 6667 ms` with 0 misses and no silent OCR quality regression.
- Follow-up 3/s attempts on 2026-07-08 fixed the false review trigger caused by
  omitted detail previews. The best clean `limit=20` run reached `7285 ms`
  (`364 ms/artifact`, about `2.75 artifacts/second`) with `20/20` parsed,
  `0` review, and `0` misses. The final stable run on
  `2026-07-08-direct-gdi-reviewfix` completed `20/20` with `0` review,
  `0` misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail-region capture,
  5 OCR workers, DataURL buffer decode, and substat `PSM.SINGLE_COLUMN` were
  tested and rejected as slower than the direct-GDI baseline.
- Final current-vs-IK-traineddata comparison on 2026-07-08:
  `npm run scan:goal:compare:validated` passed. Evidence file:
  `outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`,
  `createdAt: 2026-07-08T18:41:11.6120957+02:00`. `goal100Decision` was
  `qualified-comparison: winner=current`; `goal100.comparisonComplete` was true.
  `current` completed `100/100` parsed, `97` stored, `0` review, `0` misses,
  `4` pages, `378 ms/artifact` active average. `ik-traineddata` completed
  `97/100` parsed, `92` stored, `5` review, `3` misses and was rejected for
  parsing fewer artifacts than requested.
- The review queue now has a bounded corpus-growth workflow:
  `npm run eval:review-candidates` writes a deduplicated, Git-ignored worklist
  to `outputs/review-eval-candidates/`. This separates complete modern OCR
  samples from stale captures and prevents the parser's own guess from being
  promoted to ground truth without human confirmation. After manual checking,
  `npm run eval:prepare-confirmed` turns one candidate plus explicit expected
  labels into a paste-ready confirmed corpus snippet.

Current live limitation:

- The fast path and current-vs-IK-traineddata 100-artifact comparison are proven
  in the current live environment. 3 artifacts/second is not proven; remaining
  speed work needs a larger OCR or capture-pipeline change, not more click
  tuning.
- The visible-inventory ownership/lock extension is also live-proven in this
  environment: equipped footer reads for `Citlali` and `Linnea`, unlocked
  `locked: false`, positive locked `locked: true`, and locked/equipped
  persistence all passed on 2026-07-09.

## Inventory Kamera Comparison

| Area | Inventory Kamera | Current app status |
| --- | --- | --- |
| Safe scope | Reads inventory through screen/click automation | Same safety boundary: screen capture and read-only input only |
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
| Item wait | About 200 ms fixed wait | No separate wait capture; OCR capture verifies changed detail |
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
| Capture hot path | Direct window/screen capture without source-list scan per item | Direct GDI capture in auto-scan artifact loop |
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
| 100-artifact proof | Reference target | Qualified current-vs-IK comparison captured; `current` won with `100/100`, 0 review, 0 misses, 37.8s projected |

What is theoretically better than before:

- The app no longer spends OCR on invalid screens.
- It no longer treats click count as scanner success.
- It can prove whether `current` or `ik-traineddata` wins on the same capture
  set instead of changing engines blindly.
- It can reject fast-but-wrong results automatically.
- It can identify whether the bottleneck is OCR, capture, card readiness, or
  scroll readiness.

What is not yet proven better than IK:

- Native Tesseract speed is not integrated as the default.
- 3 artifacts/second is not proven.
- Native Inventory Kamera outside this app was not re-run in the same session;
  the completed comparison is against the bundled `ik-traineddata` scan engine.

## Theoretical Runtime Flow

For short iteration while tuning:

1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Run `npm run scan:iterate:compare:validated:wait` from a visible artifact inventory
   when starting directly after UAC, or `npm run scan:iterate:compare:validated`
   if preflight already passes.
3. Inspect `scan-performance-assessment.json`, review samples, and timings if the
   20-artifact comparison fails quality gates.

For the intended 100-artifact comparison:

1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run `npm run scan:goal:compare:validated:wait` directly after UAC, or
   `npm run scan:goal:compare:validated` if preflight already passes.
6. For each engine and limit (`2, 5, 20, 45, 100`):
   - focus Genshin once,
   - verify lookup and layout,
   - verify artifact grid and detail card,
   - click one safe grid target,
   - poll detail fingerprint,
   - skip duplicate visuals,
   - OCR only the fast artifact crop set,
   - parse through lookup constraints,
   - queue store/review writes,
   - scroll with inventory fingerprint polling,
   - stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or
     repeated misses.
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance
   assessment.
8. Declare a winner only if the 100-artifact run is qualified by quality.

Expected bottleneck sequence:

- If OCR dominates, compare `current` vs `ik-traineddata`, crop count, and
  worker pool size.
- If capture dominates, reduce payload construction and preview/crop image work.
- If card-ready dominates, tune the detail fingerprint gate.
- If scroll-ready dominates, tune page fingerprint polling and scroll notches.

## What Is Better Than Before

- Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or
  menu screens.
- Paimon/menu detection blocks before scan-loop OCR or writes.
- The normal button is one coherent guided flow instead of a separate "get to
  inventory first, then scan" workflow.
- The app has a real lookup layer instead of raw OCR plus scattered hardcoded
  assumptions.
- The scanner can compare OCR engines without changing the default blindly.
- Performance reports now include quality decisions, not just elapsed time.
- Diagnostics are concrete enough for later self-troubleshooting.
- Stale elevated runtime is detected before live soak, avoiding false evidence.

## Risks and Remaining Work

1. Keep `current` as the default OCR engine for now; it won the qualified
   current-vs-IK-traineddata live comparison.
2. If pursuing 3 artifacts/second, focus on capture/OCR pipeline changes rather
   than click timing.
3. Grow the eval corpus with confirmed real review samples before tightening
   parser thresholds further.
4. Repeat equipped/locked live samples in later sessions if confidence or UI
   behavior changes, but the first positive locked proof has passed.
5. Keep recommendations secondary until scanner quality remains stable across
   repeated live sessions and the result rail, artifact inventory, and detail
   evaluation flow are implemented.
6. Treat current speed as acceptable for the next product phase; prioritize
   artifact content extraction and review-safe value scoring before another
   broad speed pass.

## Definition of Done for the IK Target

The goal is complete only when current evidence proves all of these:

- The app is running the latest runtime signature.
- The 100-artifact scan finishes cleanly.
- Parsed count is at least 100.
- Miss rate is at or below 2%.
- Review rate is at or below 15%.
- The run is equal to or faster than the recorded IK reference or the selected
  IK-traineddata/native baseline on the same machine and inventory setup.
- The evidence bundle is saved under `outputs/live-soak/`.
- Any chosen default OCR engine is backed by the same-capture benchmark.