Improve IK-style artifact scanner pipeline
This commit is contained in:
AzuTear
@@ -0,0 +1,301 @@
|
||||
# Scanner IK Progress Report - 2026-07-07
|
||||
|
||||
This report summarizes the scanner/OCR work toward the current target:
|
||||
scan the first 100 artifacts with accuracy at least as good as Inventory Kamera
|
||||
and speed equal to or better than Inventory Kamera, without memory reads, hooks,
|
||||
injection, game-file modification, or unsafe in-game actions.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The scanner has moved from a fragile OCR-first prototype toward an
|
||||
Inventory-Kamera-style artifact scanner:
|
||||
|
||||
- Artifact scan is now the first-class path.
|
||||
- Auto-scan starts only after a validated artifact inventory/detail preflight.
|
||||
- Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and
|
||||
missing-detail states block before OCR/store/review work.
|
||||
- OCR uses a fast artifact profile that skips low-value fields and derives
|
||||
slot, set, and main-stat value through lookup constraints when safe.
|
||||
- The OCR worker pool, field crop split, page model, scroll model, and readiness
|
||||
waits now mirror the relevant IK design choices more closely.
|
||||
- Diagnostics now preserve state evidence, timings, screenshots where useful,
|
||||
entry events, focus/input events, preflight failures, and scan-loop reasons.
|
||||
- A live soak runner now measures throughput and quality, compares current vs.
|
||||
IK-traineddata engines, and refuses to run against stale Electron builds.
|
||||
|
||||
The requested final goal is not proven complete yet. The current live dev port is
|
||||
still owned by an older elevated Electron instance, so the latest code cannot be
|
||||
truthfully benchmarked against IK until the app is restarted with UAC approval
|
||||
and `npm run scan:goal:compare` completes a qualified 100-artifact run.
|
||||
|
||||
## What Changed
|
||||
|
||||
### Lookup and validation
|
||||
|
||||
- `scripts/generate-genshin-data.cjs` was extended into a stricter lookup
|
||||
package generator.
|
||||
- `src/lib/genshinLookup.ts` provides pure matching and validation for sets,
|
||||
pieces, slots, stats, characters, aliases, GOOD keys, source version, and
|
||||
validation summaries.
|
||||
- Auto-scan preflight blocks if the lookup package is invalid.
|
||||
|
||||
Why this matters:
|
||||
|
||||
IK succeeds partly because raw OCR is not trusted by itself. The app now follows
|
||||
the same principle: OCR text is normalized, matched, constrained, and derived
|
||||
against a canonical package before it is accepted.
|
||||
|
||||
### OCR and parser pipeline
|
||||
|
||||
- Artifact detail crops are split into field-specific regions:
|
||||
name, slot, main-stat label, main-stat value, level, substats, set effects,
|
||||
equipped/footer, lock, and rarity.
|
||||
- Fast auto-scan profile skips lower-value OCR work:
|
||||
set effects, slot crop, main-stat value crop, equipped footer, crop images,
|
||||
full-frame payloads, and inventory preview payloads.
|
||||
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
|
||||
constraints make that safe.
|
||||
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
|
||||
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
|
||||
- Exact visual duplicates are skipped before OCR.
|
||||
|
||||
Why this matters:
|
||||
|
||||
The fast path spends OCR only on fields that materially change the artifact
|
||||
identity or review decision. That is closer to IK's queued crop model than a
|
||||
manual-debug capture that OCRs every visible thing.
|
||||
|
||||
### Engine comparison and benchmark path
|
||||
|
||||
- `/scanner/ocr/warmup?engine=current|ik-traineddata` warms OCR workers.
|
||||
- `/scanner/benchmark-ocr?engine=current|ik-traineddata|compare` benchmarks the
|
||||
same visible artifact crops.
|
||||
- Auto-scan accepts `ocrEngine: "current" | "ik-traineddata"`.
|
||||
- `scripts/live-soak.ps1` supports:
|
||||
- `npm run scan:goal`
|
||||
- `npm run scan:goal:current`
|
||||
- `npm run scan:goal:ik`
|
||||
- `npm run scan:goal:compare`
|
||||
- `scan-performance-assessment.json` ranks runs by quality first and speed
|
||||
second.
|
||||
|
||||
Important rule:
|
||||
|
||||
A fast engine cannot win if it has too many misses or too much review. A
|
||||
qualified winner must finish cleanly, parse the requested count, keep miss rate
|
||||
at or below 2%, and keep review rate at or below 15%.
|
||||
|
||||
### Auto-scan entry and safety
|
||||
|
||||
- The normal auto button runs a guided start:
|
||||
1. focus Genshin,
|
||||
2. run a lightweight no-OCR preflight,
|
||||
3. if artifact detail is visible, use visible-inventory mode,
|
||||
4. otherwise try direct `B -> artifact tab -> first artifact tile`,
|
||||
5. if needed, fall back to the IK-style ESC/B inventory sequence,
|
||||
6. start OCR only after artifact grid and detail card pass preflight.
|
||||
- Entry captures are state evidence only. They do not create review samples,
|
||||
store artifacts, or run artifact OCR before the detail preflight passes.
|
||||
- Entry waits now poll for state readiness instead of always sleeping the full
|
||||
fixed delay.
|
||||
- Scan loop also rechecks the same safety boundary after each click and scroll.
|
||||
|
||||
Why this matters:
|
||||
|
||||
The previous failure mode was dangerous from a product-quality point of view:
|
||||
when the game was not in artifact inventory, the scanner could still take
|
||||
screenshots and try to read artifacts. The current path is explicitly blocked
|
||||
outside the artifact inventory/detail state.
|
||||
|
||||
### Scan loop and speed
|
||||
|
||||
- Grid model uses IK's 32-artifact visible page concept (`8 x 4`) instead of
|
||||
clicking the risky lower band.
|
||||
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
|
||||
duplicate reads after scroll.
|
||||
- Card readiness uses detail fingerprint polling:
|
||||
max 420 ms, 60 ms polls, changed cards may proceed after 200 ms.
|
||||
- Scroll readiness uses inventory fingerprint polling:
|
||||
max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
|
||||
- Store/review writes are queued so the next tile can be clicked before disk
|
||||
writes finish. The queue is still flushed before final summary.
|
||||
- Focus is done once at scan start; hot-loop captures do not refocus every tile.
|
||||
|
||||
Why this matters:
|
||||
|
||||
IK uses fixed waits around 200 ms after selecting inventory items and 100 ms
|
||||
after fast scrolls. The app now keeps those as safety ceilings/acceptance points
|
||||
while allowing earlier continuation when visual evidence is ready.
|
||||
|
||||
### Diagnostics and logging
|
||||
|
||||
- Scanner diagnostics now capture timeline events for runtime, focus, keypress,
|
||||
entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid,
|
||||
counts, detail/page fingerprints, and failure reasons.
|
||||
- `/scanner/status` publishes recent diagnostic evidence.
|
||||
- Review sample output is compact by default so Vite does not watch large Base64
|
||||
payloads during live soak runs.
|
||||
- The live runner writes timestamped JSON snapshots, CSV summaries, transcript,
|
||||
benchmark data, and performance assessment files under `outputs/live-soak/`.
|
||||
|
||||
Why this matters:
|
||||
|
||||
Future scanner bugs can be debugged from captured evidence instead of relying
|
||||
only on a human description of what appeared on screen.
|
||||
|
||||
## Vorgehensweise
|
||||
|
||||
1. Read the local Inventory Kamera reference under `work/Inventory_Kamera`.
|
||||
2. Copy the proven concepts, not the entire implementation:
|
||||
32 artifact targets per page, fixed coordinate ratios, queued OCR work,
|
||||
short item/scroll waits, read-only inventory navigation, and Tesseract
|
||||
traineddata comparison.
|
||||
3. Harden the app's own architecture around those concepts:
|
||||
pure lookup API, parser derivation, renderer scan orchestration,
|
||||
Electron capture/OCR boundary, sidecar input helper, and diagnostics.
|
||||
4. Add tests before trusting behavior:
|
||||
lookup validation, parser derivation, auto-entry planning/preflight,
|
||||
card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
|
||||
5. Add live tooling before claiming performance:
|
||||
bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON
|
||||
assessment, and quality-first comparison.
|
||||
|
||||
## Tests and Evidence
|
||||
|
||||
Latest repo validation after the recent changes:
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| PowerShell parse for `scripts/live-soak.ps1` | Passed |
|
||||
| `npm run scan:assessment:test` | Passed |
|
||||
| `npm run lint` | Passed |
|
||||
| Focused scanner tests | Passed |
|
||||
| `npm test` | Passed, 171 tests |
|
||||
| OCR eval seed corpus | 100% exact match, 100% field accuracy, 100% critical fields |
|
||||
| `npm run build` | Passed |
|
||||
| `git diff --check` | Passed |
|
||||
|
||||
Live evidence already collected earlier on 2026-07-07:
|
||||
|
||||
- Probe click changed artifact detail successfully.
|
||||
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
|
||||
- Limit 20 live soak completed on the first visible page.
|
||||
- Limit 45 live soak crossed into a scrolled page.
|
||||
|
||||
Current live limitation:
|
||||
|
||||
- `/health` still reports an older elevated build:
|
||||
`2026-07-07-ocr-pool4-hotloop-no-refocus`.
|
||||
- Current source expects:
|
||||
`2026-07-07-ik32-fastsubstats-active-timing`.
|
||||
- The live soak runner correctly refuses to benchmark the stale runtime.
|
||||
- A UAC restart attempt was canceled, so the latest code is not yet live.
|
||||
|
||||
## Inventory Kamera Comparison
|
||||
|
||||
| Area | Inventory Kamera | Current app status |
|
||||
| --- | --- | --- |
|
||||
| Safe scope | Reads inventory through screen/click automation | Same safety boundary: screen capture and read-only input only |
|
||||
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
|
||||
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
|
||||
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
|
||||
| Item wait | About 200 ms fixed wait | Fingerprint polling, accepts changed card after 200 ms |
|
||||
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
|
||||
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
|
||||
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
|
||||
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
|
||||
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
|
||||
| 100-artifact proof | Reference target | Not yet proven on latest app build |
|
||||
|
||||
What is theoretically better than before:
|
||||
|
||||
- The app no longer spends OCR on invalid screens.
|
||||
- It no longer treats click count as scanner success.
|
||||
- It can prove whether `current` or `ik-traineddata` wins on the same capture
|
||||
set instead of changing engines blindly.
|
||||
- It can reject fast-but-wrong results automatically.
|
||||
- It can identify whether the bottleneck is OCR, capture, card readiness, or
|
||||
scroll readiness.
|
||||
|
||||
What is not yet proven better than IK:
|
||||
|
||||
- Native Tesseract speed is not integrated as the default.
|
||||
- The latest code has not completed the 100-artifact live run.
|
||||
- Review rate and miss rate on the user's real inventory still need the new
|
||||
live report.
|
||||
|
||||
## Theoretical Runtime Flow
|
||||
|
||||
For the intended 100-artifact comparison:
|
||||
|
||||
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
|
||||
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
|
||||
3. Warm current and IK-traineddata OCR workers.
|
||||
4. Run a small bounded probe from the artifact inventory.
|
||||
5. Run `npm run scan:goal:compare`.
|
||||
6. For each engine and limit (`2, 5, 20, 45, 100`):
|
||||
- focus Genshin once,
|
||||
- verify lookup and layout,
|
||||
- verify artifact grid and detail card,
|
||||
- click one safe grid target,
|
||||
- poll detail fingerprint,
|
||||
- skip duplicate visuals,
|
||||
- OCR only the fast artifact crop set,
|
||||
- parse through lookup constraints,
|
||||
- queue store/review writes,
|
||||
- scroll with inventory fingerprint polling,
|
||||
- stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or
|
||||
repeated misses.
|
||||
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance
|
||||
assessment.
|
||||
8. Declare a winner only if the 100-artifact run is qualified by quality.
|
||||
|
||||
Expected bottleneck sequence:
|
||||
|
||||
- If OCR dominates, compare `current` vs `ik-traineddata`, crop count, and
|
||||
worker pool size.
|
||||
- If capture dominates, reduce payload construction and preview/crop image work.
|
||||
- If card-ready dominates, tune the detail fingerprint gate.
|
||||
- If scroll-ready dominates, tune page fingerprint polling and scroll notches.
|
||||
|
||||
## What Is Better Than Before
|
||||
|
||||
- Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or
|
||||
menu screens.
|
||||
- Paimon/menu detection blocks before scan-loop OCR or writes.
|
||||
- The normal button is one coherent guided flow instead of a separate "get to
|
||||
inventory first, then scan" workflow.
|
||||
- The app has a real lookup layer instead of raw OCR plus scattered hardcoded
|
||||
assumptions.
|
||||
- The scanner can compare OCR engines without changing the default blindly.
|
||||
- Performance reports now include quality decisions, not just elapsed time.
|
||||
- Diagnostics are concrete enough for later self-troubleshooting.
|
||||
- Stale elevated runtime is detected before live soak, avoiding false evidence.
|
||||
|
||||
## Risks and Remaining Work
|
||||
|
||||
1. Restart with UAC and run the latest build live.
|
||||
2. Run `npm run scan:goal:compare` from a visible artifact inventory.
|
||||
3. If the 100-artifact winner is not qualified, inspect:
|
||||
`scan-performance-assessment.json`, review samples, diagnostic timeline, and
|
||||
field timings.
|
||||
4. If `ik-traineddata` wins but Tesseract.js is still slow, evaluate native
|
||||
Tesseract integration.
|
||||
5. Grow the eval corpus with confirmed real review samples before tightening
|
||||
parser thresholds further.
|
||||
6. Validate a positive locked-artifact sample.
|
||||
7. Keep recommendations secondary until scanner quality is proven.
|
||||
|
||||
## Definition of Done for the IK Target
|
||||
|
||||
The goal is complete only when current evidence proves all of these:
|
||||
|
||||
- The app is running the latest runtime signature.
|
||||
- The 100-artifact scan finishes cleanly.
|
||||
- Parsed count is at least 100.
|
||||
- Miss rate is at or below 2%.
|
||||
- Review rate is at or below 15%.
|
||||
- The run is equal to or faster than the recorded IK reference or the selected
|
||||
IK-traineddata/native baseline on the same machine and inventory setup.
|
||||
- The evidence bundle is saved under `outputs/live-soak/`.
|
||||
- Any chosen default OCR engine is backed by the same-capture benchmark.
|
||||
Reference in New Issue
Block a user