Improve IK-style artifact scanner pipeline

2026-07-07 22:02:24 +02:00
parent 8ebbe91c39
commit f791d1464c
70 changed files with 7408 additions and 445 deletions
@@ -23,6 +23,12 @@ Validated live on 2026-07-07 with Genshin open in the artifact inventory at
 This proves that the current elevated app plus helper path can deliver mouse
 movement and click input to the focused Genshin client in this environment.

+Latest-source timing is not proven while `/health.appBuild.signature` differs
+from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`. On 2026-07-07 the port
+was still owned by an older elevated runtime, so goal scans were intentionally
+blocked by the stale-build gate. Restart the elevated app through
+`npm run dev:admin` and confirm UAC before collecting new 100-artifact evidence.
+
 ## Elevation And UAC

 Use:
@@ -98,6 +104,66 @@ For live validation, prefer a bounded scan first:
 Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
 ```

+The visible-inventory path remains the safest first check. The normal guided
+entry tries the read-only direct world path first:
+`B -> artifact tab -> first artifact tile`. If that does not produce a visible
+artifact detail card, it falls back to the Inventory Kamera-compatible sequence:
+`ESC -> B -> artifact tab -> first artifact tile`.
+
+The normal Auto-Scan button uses a guided start. It first takes one lightweight
+preflight capture without OCR, full-frame payload, review scoring, or storing.
+If an artifact detail card is already visible, it starts the visible-inventory
+scan. Otherwise it runs the guided entry above. OCR/review/store work starts
+only after the artifact-detail preflight passes.
+Guided entry uses short state polling for the Inventory screen, artifact grid,
+and first detail card instead of waiting the full fixed delay every time; if the
+state never appears, the same timeout budget returns the last diagnostic capture.
+
+```powershell
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=auto-entry&limit=2"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2&engine=ik-traineddata"
+```
+
+Those paths send only read-only navigation. `ESC` is not a universal "go to
+world" command: from the world it opens the Paimon menu, while from the
+already-open Paimon menu it returns to the world. This is why the normal
+`auto-entry` path first tries `B` directly and uses the IK-style `ESC -> B`
+fallback only when direct entry did not reach an artifact detail card.
+
+The scan starts only after a valid lookup package, supported 16:9 layout,
+detected artifact grid, Genshin-client capture, and visual artifact-detail
+markers are all present. If any preflight check fails, keep using the
+visible-inventory path while tuning the entry step.
+
+The visual preflight also classifies the Paimon menu. The Paimon profile/card
+grid can look like an inventory grid if only fixed 16:9 coordinates are used, so
+the scanner must reject `paimonMenu.present` before any artifact OCR, review
+sample creation, store write, or grid scan starts. The guided entry may still
+take lightweight skip-OCR captures while navigating, but those captures are only
+state evidence.
+
+The same guard also runs inside the scan loop. If the app is on the main game
+screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
+without an artifact detail card, auto-scan must block instead of clicking tiles
+or trying OCR.
+After each click the loop polls the detail fingerprint with a short bounded
+budget instead of sleeping blindly. The current budget is 420 ms with 60 ms
+polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it
+does not change the loop retries or stops through the normal miss guards. If the
+card changed but remains animated, the loop now proceeds after 200 ms, matching
+Inventory Kamera's select-next-item wait more closely without removing the
+detail-change guard.
+The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
+not re-run the focus helper before every tile, which avoids an OS focus ping on
+each artifact while still relying on click readback, foreground checks, and the
+detail-card guard for safety.
+After a scroll, the loop now uses the same cheap fingerprint polling model for
+the inventory pane: it proceeds as soon as the next page fingerprint changed and
+stabilized instead of always sleeping the old fixed 760 ms settle delay. Changed
+but still animated inventory pages may proceed after 100 ms, again matching IK's
+fast-scroll wait while still blocking unchanged pages.
+
 Then poll:

 ```powershell
@@ -105,10 +171,221 @@ Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12
 ```

+Before live timing, verify that the endpoint is the current app instance:
+
+```powershell
+Invoke-RestMethod "http://127.0.0.1:17317/health" |
+  ConvertTo-Json -Depth 6
+```
+
+The response must include `appBuild.signature` and
+`appBuild.expectedOcrWorkerPoolSize`. If `appBuild` is missing, or
+`/scanner/status` still reports the old OCR warmup start time, the local port is
+still owned by a stale elevated Electron process. Close the old Administrator
+window/app and restart with `npm run dev:admin` before running scanner probes.
+
+Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
+`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
+`activeAverageMsPerParsed`, `averageCaptureMs`, `averageOcrMs`,
+`artifactsPerMinute`, and `projectedMsFor100`. `elapsedMs` is end-to-end
+including queued writes; `activeScanMs` is the click/capture/OCR loop before
+the final store/review flush. A run only counts as speed
+evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
+together; raw click count alone is not scanner throughput. If `averageOcrMs`
+dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
+queue. If `averageCaptureMs` dominates, crop payload/capture work is the
+bottleneck.
+
 The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
 temporary scan limit. It does not change the normal UI setting. The normal
 hotkeys and buttons still use the UI's configured scan limit.

+Lookup and benchmark utility endpoints:
+
+```powershell
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/status"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/regenerate"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup?engine=ik-traineddata"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=ik-traineddata"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=compare"
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&profile=full"
+```
+
+The benchmark endpoint measures the current Tesseract.js engine and the
+Inventory-Kamera-traineddata Tesseract.js path against the artifact crop set and
+returns timing/field counts, min/p50/p90/max timing, OCR p50/p90 timing,
+20/45/100-artifact projections, skipped-OCR count, and the active OCR worker
+pool size. It also returns per-field OCR timings under
+`ocrFieldAverages`, which is the first place to look before changing crop or
+parser behavior. Individual captures also report whether the artifact was
+detected as `sanctified`; level/substat crops are shifted in that state to match
+Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
+which omits the low-value set-effect crop, the slot crop that can be derived
+from the matched artifact piece name, and the main-stat-value crop that can be
+derived from slot, main-stat label, and level. The fast profile also uses
+Inventory Kamera's tighter substat crop height; full/manual captures keep the
+larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
+captures while keeping the detail screenshot, OCR text, crop rect metadata, and
+timings. OCR crops are passed to Tesseract as PNG buffers internally, not as
+Base64 DataURLs, to avoid encode/decode overhead in batch scans. When
+`skipOcrUnlessArtifactDetail` blocks OCR because no artifact detail card is
+visible, OCR crop preprocessing is skipped too. Auto-scan readiness and scroll
+checks use native detail/inventory fingerprints and omit preview DataURLs in
+poll captures. Fast preflight/poll captures also omit crop list construction, crop images, and lock-state
+detection unless a caller explicitly overrides that option; add
+`profile=full` to OCR every artifact detail crop for debugging. It uses the same
+artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
+skipped and the response shows `skippedOcrCaptures` instead of burning time on
+invalid crops.
+For speed, the fast auto-scan profile also skips the optional Equipped footer
+OCR. Name, level, main-stat label, and substats remain in the OCR hot path;
+slot, set, and main-stat value are derived when the lookup/parser can validate
+them. Use a full/manual capture when equipped ownership or every debug crop matters.
+Local store/review writes are serialized through an internal queue but no longer
+block the next inventory click. The scan still flushes the queue before it
+returns its final summary, so `stored` and `review` counts remain final-state
+numbers.
+The app warms the default OCR worker pool in the background after startup; check
+`/scanner/status` -> `ocrWarmup.current` before timing the first artifact. Use
+`/scanner/ocr/warmup?engine=ik-traineddata` before comparing Inventory
+Kamera-traineddata timings so the benchmark is not dominated by worker creation.
+`engine=ik-traineddata` uses Inventory Kamera's local
+`genshin_fast_09_04_21.traineddata` through Tesseract.js when the file is found
+in `data/tessdata`, `IK_TESSDATA_DIR`, `work/Inventory_Kamera`, `work/refs`,
+or the local `_ik_ref*` folders.
+`engine=compare` runs `current` and `ik-traineddata` against the same visible
+artifact detail state. The auto-scan default must stay `current` until the IK
+traineddata path wins on the same captures. For a controlled live comparison,
+start the scanner with `engine=ik-traineddata`; this only changes the OCR
+worker language for that run and leaves the default UI/hotkey path on
+`current`.
+The OCR pool defaults to four workers because the fast artifact crop set has
+four useful OCR parameter groups; set `GAA_OCR_WORKERS=1..8` before startup to
+benchmark a different worker count. Inventory Kamera's native engine pool is
+still the reference design, but the current app path remains Tesseract.js until
+native OCR is integrated and measured. Crops are scheduled across the whole
+worker pool and each worker caches its last Tesseract parameter profile; this is
+closer to Inventory Kamera's multi-engine field OCR than the earlier
+parameter-group-serial scheduler.
+
+## Diagnostic Evidence
+
+The Diagnose page contains a compact evidence timeline for scanner work. It logs
+runtime pings, focus attempts, key presses, entry captures, artifact-tab clicks,
+preflight failures, grid/count metadata, detail fingerprints, and detail/inventory
+screenshots. The same last events are also published through:
+
+```powershell
+Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
+  ConvertTo-Json -Depth 18
+```
+
+Use this before changing scanner behavior: run the smallest failing action, read
+the evidence timeline, then decide whether the failure is focus/input, entry
+navigation, grid detection, capture quality, OCR, or parser validation.
+
+If Paimon entry shows `entry key ESC` or `entry key B` with `eventsSent: 0`, the
+running `InputHelper.exe` probably predates keyboard support or is blocked. Stop
+the elevated app/helper, run `npm run helper:build`, then restart with
+`npm run dev:admin` so the app loads the rebuilt helper.
+
+## Soak-Test Helper
+
+After the elevated app is running and Genshin is open on the artifact inventory,
+the non-elevated terminal can drive the local dev-control endpoints and save a
+full evidence bundle:
+
+```powershell
+npm run scan:soak
+```
+
+The helper writes timestamped JSON snapshots and a transcript to:
+
+```text
+outputs/live-soak/<timestamp>/
+```
+
+Default sequence:
+
+1. `/health`
+2. `/scanner/status`
+3. `/capture/smart?skipOcr=1`
+4. `/automation/probe-click?index=1`
+5. `/automation/probe-click?index=3`
+6. `/scanner/start?limit=2`
+7. `/scanner/start?limit=5`
+8. `/scanner/start?limit=10`
+9. `/scanner/start?limit=20`
+10. `/review/samples?limit=30`
+
+For the actual Inventory-Kamera speed target, use the explicit goal run after
+`/health` shows the current `appBuild`:
+
+```powershell
+npm run scan:goal
+npm run scan:goal:current
+npm run scan:goal:ik
+npm run scan:goal:compare
+```
+
+That run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
+limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
+`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
+uses the default `current` scan engine; use `scan:goal:ik` for a native
+IK-traineddata scan pass. Use `scan:goal:compare` to run both scan engines
+back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for
+`averageMsPerParsed`, `activeAverageMsPerParsed`, `averageCaptureMs`,
+`captureP50Ms`, `captureP90Ms`, `averageOcrMs`, `ocrP50Ms`, `ocrP90Ms`,
+`averageCardReadyMs`, `averageScrollReadyMs`, `artifactsPerMinute`, and
+`projectedMsFor100`.
+The run also writes `scan-performance-assessment.json`, which groups results by
+limit, picks the best qualified engine, and labels the dominant bottleneck as
+OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
+run, parse the requested count, keep miss rate under 2%, and keep review rate
+at or below 15%; review and miss rates are penalized before active average speed
+is used as the tie-breaker.
+
+The assessment ranking can be verified without Genshin or the Electron app:
+
+```powershell
+npm run scan:assessment:test
+```
+
+This self-test rejects synthetic runs that are fast but have too many misses or
+too many review samples, so the final IK comparison cannot be won by speed alone.
+
+For the current implementation summary and IK comparison rationale, see
+[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
+
+Use the readiness timings to compare against Inventory Kamera's fixed waits:
+IK waits about 200 ms after selecting the next inventory item and about 100 ms
+after fast scrolls. If `averageCardReadyMs` or `averageScrollReadyMs` dominates
+the active average while OCR is already low, tune the fingerprint gate before
+touching OCR again.
+
+The runner reads `APP_RUNTIME_SIGNATURE` from `electron/main.ts` and refuses
+to run against a stale Electron process when `/health.appBuild.signature` does
+not match the current source. Use `-AllowStaleBuild` only for deliberate
+debugging of an older instance.
+Current dev builds also expose `/dev/shutdown` on localhost. The start cleanup
+script calls it before falling back to `Stop-Process`, so a previous elevated
+app can shut itself down cleanly even when the caller cannot terminate an
+administrator process directly. Older builds without that endpoint still need
+manual close or a confirmed `npm run dev:admin` restart.
+
+Review samples are saved as a compact summary by default so Vite does not try to
+watch large Base64 payloads under `outputs/`. Full review payloads can be saved
+with `-SaveFullReviewSamples` when needed.
+
+It stops on a failed probe, blocked scan, stopped scan, or timeout unless
+`-ContinueAfterBlocked` is supplied directly:
+
+```powershell
+powershell -NoProfile -ExecutionPolicy Bypass -File scripts\live-soak.ps1 -Limits 2,5 -ContinueAfterBlocked
+```
+
 ## Anti-Cheat And Safety Boundary

 Do not describe the current implementation as bypassing anti-cheat. The app
@@ -133,7 +410,9 @@ The current 16:9 layout profile is calibrated from a 1920x1080 English
 artifact-inventory capture:

 - detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
- inventory grid: `8 x 5`
+- inventory grid: `8 x 4` safe automated targets, matching Inventory Kamera's
+  32-artifact full-page model. The apparent lower fifth row is in the bottom
+  control band and is intentionally not clicked during auto-scan.
 - first tile center: `x=179`, `y=254`, `row=0`, `col=0`
 - second tile center: `x=325`, `y=254`, `row=0`, `col=1`
 - inventory count crop successfully read `2059/2400` in the live session
@@ -152,3 +431,5 @@ Before marking an automation change done:
 5. If Genshin is available, run `/automation/probe-click?index=1`.
 6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
 7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
+8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
+   a non-stale `npm run scan:goal:compare` run.