Improve IK-style artifact scanner pipeline
This commit is contained in:
AzuTear
+19
-1
@@ -55,6 +55,9 @@ flowchart LR
|
||||
| `src/App.tsx` | Main app shell, scan view, triage view, build view, overlay preview |
|
||||
| `src/lib/artifactOcrParser.ts` | Converts OCR output into a parsed artifact candidate with confidence and notes |
|
||||
| `src/lib/fuzzyMatch.ts` | Generic fuzzy string matching for OCR text against known game data |
|
||||
| `src/lib/genshinLookup.ts` | Pure lookup and validation API for generated Genshin data |
|
||||
| `src/lib/autoScanEntry.ts` | Pure entry-mode planning and auto-scan preflight validation |
|
||||
| `src/lib/cardReadyGate.ts` | Detail/page fingerprint readiness gate for scan timing |
|
||||
| `src/lib/scoring.ts` | Recommendation and build scoring logic |
|
||||
| `src/lib/demoData.ts` | Temporary local demo snapshot |
|
||||
| `src/data/genshinGameData.json` | Generated local dictionary of characters, artifact sets, slots, and stats |
|
||||
@@ -131,6 +134,17 @@ sequenceDiagram
|
||||
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
- Click verification: after each click the parsed detail-panel signature should change. An unchanged signature is a soft miss (it can also mean two OCR-identical neighbor pieces, common among +0 artifacts), so it is retried once with a small offset, logged with the stuck artifact name, and then skipped - never fatal on its own. The scan aborts only when the first ~6 clicks of page 1 produce nothing new (diagnosis hint: elevated Genshin blocks SendInput via UIPI, or grid coordinates are wrong) or a later page yields zero new artifacts.
|
||||
- Scan stats separate clicked (click attempts), parsed (readable captures), stored (persisted), review (review samples), duplicates, and misses, so "scanned" cannot be mistaken for "successfully read".
|
||||
- Artifact grid automation uses Inventory Kamera's 32-target full-page model
|
||||
(`8 x 4` safe click targets). The apparent lower fifth row sits in the
|
||||
bottom control band on 16:9 captures and is not clicked automatically.
|
||||
- Guided auto-entry is state gated. The normal scan button first performs a
|
||||
lightweight no-OCR preflight; OCR/store/review work starts only after the
|
||||
artifact inventory grid and artifact detail card are visually confirmed.
|
||||
- The entry path tries direct inventory (`B`) first and uses the IK-style
|
||||
ESC/B fallback only when direct entry does not reach an artifact detail card.
|
||||
- Card and page waits are fingerprint based. The scanner can proceed as soon as
|
||||
the expected visual state changes and stabilizes, while still accepting IK-like
|
||||
200 ms item and 100 ms scroll readiness points.
|
||||
- Scrolling sends one wheel notch per grid row with the cursor anchored over the inventory (assumption: roughly one row per notch; overlap is absorbed by dedupe, and a page without new artifacts stops the scan).
|
||||
- The scan never deletes, enhances, feeds, locks, or spends anything; it only selects tiles to read them.
|
||||
|
||||
@@ -156,4 +170,8 @@ Parsed artifacts from both modes are persisted into `artifact-store.json` keyed
|
||||
|
||||
## Performance
|
||||
|
||||
Current OCR is prototype-grade and may be slower than the target scanner. Two batch-scan bottlenecks were removed: input/capture no longer spawn a PowerShell process (and recompile Win32 interop) per action, and the Tesseract worker is created once and reused across captures. The eventual batch scanner should still move expensive capture/OCR/build work into workers or a native sidecar.
|
||||
Current OCR is still measured against the IK target rather than assumed good.
|
||||
The app keeps a Tesseract.js worker pool, can use the Inventory-Kamera
|
||||
`genshin_fast_09_04_21.traineddata` path for comparison, and reports capture,
|
||||
OCR, card-ready, scroll-ready, active-scan, and projected-100 timings. A default
|
||||
engine change requires a same-capture benchmark and a qualified live soak result.
|
||||
|
||||
@@ -23,6 +23,12 @@ Validated live on 2026-07-07 with Genshin open in the artifact inventory at
|
||||
This proves that the current elevated app plus helper path can deliver mouse
|
||||
movement and click input to the focused Genshin client in this environment.
|
||||
|
||||
Latest-source timing is not proven while `/health.appBuild.signature` differs
|
||||
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`. On 2026-07-07 the port
|
||||
was still owned by an older elevated runtime, so goal scans were intentionally
|
||||
blocked by the stale-build gate. Restart the elevated app through
|
||||
`npm run dev:admin` and confirm UAC before collecting new 100-artifact evidence.
|
||||
|
||||
## Elevation And UAC
|
||||
|
||||
Use:
|
||||
@@ -98,6 +104,66 @@ For live validation, prefer a bounded scan first:
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
|
||||
```
|
||||
|
||||
The visible-inventory path remains the safest first check. The normal guided
|
||||
entry tries the read-only direct world path first:
|
||||
`B -> artifact tab -> first artifact tile`. If that does not produce a visible
|
||||
artifact detail card, it falls back to the Inventory Kamera-compatible sequence:
|
||||
`ESC -> B -> artifact tab -> first artifact tile`.
|
||||
|
||||
The normal Auto-Scan button uses a guided start. It first takes one lightweight
|
||||
preflight capture without OCR, full-frame payload, review scoring, or storing.
|
||||
If an artifact detail card is already visible, it starts the visible-inventory
|
||||
scan. Otherwise it runs the guided entry above. OCR/review/store work starts
|
||||
only after the artifact-detail preflight passes.
|
||||
Guided entry uses short state polling for the Inventory screen, artifact grid,
|
||||
and first detail card instead of waiting the full fixed delay every time; if the
|
||||
state never appears, the same timeout budget returns the last diagnostic capture.
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=auto-entry&limit=2"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2&engine=ik-traineddata"
|
||||
```
|
||||
|
||||
Those paths send only read-only navigation. `ESC` is not a universal "go to
|
||||
world" command: from the world it opens the Paimon menu, while from the
|
||||
already-open Paimon menu it returns to the world. This is why the normal
|
||||
`auto-entry` path first tries `B` directly and uses the IK-style `ESC -> B`
|
||||
fallback only when direct entry did not reach an artifact detail card.
|
||||
|
||||
The scan starts only after a valid lookup package, supported 16:9 layout,
|
||||
detected artifact grid, Genshin-client capture, and visual artifact-detail
|
||||
markers are all present. If any preflight check fails, keep using the
|
||||
visible-inventory path while tuning the entry step.
|
||||
|
||||
The visual preflight also classifies the Paimon menu. The Paimon profile/card
|
||||
grid can look like an inventory grid if only fixed 16:9 coordinates are used, so
|
||||
the scanner must reject `paimonMenu.present` before any artifact OCR, review
|
||||
sample creation, store write, or grid scan starts. The guided entry may still
|
||||
take lightweight skip-OCR captures while navigating, but those captures are only
|
||||
state evidence.
|
||||
|
||||
The same guard also runs inside the scan loop. If the app is on the main game
|
||||
screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
|
||||
without an artifact detail card, auto-scan must block instead of clicking tiles
|
||||
or trying OCR.
|
||||
After each click the loop polls the detail fingerprint with a short bounded
|
||||
budget instead of sleeping blindly. The current budget is 420 ms with 60 ms
|
||||
polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it
|
||||
does not change the loop retries or stops through the normal miss guards. If the
|
||||
card changed but remains animated, the loop now proceeds after 200 ms, matching
|
||||
Inventory Kamera's select-next-item wait more closely without removing the
|
||||
detail-change guard.
|
||||
The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
|
||||
not re-run the focus helper before every tile, which avoids an OS focus ping on
|
||||
each artifact while still relying on click readback, foreground checks, and the
|
||||
detail-card guard for safety.
|
||||
After a scroll, the loop now uses the same cheap fingerprint polling model for
|
||||
the inventory pane: it proceeds as soon as the next page fingerprint changed and
|
||||
stabilized instead of always sleeping the old fixed 760 ms settle delay. Changed
|
||||
but still animated inventory pages may proceed after 100 ms, again matching IK's
|
||||
fast-scroll wait while still blocking unchanged pages.
|
||||
|
||||
Then poll:
|
||||
|
||||
```powershell
|
||||
@@ -105,10 +171,221 @@ Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
|
||||
ConvertTo-Json -Depth 12
|
||||
```
|
||||
|
||||
Before live timing, verify that the endpoint is the current app instance:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/health" |
|
||||
ConvertTo-Json -Depth 6
|
||||
```
|
||||
|
||||
The response must include `appBuild.signature` and
|
||||
`appBuild.expectedOcrWorkerPoolSize`. If `appBuild` is missing, or
|
||||
`/scanner/status` still reports the old OCR warmup start time, the local port is
|
||||
still owned by a stale elevated Electron process. Close the old Administrator
|
||||
window/app and restart with `npm run dev:admin` before running scanner probes.
|
||||
|
||||
Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
|
||||
`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
|
||||
`activeAverageMsPerParsed`, `averageCaptureMs`, `averageOcrMs`,
|
||||
`artifactsPerMinute`, and `projectedMsFor100`. `elapsedMs` is end-to-end
|
||||
including queued writes; `activeScanMs` is the click/capture/OCR loop before
|
||||
the final store/review flush. A run only counts as speed
|
||||
evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
|
||||
together; raw click count alone is not scanner throughput. If `averageOcrMs`
|
||||
dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
|
||||
queue. If `averageCaptureMs` dominates, crop payload/capture work is the
|
||||
bottleneck.
|
||||
|
||||
The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
|
||||
temporary scan limit. It does not change the normal UI setting. The normal
|
||||
hotkeys and buttons still use the UI's configured scan limit.
|
||||
|
||||
Lookup and benchmark utility endpoints:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/status"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/regenerate"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup?engine=ik-traineddata"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=ik-traineddata"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=compare"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&profile=full"
|
||||
```
|
||||
|
||||
The benchmark endpoint measures the current Tesseract.js engine and the
|
||||
Inventory-Kamera-traineddata Tesseract.js path against the artifact crop set and
|
||||
returns timing/field counts, min/p50/p90/max timing, OCR p50/p90 timing,
|
||||
20/45/100-artifact projections, skipped-OCR count, and the active OCR worker
|
||||
pool size. It also returns per-field OCR timings under
|
||||
`ocrFieldAverages`, which is the first place to look before changing crop or
|
||||
parser behavior. Individual captures also report whether the artifact was
|
||||
detected as `sanctified`; level/substat crops are shifted in that state to match
|
||||
Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
|
||||
which omits the low-value set-effect crop, the slot crop that can be derived
|
||||
from the matched artifact piece name, and the main-stat-value crop that can be
|
||||
derived from slot, main-stat label, and level. The fast profile also uses
|
||||
Inventory Kamera's tighter substat crop height; full/manual captures keep the
|
||||
larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
|
||||
captures while keeping the detail screenshot, OCR text, crop rect metadata, and
|
||||
timings. OCR crops are passed to Tesseract as PNG buffers internally, not as
|
||||
Base64 DataURLs, to avoid encode/decode overhead in batch scans. When
|
||||
`skipOcrUnlessArtifactDetail` blocks OCR because no artifact detail card is
|
||||
visible, OCR crop preprocessing is skipped too. Auto-scan readiness and scroll
|
||||
checks use native detail/inventory fingerprints and omit preview DataURLs in
|
||||
poll captures. Fast preflight/poll captures also omit crop list construction, crop images, and lock-state
|
||||
detection unless a caller explicitly overrides that option; add
|
||||
`profile=full` to OCR every artifact detail crop for debugging. It uses the same
|
||||
artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
|
||||
skipped and the response shows `skippedOcrCaptures` instead of burning time on
|
||||
invalid crops.
|
||||
For speed, the fast auto-scan profile also skips the optional Equipped footer
|
||||
OCR. Name, level, main-stat label, and substats remain in the OCR hot path;
|
||||
slot, set, and main-stat value are derived when the lookup/parser can validate
|
||||
them. Use a full/manual capture when equipped ownership or every debug crop matters.
|
||||
Local store/review writes are serialized through an internal queue but no longer
|
||||
block the next inventory click. The scan still flushes the queue before it
|
||||
returns its final summary, so `stored` and `review` counts remain final-state
|
||||
numbers.
|
||||
The app warms the default OCR worker pool in the background after startup; check
|
||||
`/scanner/status` -> `ocrWarmup.current` before timing the first artifact. Use
|
||||
`/scanner/ocr/warmup?engine=ik-traineddata` before comparing Inventory
|
||||
Kamera-traineddata timings so the benchmark is not dominated by worker creation.
|
||||
`engine=ik-traineddata` uses Inventory Kamera's local
|
||||
`genshin_fast_09_04_21.traineddata` through Tesseract.js when the file is found
|
||||
in `data/tessdata`, `IK_TESSDATA_DIR`, `work/Inventory_Kamera`, `work/refs`,
|
||||
or the local `_ik_ref*` folders.
|
||||
`engine=compare` runs `current` and `ik-traineddata` against the same visible
|
||||
artifact detail state. The auto-scan default must stay `current` until the IK
|
||||
traineddata path wins on the same captures. For a controlled live comparison,
|
||||
start the scanner with `engine=ik-traineddata`; this only changes the OCR
|
||||
worker language for that run and leaves the default UI/hotkey path on
|
||||
`current`.
|
||||
The OCR pool defaults to four workers because the fast artifact crop set has
|
||||
four useful OCR parameter groups; set `GAA_OCR_WORKERS=1..8` before startup to
|
||||
benchmark a different worker count. Inventory Kamera's native engine pool is
|
||||
still the reference design, but the current app path remains Tesseract.js until
|
||||
native OCR is integrated and measured. Crops are scheduled across the whole
|
||||
worker pool and each worker caches its last Tesseract parameter profile; this is
|
||||
closer to Inventory Kamera's multi-engine field OCR than the earlier
|
||||
parameter-group-serial scheduler.
|
||||
|
||||
## Diagnostic Evidence
|
||||
|
||||
The Diagnose page contains a compact evidence timeline for scanner work. It logs
|
||||
runtime pings, focus attempts, key presses, entry captures, artifact-tab clicks,
|
||||
preflight failures, grid/count metadata, detail fingerprints, and detail/inventory
|
||||
screenshots. The same last events are also published through:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
|
||||
ConvertTo-Json -Depth 18
|
||||
```
|
||||
|
||||
Use this before changing scanner behavior: run the smallest failing action, read
|
||||
the evidence timeline, then decide whether the failure is focus/input, entry
|
||||
navigation, grid detection, capture quality, OCR, or parser validation.
|
||||
|
||||
If Paimon entry shows `entry key ESC` or `entry key B` with `eventsSent: 0`, the
|
||||
running `InputHelper.exe` probably predates keyboard support or is blocked. Stop
|
||||
the elevated app/helper, run `npm run helper:build`, then restart with
|
||||
`npm run dev:admin` so the app loads the rebuilt helper.
|
||||
|
||||
## Soak-Test Helper
|
||||
|
||||
After the elevated app is running and Genshin is open on the artifact inventory,
|
||||
the non-elevated terminal can drive the local dev-control endpoints and save a
|
||||
full evidence bundle:
|
||||
|
||||
```powershell
|
||||
npm run scan:soak
|
||||
```
|
||||
|
||||
The helper writes timestamped JSON snapshots and a transcript to:
|
||||
|
||||
```text
|
||||
outputs/live-soak/<timestamp>/
|
||||
```
|
||||
|
||||
Default sequence:
|
||||
|
||||
1. `/health`
|
||||
2. `/scanner/status`
|
||||
3. `/capture/smart?skipOcr=1`
|
||||
4. `/automation/probe-click?index=1`
|
||||
5. `/automation/probe-click?index=3`
|
||||
6. `/scanner/start?limit=2`
|
||||
7. `/scanner/start?limit=5`
|
||||
8. `/scanner/start?limit=10`
|
||||
9. `/scanner/start?limit=20`
|
||||
10. `/review/samples?limit=30`
|
||||
|
||||
For the actual Inventory-Kamera speed target, use the explicit goal run after
|
||||
`/health` shows the current `appBuild`:
|
||||
|
||||
```powershell
|
||||
npm run scan:goal
|
||||
npm run scan:goal:current
|
||||
npm run scan:goal:ik
|
||||
npm run scan:goal:compare
|
||||
```
|
||||
|
||||
That run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
|
||||
limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
|
||||
`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
|
||||
uses the default `current` scan engine; use `scan:goal:ik` for a native
|
||||
IK-traineddata scan pass. Use `scan:goal:compare` to run both scan engines
|
||||
back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for
|
||||
`averageMsPerParsed`, `activeAverageMsPerParsed`, `averageCaptureMs`,
|
||||
`captureP50Ms`, `captureP90Ms`, `averageOcrMs`, `ocrP50Ms`, `ocrP90Ms`,
|
||||
`averageCardReadyMs`, `averageScrollReadyMs`, `artifactsPerMinute`, and
|
||||
`projectedMsFor100`.
|
||||
The run also writes `scan-performance-assessment.json`, which groups results by
|
||||
limit, picks the best qualified engine, and labels the dominant bottleneck as
|
||||
OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
|
||||
run, parse the requested count, keep miss rate under 2%, and keep review rate
|
||||
at or below 15%; review and miss rates are penalized before active average speed
|
||||
is used as the tie-breaker.
|
||||
|
||||
The assessment ranking can be verified without Genshin or the Electron app:
|
||||
|
||||
```powershell
|
||||
npm run scan:assessment:test
|
||||
```
|
||||
|
||||
This self-test rejects synthetic runs that are fast but have too many misses or
|
||||
too many review samples, so the final IK comparison cannot be won by speed alone.
|
||||
|
||||
For the current implementation summary and IK comparison rationale, see
|
||||
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
|
||||
|
||||
Use the readiness timings to compare against Inventory Kamera's fixed waits:
|
||||
IK waits about 200 ms after selecting the next inventory item and about 100 ms
|
||||
after fast scrolls. If `averageCardReadyMs` or `averageScrollReadyMs` dominates
|
||||
the active average while OCR is already low, tune the fingerprint gate before
|
||||
touching OCR again.
|
||||
|
||||
The runner reads `APP_RUNTIME_SIGNATURE` from `electron/main.ts` and refuses
|
||||
to run against a stale Electron process when `/health.appBuild.signature` does
|
||||
not match the current source. Use `-AllowStaleBuild` only for deliberate
|
||||
debugging of an older instance.
|
||||
Current dev builds also expose `/dev/shutdown` on localhost. The start cleanup
|
||||
script calls it before falling back to `Stop-Process`, so a previous elevated
|
||||
app can shut itself down cleanly even when the caller cannot terminate an
|
||||
administrator process directly. Older builds without that endpoint still need
|
||||
manual close or a confirmed `npm run dev:admin` restart.
|
||||
|
||||
Review samples are saved as a compact summary by default so Vite does not try to
|
||||
watch large Base64 payloads under `outputs/`. Full review payloads can be saved
|
||||
with `-SaveFullReviewSamples` when needed.
|
||||
|
||||
It stops on a failed probe, blocked scan, stopped scan, or timeout unless
|
||||
`-ContinueAfterBlocked` is supplied directly:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\live-soak.ps1 -Limits 2,5 -ContinueAfterBlocked
|
||||
```
|
||||
|
||||
## Anti-Cheat And Safety Boundary
|
||||
|
||||
Do not describe the current implementation as bypassing anti-cheat. The app
|
||||
@@ -133,7 +410,9 @@ The current 16:9 layout profile is calibrated from a 1920x1080 English
|
||||
artifact-inventory capture:
|
||||
|
||||
- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
|
||||
- inventory grid: `8 x 5`
|
||||
- inventory grid: `8 x 4` safe automated targets, matching Inventory Kamera's
|
||||
32-artifact full-page model. The apparent lower fifth row is in the bottom
|
||||
control band and is intentionally not clicked during auto-scan.
|
||||
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
|
||||
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`
|
||||
- inventory count crop successfully read `2059/2400` in the live session
|
||||
@@ -152,3 +431,5 @@ Before marking an automation change done:
|
||||
5. If Genshin is available, run `/automation/probe-click?index=1`.
|
||||
6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
|
||||
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
|
||||
8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
|
||||
a non-stale `npm run scan:goal:compare` run.
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
|
||||
- [ ] The expected crop or capture behavior is clear.
|
||||
- [ ] Genshin is not accessed through memory reads, hooks, injection, or game files.
|
||||
- [ ] Auto-scan starts only from confirmed artifact inventory plus visible detail card, or blocks with a reason.
|
||||
- [ ] Capture failures are shown to the user.
|
||||
- [ ] OCR uncertainty remains inspectable in Details.
|
||||
- [ ] Parser output does not silently trust low-confidence text.
|
||||
@@ -12,6 +13,18 @@
|
||||
- [ ] `npm run build` passes.
|
||||
- [ ] Manual Smart Capture is tested when possible.
|
||||
|
||||
## IK-Speed Or OCR-Engine Claim
|
||||
|
||||
- [ ] `/health.appBuild.signature` matches the current `APP_RUNTIME_SIGNATURE`.
|
||||
- [ ] `npm run scan:assessment:test` passes.
|
||||
- [ ] The run includes `scan-performance-assessment.json`.
|
||||
- [ ] The 100-artifact run finishes cleanly.
|
||||
- [ ] Parsed count is at least the requested count.
|
||||
- [ ] Miss rate is at or below 2%.
|
||||
- [ ] Review rate is at or below 15%.
|
||||
- [ ] Speed comparison uses active scan timing plus quality, not click count alone.
|
||||
- [ ] The default OCR engine is changed only after same-capture benchmark evidence.
|
||||
|
||||
## UI Change
|
||||
|
||||
- [ ] The main workflow remains visible without unnecessary scrolling.
|
||||
|
||||
@@ -16,6 +16,7 @@ This document contains Architecture Decision Records.
|
||||
| ADR-008 | Replace the PowerShell input/capture helper with a C# sidecar | Accepted | 2026-07-05 |
|
||||
| ADR-009 | Resolution-anchored layout profiles and OCR preprocessing over color detection | Accepted | 2026-07-05 |
|
||||
| ADR-010 | Elevated dev runner and bounded live automation probes | Accepted | 2026-07-07 |
|
||||
| ADR-011 | Quality-gated Inventory Kamera comparison before OCR default changes | Accepted | 2026-07-07 |
|
||||
|
||||
## ADR-001: Build A Local Electron App First
|
||||
|
||||
@@ -286,3 +287,49 @@ Document the workflow in [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
- The implementation remains inside the allowed safety boundary: no memory
|
||||
reads, hooks, injection, game-file modification, deleting, feeding, enhancing,
|
||||
locking/unlocking, or spending resources.
|
||||
|
||||
## ADR-011: Quality-Gated Inventory Kamera Comparison Before OCR Default Changes
|
||||
|
||||
### Status
|
||||
|
||||
Accepted
|
||||
|
||||
### Context
|
||||
|
||||
The product target is not merely to click through 100 artifacts quickly. It is
|
||||
to scan the first 100 artifacts with accuracy at least as good as Inventory
|
||||
Kamera and speed equal to or better than Inventory Kamera. A faster scan that
|
||||
creates too many misses, review samples, or false positives is worse than a
|
||||
slower qualified run.
|
||||
|
||||
The app can now compare the current OCR path with Inventory Kamera's
|
||||
`genshin_fast_09_04_21.traineddata` through the same visible crop set. It also
|
||||
has hot-loop timing fields for capture, OCR, card readiness, scroll readiness,
|
||||
active scan time, and projected 100-artifact time.
|
||||
|
||||
### Decision
|
||||
|
||||
Use a quality-gated live soak and benchmark before changing the default OCR
|
||||
engine or claiming IK parity. A qualified scan result must:
|
||||
|
||||
- finish cleanly,
|
||||
- parse at least the requested count,
|
||||
- keep miss rate at or below 2%,
|
||||
- keep review rate at or below 15%,
|
||||
- report active scan timing and bottlenecks,
|
||||
- come from a runtime whose `/health.appBuild.signature` matches the current
|
||||
source.
|
||||
|
||||
`scripts/live-soak.ps1` writes the evidence bundle and
|
||||
`scan-performance-assessment.json`. `npm run scan:assessment:test` verifies that
|
||||
the ranking logic rejects fast but low-quality synthetic runs without needing
|
||||
Genshin.
|
||||
|
||||
### Consequences
|
||||
|
||||
- Speed claims cannot be based on click count or elapsed time alone.
|
||||
- A new OCR engine cannot become the default just because it is theoretically
|
||||
closer to IK; it must win the same-capture benchmark and a qualified live run.
|
||||
- Stale elevated Electron instances are treated as invalid evidence, not as a
|
||||
harmless warning.
|
||||
- The goal remains open until the 100-artifact qualified comparison is captured.
|
||||
|
||||
+14
-2
@@ -3,6 +3,8 @@
|
||||
This document is the source of truth for project intent, scope, runtime facts, and operational expectations.
|
||||
|
||||
For implementation structure, see [ARCHITECTURE.md](ARCHITECTURE.md). For engineering standards, see [CONVENTIONS.md](CONVENTIONS.md).
|
||||
For the latest Inventory-Kamera comparison work, see
|
||||
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
|
||||
|
||||
## Project Identity
|
||||
|
||||
@@ -58,6 +60,7 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
|
||||
| --- | --- | --- |
|
||||
| Safety | Never perform irreversible in-game actions. | Code review and manual test |
|
||||
| Performance | Single artifact read should feel interactive and batch scan should not stall on false progress. | Capture latency monitored manually; auto-scan stops on blocked verification |
|
||||
| IK target | First 100 artifacts should scan with accuracy at least as good as Inventory Kamera and equal or better speed. | `npm run scan:goal:compare` quality-gated report |
|
||||
| Privacy | Captures and parsed data stay local by default. | No remote upload in scanner path |
|
||||
| Reliability | Uncertain OCR must be visible to the user. | Confidence and details view |
|
||||
| Learning loop | Scanner mistakes should become reusable local review samples. | `review-samples.jsonl` |
|
||||
@@ -95,6 +98,9 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
|
||||
- The parser already uses known sets, pieces, slots, stat aliases, set aliases, character aliases, and derived slot/set mapping.
|
||||
- Review samples, learned replacements, parser notes, and stored artifacts already persist locally.
|
||||
- The auto-scan loop is no longer a naive click spammer: it has preflight, verification, miss handling, page fingerprinting, and stop conditions.
|
||||
- The scanner now has an Inventory-Kamera comparison path: 32 safe artifact
|
||||
targets per page, lookup-derived fields, fast OCR crop profile, current vs.
|
||||
IK-traineddata benchmark endpoint, and a quality-gated live soak runner.
|
||||
- Elevated live automation is validated in the current dev environment:
|
||||
`/automation/probe-click?index=1` changed the selected artifact and
|
||||
`/scanner/start?limit=2` completed with 2/2 verified reads and 0 misses.
|
||||
@@ -108,6 +114,9 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
|
||||
- Learned fixes currently focus on text replacements; they do not yet update crop offsets, UI profile variants, or scanner targeting rules in a structured way.
|
||||
- The scan page is cleaner than before, but it still exposes too much operator/debug state in the main flow.
|
||||
- Recommendations and build logic exist, but the scanner is not yet reliable enough to make them the core focus.
|
||||
- The latest source has not yet completed the final 100-artifact live comparison
|
||||
because the current dev-control port is still owned by a stale elevated
|
||||
Electron process. Live timing must wait for a UAC-approved restart.
|
||||
|
||||
### Current product conclusion
|
||||
|
||||
@@ -233,8 +242,11 @@ Status:
|
||||
2. Tighten the game data generator and parser contract, then backfill regression tests from real bad samples.
|
||||
3. Continue moving auto-scan behavior out of `App.tsx` and into isolated scanner modules.
|
||||
4. Soak-test the elevated C# helper automation path with gradually larger scan limits and page scroll transitions.
|
||||
5. Extend the learning system from text-only fixes into crop/UI profile tuning.
|
||||
6. Resume recommendation work only when scan accuracy is consistently trustworthy.
|
||||
5. Run `npm run scan:goal:compare` after `/health.appBuild.signature` matches
|
||||
the current source and use the quality-gated 100-artifact report as the IK
|
||||
target evidence.
|
||||
6. Extend the learning system from text-only fixes into crop/UI profile tuning.
|
||||
7. Resume recommendation work only when scan accuracy is consistently trustworthy.
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -2,12 +2,15 @@
|
||||
|
||||
Field-level accuracy measurement for the artifact OCR parser. This is the gate
|
||||
every OCR, crop, layout, or parser change runs against (see ADR-007).
|
||||
It is necessary but not sufficient for the IK target: live scan speed and
|
||||
review/miss rates are measured by `npm run scan:goal:compare`.
|
||||
|
||||
## Run it
|
||||
|
||||
```powershell
|
||||
npm run eval # full accuracy report for the seed corpus
|
||||
npm test # runs the eval gate alongside the rest of the suite
|
||||
npm run scan:assessment:test # verifies quality-first scan ranking logic
|
||||
```
|
||||
|
||||
The report prints exact-match rate, overall field accuracy, a per-field
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
# Scanner IK Progress Report - 2026-07-07
|
||||
|
||||
This report summarizes the scanner/OCR work toward the current target:
|
||||
scan the first 100 artifacts with accuracy at least as good as Inventory Kamera
|
||||
and speed equal to or better than Inventory Kamera, without memory reads, hooks,
|
||||
injection, game-file modification, or unsafe in-game actions.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The scanner has moved from a fragile OCR-first prototype toward an
|
||||
Inventory-Kamera-style artifact scanner:
|
||||
|
||||
- Artifact scan is now the first-class path.
|
||||
- Auto-scan starts only after a validated artifact inventory/detail preflight.
|
||||
- Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and
|
||||
missing-detail states block before OCR/store/review work.
|
||||
- OCR uses a fast artifact profile that skips low-value fields and derives
|
||||
slot, set, and main-stat value through lookup constraints when safe.
|
||||
- The OCR worker pool, field crop split, page model, scroll model, and readiness
|
||||
waits now mirror the relevant IK design choices more closely.
|
||||
- Diagnostics now preserve state evidence, timings, screenshots where useful,
|
||||
entry events, focus/input events, preflight failures, and scan-loop reasons.
|
||||
- A live soak runner now measures throughput and quality, compares current vs.
|
||||
IK-traineddata engines, and refuses to run against stale Electron builds.
|
||||
|
||||
The requested final goal is not proven complete yet. The current live dev port is
|
||||
still owned by an older elevated Electron instance, so the latest code cannot be
|
||||
truthfully benchmarked against IK until the app is restarted with UAC approval
|
||||
and `npm run scan:goal:compare` completes a qualified 100-artifact run.
|
||||
|
||||
## What Changed
|
||||
|
||||
### Lookup and validation
|
||||
|
||||
- `scripts/generate-genshin-data.cjs` was extended into a stricter lookup
|
||||
package generator.
|
||||
- `src/lib/genshinLookup.ts` provides pure matching and validation for sets,
|
||||
pieces, slots, stats, characters, aliases, GOOD keys, source version, and
|
||||
validation summaries.
|
||||
- Auto-scan preflight blocks if the lookup package is invalid.
|
||||
|
||||
Why this matters:
|
||||
|
||||
IK succeeds partly because raw OCR is not trusted by itself. The app now follows
|
||||
the same principle: OCR text is normalized, matched, constrained, and derived
|
||||
against a canonical package before it is accepted.
|
||||
|
||||
### OCR and parser pipeline
|
||||
|
||||
- Artifact detail crops are split into field-specific regions:
|
||||
name, slot, main-stat label, main-stat value, level, substats, set effects,
|
||||
equipped/footer, lock, and rarity.
|
||||
- Fast auto-scan profile skips lower-value OCR work:
|
||||
set effects, slot crop, main-stat value crop, equipped footer, crop images,
|
||||
full-frame payloads, and inventory preview payloads.
|
||||
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
|
||||
constraints make that safe.
|
||||
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
|
||||
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
|
||||
- Exact visual duplicates are skipped before OCR.
|
||||
|
||||
Why this matters:
|
||||
|
||||
The fast path spends OCR only on fields that materially change the artifact
|
||||
identity or review decision. That is closer to IK's queued crop model than a
|
||||
manual-debug capture that OCRs every visible thing.
|
||||
|
||||
### Engine comparison and benchmark path
|
||||
|
||||
- `/scanner/ocr/warmup?engine=current|ik-traineddata` warms OCR workers.
|
||||
- `/scanner/benchmark-ocr?engine=current|ik-traineddata|compare` benchmarks the
|
||||
same visible artifact crops.
|
||||
- Auto-scan accepts `ocrEngine: "current" | "ik-traineddata"`.
|
||||
- `scripts/live-soak.ps1` supports:
|
||||
- `npm run scan:goal`
|
||||
- `npm run scan:goal:current`
|
||||
- `npm run scan:goal:ik`
|
||||
- `npm run scan:goal:compare`
|
||||
- `scan-performance-assessment.json` ranks runs by quality first and speed
|
||||
second.
|
||||
|
||||
Important rule:
|
||||
|
||||
A fast engine cannot win if it has too many misses or too much review. A
|
||||
qualified winner must finish cleanly, parse the requested count, keep miss rate
|
||||
at or below 2%, and keep review rate at or below 15%.
|
||||
|
||||
### Auto-scan entry and safety
|
||||
|
||||
- The normal auto button runs a guided start:
|
||||
1. focus Genshin,
|
||||
2. run a lightweight no-OCR preflight,
|
||||
3. if artifact detail is visible, use visible-inventory mode,
|
||||
4. otherwise try direct `B -> artifact tab -> first artifact tile`,
|
||||
5. if needed, fall back to the IK-style ESC/B inventory sequence,
|
||||
6. start OCR only after artifact grid and detail card pass preflight.
|
||||
- Entry captures are state evidence only. They do not create review samples,
|
||||
store artifacts, or run artifact OCR before the detail preflight passes.
|
||||
- Entry waits now poll for state readiness instead of always sleeping the full
|
||||
fixed delay.
|
||||
- Scan loop also rechecks the same safety boundary after each click and scroll.
|
||||
|
||||
Why this matters:
|
||||
|
||||
The previous failure mode was dangerous from a product-quality point of view:
|
||||
when the game was not in artifact inventory, the scanner could still take
|
||||
screenshots and try to read artifacts. The current path is explicitly blocked
|
||||
outside the artifact inventory/detail state.
|
||||
|
||||
### Scan loop and speed
|
||||
|
||||
- Grid model uses IK's 32-artifact visible page concept (`8 x 4`) instead of
|
||||
clicking the risky lower band.
|
||||
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
|
||||
duplicate reads after scroll.
|
||||
- Card readiness uses detail fingerprint polling:
|
||||
max 420 ms, 60 ms polls, changed cards may proceed after 200 ms.
|
||||
- Scroll readiness uses inventory fingerprint polling:
|
||||
max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
|
||||
- Store/review writes are queued so the next tile can be clicked before disk
|
||||
writes finish. The queue is still flushed before final summary.
|
||||
- Focus is done once at scan start; hot-loop captures do not refocus every tile.
|
||||
|
||||
Why this matters:
|
||||
|
||||
IK uses fixed waits around 200 ms after selecting inventory items and 100 ms
|
||||
after fast scrolls. The app now keeps those as safety ceilings/acceptance points
|
||||
while allowing earlier continuation when visual evidence is ready.
|
||||
|
||||
### Diagnostics and logging
|
||||
|
||||
- Scanner diagnostics now capture timeline events for runtime, focus, keypress,
|
||||
entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid,
|
||||
counts, detail/page fingerprints, and failure reasons.
|
||||
- `/scanner/status` publishes recent diagnostic evidence.
|
||||
- Review sample output is compact by default so Vite does not watch large Base64
|
||||
payloads during live soak runs.
|
||||
- The live runner writes timestamped JSON snapshots, CSV summaries, transcript,
|
||||
benchmark data, and performance assessment files under `outputs/live-soak/`.
|
||||
|
||||
Why this matters:
|
||||
|
||||
Future scanner bugs can be debugged from captured evidence instead of relying
|
||||
only on a human description of what appeared on screen.
|
||||
|
||||
## Vorgehensweise
|
||||
|
||||
1. Read the local Inventory Kamera reference under `work/Inventory_Kamera`.
|
||||
2. Copy the proven concepts, not the entire implementation:
|
||||
32 artifact targets per page, fixed coordinate ratios, queued OCR work,
|
||||
short item/scroll waits, read-only inventory navigation, and Tesseract
|
||||
traineddata comparison.
|
||||
3. Harden the app's own architecture around those concepts:
|
||||
pure lookup API, parser derivation, renderer scan orchestration,
|
||||
Electron capture/OCR boundary, sidecar input helper, and diagnostics.
|
||||
4. Add tests before trusting behavior:
|
||||
lookup validation, parser derivation, auto-entry planning/preflight,
|
||||
card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
|
||||
5. Add live tooling before claiming performance:
|
||||
bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON
|
||||
assessment, and quality-first comparison.
|
||||
|
||||
## Tests and Evidence
|
||||
|
||||
Latest repo validation after the recent changes:
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| PowerShell parse for `scripts/live-soak.ps1` | Passed |
|
||||
| `npm run scan:assessment:test` | Passed |
|
||||
| `npm run lint` | Passed |
|
||||
| Focused scanner tests | Passed |
|
||||
| `npm test` | Passed, 171 tests |
|
||||
| OCR eval seed corpus | 100% exact match, 100% field accuracy, 100% critical fields |
|
||||
| `npm run build` | Passed |
|
||||
| `git diff --check` | Passed |
|
||||
|
||||
Live evidence already collected earlier on 2026-07-07:
|
||||
|
||||
- Probe click changed artifact detail successfully.
|
||||
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
|
||||
- Limit 20 live soak completed on the first visible page.
|
||||
- Limit 45 live soak crossed into a scrolled page.
|
||||
|
||||
Current live limitation:
|
||||
|
||||
- `/health` still reports an older elevated build:
|
||||
`2026-07-07-ocr-pool4-hotloop-no-refocus`.
|
||||
- Current source expects:
|
||||
`2026-07-07-ik32-fastsubstats-active-timing`.
|
||||
- The live soak runner correctly refuses to benchmark the stale runtime.
|
||||
- A UAC restart attempt was canceled, so the latest code is not yet live.
|
||||
|
||||
## Inventory Kamera Comparison
|
||||
|
||||
| Area | Inventory Kamera | Current app status |
|
||||
| --- | --- | --- |
|
||||
| Safe scope | Reads inventory through screen/click automation | Same safety boundary: screen capture and read-only input only |
|
||||
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
|
||||
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
|
||||
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
|
||||
| Item wait | About 200 ms fixed wait | Fingerprint polling, accepts changed card after 200 ms |
|
||||
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
|
||||
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
|
||||
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
|
||||
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
|
||||
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
|
||||
| 100-artifact proof | Reference target | Not yet proven on latest app build |
|
||||
|
||||
What is theoretically better than before:
|
||||
|
||||
- The app no longer spends OCR on invalid screens.
|
||||
- It no longer treats click count as scanner success.
|
||||
- It can prove whether `current` or `ik-traineddata` wins on the same capture
|
||||
set instead of changing engines blindly.
|
||||
- It can reject fast-but-wrong results automatically.
|
||||
- It can identify whether the bottleneck is OCR, capture, card readiness, or
|
||||
scroll readiness.
|
||||
|
||||
What is not yet proven better than IK:
|
||||
|
||||
- Native Tesseract speed is not integrated as the default.
|
||||
- The latest code has not completed the 100-artifact live run.
|
||||
- Review rate and miss rate on the user's real inventory still need the new
|
||||
live report.
|
||||
|
||||
## Theoretical Runtime Flow
|
||||
|
||||
For the intended 100-artifact comparison:
|
||||
|
||||
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
|
||||
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
|
||||
3. Warm current and IK-traineddata OCR workers.
|
||||
4. Run a small bounded probe from the artifact inventory.
|
||||
5. Run `npm run scan:goal:compare`.
|
||||
6. For each engine and limit (`2, 5, 20, 45, 100`):
|
||||
- focus Genshin once,
|
||||
- verify lookup and layout,
|
||||
- verify artifact grid and detail card,
|
||||
- click one safe grid target,
|
||||
- poll detail fingerprint,
|
||||
- skip duplicate visuals,
|
||||
- OCR only the fast artifact crop set,
|
||||
- parse through lookup constraints,
|
||||
- queue store/review writes,
|
||||
- scroll with inventory fingerprint polling,
|
||||
- stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or
|
||||
repeated misses.
|
||||
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance
|
||||
assessment.
|
||||
8. Declare a winner only if the 100-artifact run is qualified by quality.
|
||||
|
||||
Expected bottleneck sequence:
|
||||
|
||||
- If OCR dominates, compare `current` vs `ik-traineddata`, crop count, and
|
||||
worker pool size.
|
||||
- If capture dominates, reduce payload construction and preview/crop image work.
|
||||
- If card-ready dominates, tune the detail fingerprint gate.
|
||||
- If scroll-ready dominates, tune page fingerprint polling and scroll notches.
|
||||
|
||||
## What Is Better Than Before
|
||||
|
||||
- Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or
|
||||
menu screens.
|
||||
- Paimon/menu detection blocks before scan-loop OCR or writes.
|
||||
- The normal button is one coherent guided flow instead of a separate "get to
|
||||
inventory first, then scan" workflow.
|
||||
- The app has a real lookup layer instead of raw OCR plus scattered hardcoded
|
||||
assumptions.
|
||||
- The scanner can compare OCR engines without changing the default blindly.
|
||||
- Performance reports now include quality decisions, not just elapsed time.
|
||||
- Diagnostics are concrete enough for later self-troubleshooting.
|
||||
- Stale elevated runtime is detected before live soak, avoiding false evidence.
|
||||
|
||||
## Risks and Remaining Work
|
||||
|
||||
1. Restart with UAC and run the latest build live.
|
||||
2. Run `npm run scan:goal:compare` from a visible artifact inventory.
|
||||
3. If the 100-artifact winner is not qualified, inspect:
|
||||
`scan-performance-assessment.json`, review samples, diagnostic timeline, and
|
||||
field timings.
|
||||
4. If `ik-traineddata` wins but Tesseract.js is still slow, evaluate native
|
||||
Tesseract integration.
|
||||
5. Grow the eval corpus with confirmed real review samples before tightening
|
||||
parser thresholds further.
|
||||
6. Validate a positive locked-artifact sample.
|
||||
7. Keep recommendations secondary until scanner quality is proven.
|
||||
|
||||
## Definition of Done for the IK Target
|
||||
|
||||
The goal is complete only when current evidence proves all of these:
|
||||
|
||||
- The app is running the latest runtime signature.
|
||||
- The 100-artifact scan finishes cleanly.
|
||||
- Parsed count is at least 100.
|
||||
- Miss rate is at or below 2%.
|
||||
- Review rate is at or below 15%.
|
||||
- The run is equal to or faster than the recorded IK reference or the selected
|
||||
IK-traineddata/native baseline on the same machine and inventory setup.
|
||||
- The evidence bundle is saved under `outputs/live-soak/`.
|
||||
- Any chosen default OCR engine is backed by the same-capture benchmark.
|
||||
@@ -5,6 +5,22 @@ Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in
|
||||
live automation runbook, see
|
||||
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
|
||||
## Current IK-Speed Target Status
|
||||
|
||||
See [scanner-ik-progress-report.md](scanner-ik-progress-report.md) for the full
|
||||
report.
|
||||
|
||||
Current status:
|
||||
|
||||
- The scanner architecture now follows the relevant Inventory Kamera model:
|
||||
32 artifact targets per page, lookup-derived fields, fast artifact OCR profile,
|
||||
short readiness gates, page-overlap planning, and queued OCR/store work.
|
||||
- The live runner can compare `current` and `ik-traineddata` engines and rejects
|
||||
runs that are fast but fail miss/review quality thresholds.
|
||||
- The final 100-artifact IK target is not proven yet. The dev-control port is
|
||||
currently owned by an older elevated Electron build, and the runner correctly
|
||||
refuses stale timing evidence until the app is restarted with UAC approval.
|
||||
|
||||
## Done (implemented, unit-tested, build green)
|
||||
|
||||
- **OCR eval harness** — `src/eval/`, `npm run eval`, gate in `npm test`. See
|
||||
@@ -37,6 +53,62 @@ live automation runbook, see
|
||||
- **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live
|
||||
with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records,
|
||||
2 review samples, and 0 misses.
|
||||
- **Auto-scan OCR performance pass** - auto-scan captures now use an artifact
|
||||
OCR mode that skips inventory-count OCR on each tile, keeps equipped-character
|
||||
OCR, raises the substat crop to catch artifact level, stores automatic review
|
||||
samples without full-screen/inventory screenshots, reads only the tail of large
|
||||
JSONL files, avoids review noise when only level/equipped is missing, starts
|
||||
the scan with an OCR-free preflight capture, skips exact visual duplicates
|
||||
before OCR, prevents repeated startup review reprocessing, omits full-frame
|
||||
and inventory-preview Base64 payloads from tile captures, and applies
|
||||
crop-specific Tesseract page-segmentation/whitelist parameters.
|
||||
- **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the
|
||||
dev-control health/status, smart-capture, probe-click, bounded scan, and
|
||||
review-tail endpoints and writes evidence to `outputs/live-soak/`. On
|
||||
2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10,
|
||||
and 20 against the elevated running app. The limit 20 run finished `done` with
|
||||
20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1
|
||||
miss, and 1 page.
|
||||
- **Scroll/page-transition live soak** - after the helper and loop fixes,
|
||||
`scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture`
|
||||
completed `done` on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35
|
||||
stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the
|
||||
scanner can cross from the first visible page into a scrolled page in the live
|
||||
1920x1080 setup.
|
||||
- **Lookup package layer** - `scripts/generate-genshin-data.cjs` now emits
|
||||
normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source
|
||||
version metadata, generated time, and validation summary. `src/lib/genshinLookup.ts`
|
||||
provides pure matching and validation APIs, and the scanner status/dev-control
|
||||
path exposes lookup validity. Auto-scan preflight blocks when the lookup package
|
||||
is invalid.
|
||||
- **Inventory-Kamera-style field split** - artifact detail crops now separate
|
||||
name, slot, main-stat label, main-stat value, level, substats, set effects, and
|
||||
footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives
|
||||
slot/set/main-stat through lookup constraints before falling back to review.
|
||||
- **Paimon-menu auto-entry scaffold** - auto-scan supports
|
||||
`scanEntryMode: "paimon-menu"` and `/scanner/start?entry=paimon-menu&limit=N`.
|
||||
The entry sends only read-only navigation (`ESC`, `B`, artifact-tab click),
|
||||
then requires a valid lookup, supported layout, and detected artifact grid
|
||||
before the scan loop starts. The existing visible-inventory start remains the
|
||||
fallback/debug path.
|
||||
- **OCR benchmark endpoint scaffold** - `/scanner/benchmark-ocr?limit=N` captures
|
||||
identical artifact crops with the current engine and returns timing/field counts.
|
||||
`/scanner/benchmark-ocr?engine=compare` can also compare the local
|
||||
Inventory-Kamera-traineddata Tesseract.js path when
|
||||
`genshin_fast_09_04_21.traineddata` is present in `data/tessdata`, `work/`, or
|
||||
`IK_TESSDATA_DIR`. The OCR worker pool defaults to four workers and can be
|
||||
tuned with `GAA_OCR_WORKERS=1..8`. Native Tesseract is still not the default
|
||||
and should only replace `tesseract.js` after the benchmark proves it faster
|
||||
and more accurate on the same crops.
|
||||
- **Quality-gated live comparison** - `scripts/live-soak.ps1` now supports
|
||||
goal runs for `current`, `ik-traineddata`, and `compare`, writes CSV/JSON
|
||||
summaries, groups results by limit, identifies timing bottlenecks, and rejects
|
||||
winners that miss the requested count, exceed 2% misses, or exceed 15% review.
|
||||
`npm run scan:assessment:test` verifies this ranking logic without Genshin.
|
||||
- **State-polled guided entry** - the guided auto-entry waits for Inventory,
|
||||
artifact grid, and first detail card evidence instead of sleeping the full
|
||||
fixed delay every time. OCR/review/store work still starts only after artifact
|
||||
detail preflight passes.
|
||||
|
||||
## Remaining — needs the live environment or a UI pass
|
||||
|
||||
@@ -46,12 +118,27 @@ resolution or without UI work best tested live:
|
||||
1. **Validate/tune OCR preprocessing** on more real captures — confirm invert +
|
||||
threshold + upscale factor help (not hurt) actual Tesseract reads. The
|
||||
text-level eval harness cannot measure image preprocessing.
|
||||
2. **Validate locked=true** against a known locked artifact — unlocked/grey lock
|
||||
2. **Wire and benchmark native IK-traineddata OCR** against the same crop set.
|
||||
The current benchmark can use IK-traineddata through Tesseract.js; native
|
||||
Tesseract integration remains the next implementation step before any engine
|
||||
default changes.
|
||||
3. **Validate guided entry live** from world, visible inventory, and Paimon/menu
|
||||
states with limits 2, 20, and 45. Confirm the artifact-tab coordinate in the
|
||||
user's current 16:9 layout and keep `visible-inventory` as fallback if the
|
||||
menu path is blocked.
|
||||
4. **Validate locked=true** against a known locked artifact — unlocked/grey lock
|
||||
was live-checked; a gold locked icon still needs a positive sample.
|
||||
|
||||
3. **Broader scan soak test** — after the bounded two-item live scan passed,
|
||||
5. **Broader scan soak test** — after the bounded two-item live scan passed,
|
||||
the next automation validation should increase the limit gradually and watch
|
||||
for repeated pages, scroll behavior, duplicate handling, and OCR review rate.
|
||||
6. **100-artifact IK comparison** — after `/health.appBuild.signature` matches
|
||||
current source, run `npm run scan:goal:compare` and compare qualified
|
||||
100-artifact results.
|
||||
|
||||
Visible-page limits up to 20 and a scroll/page-transition limit of 45 have
|
||||
passed. The remaining soak work is now OCR accuracy, review-rate reduction, and
|
||||
larger runs after the review corpus has grown.
|
||||
|
||||
## Grow the eval corpus
|
||||
|
||||
|
||||
Reference in New Issue
Block a user