feat(scanner): add native artifact pipeline
Add native IK-style capture processing, Artifact Inventory, explicit promotion and single-result review. Confirm the three live OCR corrections in the eval corpus and preserve extraction/value separation.
This commit is contained in:
AzuTear
+101
-467
@@ -9,34 +9,31 @@ Validated live with Genshin open in the artifact inventory at 1920x1080,
|
||||
English UI:
|
||||
|
||||
- `npm run dev:admin` starts the app elevated after the user confirms UAC.
|
||||
- Runtime status reported `isElevated: true`, `genshinFound: true`, and
|
||||
- Runtime status reports `isElevated: true`, `genshinFound: true`, and
|
||||
`targetProcess: "GenshinImpact"`.
|
||||
- The safe probe endpoint `/automation/probe-click?index=1` focused Genshin,
|
||||
moved the cursor to the second visible inventory tile, clicked it, and changed
|
||||
the artifact detail panel fingerprint.
|
||||
- Probe result: `clicked: true`, `inputBlocked: false`,
|
||||
`foregroundProcess: "GenshinImpact"`, and `changed: true`.
|
||||
- A bounded live auto-scan via `/scanner/start?limit=2` completed with:
|
||||
`clicked: 2`, `attempted: 2`, `verified: 2`, `parsed: 2`, `stored: 2`,
|
||||
`review: 2`, `misses: 0`, `status: "done"`.
|
||||
- On 2026-07-08, a visible-inventory 50-artifact run completed with `50/50`
|
||||
parsed and stored, `0` review, `0` duplicates, and `0` misses. Throughput
|
||||
was still slow at `61765 ms` elapsed (`1235 ms/artifact`).
|
||||
- On 2026-07-09, the current-engine visible-inventory path completed
|
||||
`/scanner/start?entry=visible-inventory&limit=20&engine=current` with
|
||||
`20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and
|
||||
`0` misses in `8047 ms` elapsed (`402 ms/artifact`). A same-session artifact
|
||||
detail capture also persisted an equipped footer as `equipped: "Citlali"` and
|
||||
an unlocked grey lock as `locked: false`.
|
||||
- `/automation/probe-click?index=1` performs one read-only inventory tile
|
||||
selection click.
|
||||
- The visible-inventory auto-scan path is the production baseline. It requires
|
||||
the Artifact inventory to already be open with a visible artifact detail card.
|
||||
- On 2026-07-09, `/scanner/start?entry=visible-inventory&limit=100` completed
|
||||
with `100/100` attempted, verified, and parsed, `98` stored, `2` duplicates,
|
||||
`0` review samples, `0` misses, `4` pages, and `393 ms/artifact`.
|
||||
- On 2026-07-09, `npm run scan:native:smoke` completed against runtime
|
||||
signature `2026-07-09-native-ik-visual-probe`: native visual preflight passed
|
||||
at 1920x1080 with `32` grid targets, guarded probe changed the detail panel,
|
||||
native capture wrote `2/2` artifact card crops in `510 ms`, and post-capture
|
||||
processing parsed `2/2` with `0` review, `0` errors, `queueConcurrency: 2`,
|
||||
and `persisted: false`.
|
||||
- The same native path later completed dry 20/50/100-item runs. The final run
|
||||
captured `100/100` crops across 4 pages in `24,673 ms`, parsed `100/100`,
|
||||
produced `0` errors, routed 3 implausible OCR values into Review, and wrote
|
||||
nothing to the artifact store. Full evidence and the parser/timestamp fixes
|
||||
are recorded in
|
||||
[NATIVE_SCANNER_VALIDATION_2026-07-09.md](NATIVE_SCANNER_VALIDATION_2026-07-09.md).
|
||||
|
||||
This proves that the current elevated app plus helper path can deliver mouse
|
||||
movement and click input to the focused Genshin client in this environment.
|
||||
|
||||
Latest-source timing is not proven while `/health.appBuild.signature` differs
|
||||
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`, or after source changes
|
||||
that have not been loaded by a fresh elevated runtime. Restart the elevated app
|
||||
through `npm run dev:admin` and confirm UAC before collecting new 50/100
|
||||
artifact evidence.
|
||||
The old alternate OCR-engine and comparison paths have been retired. The app now
|
||||
runs the current OCR/capture path only. Performance claims should be stated as
|
||||
current-run repeatability evidence, not as external-tool parity.
|
||||
|
||||
## Elevation And UAC
|
||||
|
||||
@@ -55,8 +52,7 @@ outputs/admin-start/admin-dev.log
|
||||
```
|
||||
|
||||
The user must confirm the Windows UAC prompt. The app cannot and must not click
|
||||
the Secure Desktop UAC prompt for itself. After confirmation, the app can verify
|
||||
its own runtime through the dev status endpoint.
|
||||
the Secure Desktop UAC prompt for itself.
|
||||
|
||||
Useful checks:
|
||||
|
||||
@@ -70,11 +66,12 @@ Expected runtime facts before automatic scan:
|
||||
- `isElevated: true`
|
||||
- `genshinFound: true`
|
||||
- `targetProcess: "GenshinImpact"`
|
||||
- hotkeys registered
|
||||
- `/health.appBuild.signature` matches `APP_RUNTIME_SIGNATURE` in
|
||||
`electron/main.ts`
|
||||
|
||||
## Mouse And Click Validation
|
||||
|
||||
Use the probe before broad auto-scan work:
|
||||
Use a probe before broad auto-scan work:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
|
||||
@@ -86,8 +83,8 @@ feed, enhance, lock, unlock, spend, or modify game resources.
|
||||
|
||||
Interpretation:
|
||||
|
||||
- `click.ok: true`, `clicked: true`, `inputBlocked: false` means Windows did not
|
||||
block SendInput/UIPI in the current configuration.
|
||||
- `click.ok: true`, `clicked: true`, and `inputBlocked: false` means Windows did
|
||||
not block SendInput/UIPI in the current configuration.
|
||||
- `focused: true` and `foregroundProcess: "GenshinImpact"` means the click was
|
||||
sent while Genshin was foreground.
|
||||
- `changed: true` means the detail panel changed after the click.
|
||||
@@ -95,83 +92,14 @@ Interpretation:
|
||||
neighboring artifacts render identically; retry with another `index`, `row`,
|
||||
or `col`.
|
||||
|
||||
Examples:
|
||||
|
||||
```powershell
|
||||
# Second visible tile
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1"
|
||||
|
||||
# Specific grid cell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?row=0&col=3"
|
||||
```
|
||||
|
||||
## Bounded Live Auto-Scan
|
||||
|
||||
For live validation, prefer a bounded scan first:
|
||||
Start with a tiny bounded run:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"
|
||||
```
|
||||
|
||||
The visible-inventory path is the merge-relevant safe path. It requires the
|
||||
Artifact inventory to already be open with a visible artifact detail card.
|
||||
|
||||
The normal Auto-Scan button uses a guided start. It first takes one lightweight
|
||||
preflight capture without OCR, full-frame payload, review scoring, or storing.
|
||||
If an artifact detail card is already visible, it starts the visible-inventory
|
||||
scan. Otherwise it blocks with an operator-facing status and asks the user to
|
||||
open the Artifact inventory with a visible detail card. OCR/review/store work
|
||||
starts only after the artifact-detail preflight passes.
|
||||
|
||||
The explicit Dev-Control entry modes below remain available for targeted
|
||||
experiments only. They send read-only navigation, but they are not the
|
||||
merge-ready default because live testing showed that `auto-entry` can leave the
|
||||
app in the Paimon menu when the starting state is not what the choreography
|
||||
expects.
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=auto-entry&limit=2"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2&engine=ik-traineddata"
|
||||
```
|
||||
|
||||
Those paths send only read-only navigation. `ESC` is not a universal "go to
|
||||
world" command: from the world it opens the Paimon menu, while from the
|
||||
already-open Paimon menu it returns to the world. This is why the normal
|
||||
`auto-entry` path first tries `B` directly and uses the IK-style `ESC -> B`
|
||||
fallback only when direct entry did not reach an artifact detail card.
|
||||
|
||||
The scan starts only after a valid lookup package, supported 16:9 layout,
|
||||
detected artifact grid, Genshin-client capture, and visual artifact-detail
|
||||
markers are all present. If any preflight check fails, keep using the
|
||||
visible-inventory path while tuning the entry step.
|
||||
|
||||
The visual preflight also classifies the Paimon menu. The Paimon profile/card
|
||||
grid can look like an inventory grid if only fixed 16:9 coordinates are used, so
|
||||
the scanner must reject `paimonMenu.present` before any artifact OCR, review
|
||||
sample creation, store write, or grid scan starts. The guided entry may still
|
||||
take lightweight skip-OCR captures while navigating, but those captures are only
|
||||
state evidence.
|
||||
|
||||
The same guard also runs inside the scan loop. If the app is on the main game
|
||||
screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
|
||||
without an artifact detail card, auto-scan must block instead of clicking tiles
|
||||
or trying OCR.
|
||||
After each click the loop now performs one fast artifact capture/OCR pass and
|
||||
uses that capture's detail fingerprint to verify that the selected artifact
|
||||
changed. This removes the old separate card-ready capture from the hot path. If
|
||||
the detail fingerprint is unchanged, the loop retries once and then follows the
|
||||
normal miss/block guards.
|
||||
The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
|
||||
not re-run the focus helper before every tile, which avoids an OS focus ping on
|
||||
each artifact while still relying on click readback, foreground checks, and the
|
||||
detail-card guard for safety.
|
||||
After a scroll, the loop now uses the same cheap fingerprint polling model for
|
||||
the inventory pane: it proceeds as soon as the next page fingerprint changed and
|
||||
stabilized instead of always sleeping the old fixed 760 ms settle delay. Changed
|
||||
but still animated inventory pages may proceed after 100 ms, again matching IK's
|
||||
fast-scroll wait while still blocking unchanged pages.
|
||||
|
||||
Then poll:
|
||||
|
||||
```powershell
|
||||
@@ -179,219 +107,57 @@ Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
|
||||
ConvertTo-Json -Depth 12
|
||||
```
|
||||
|
||||
Before live timing, verify that the endpoint is the current app instance:
|
||||
The scan starts only after a valid lookup package, supported 16:9 layout,
|
||||
detected artifact grid, Genshin-client capture, and visual artifact-detail
|
||||
markers are all present. If any preflight check fails, keep using the
|
||||
visible-inventory path and fix the blocking evidence before trying broader
|
||||
runs.
|
||||
|
||||
The loop verifies detail fingerprints after clicks, retries one unchanged detail
|
||||
read, stores parsed artifacts in a batch, and flushes pending writes before the
|
||||
final summary. Stats separate clicked, attempted, verified, parsed, stored,
|
||||
review, duplicates, and misses so progress is not confused with successful
|
||||
reads.
|
||||
|
||||
For the native IK-style path, prefer the dedicated smoke runner:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/health" |
|
||||
ConvertTo-Json -Depth 6
|
||||
npm run scan:native:smoke
|
||||
npm run scan:native:smoke:5
|
||||
```
|
||||
|
||||
The response must include `appBuild.signature` and
|
||||
`appBuild.expectedOcrWorkerPoolSize`. If `appBuild` is missing, or
|
||||
`/scanner/status` still reports the old OCR warmup start time, the local port is
|
||||
still owned by a stale elevated Electron process. Close the old Administrator
|
||||
window/app and restart with `npm run dev:admin` before running scanner probes.
|
||||
It checks `/health`, `/scanner/native/data`,
|
||||
`/scanner/native/preflight?category=artifacts` including the native visual
|
||||
blank-capture guard, runs the guarded
|
||||
`/automation/probe-click?index=1`, starts `/scanner/start?limit=N&category=artifacts`,
|
||||
waits for the native helper to finish, runs `/scanner/native/process` with
|
||||
`persist=0`, and loads `/scanner/native/results`. Evidence is written to:
|
||||
|
||||
Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
|
||||
`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
|
||||
`activeAverageMsPerParsed`, `averageCaptureMs`,
|
||||
`averageCaptureRoundTripMs`, `averageCaptureRoundTripOverheadMs`,
|
||||
`averageOcrMs`, `artifactsPerMinute`, and `projectedMsFor100`.
|
||||
`elapsedMs` is end-to-end including queued writes; `activeScanMs` is the
|
||||
click/capture/OCR loop before the final store/review flush. A run only counts as speed
|
||||
evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
|
||||
together; raw click count alone is not scanner throughput. If `averageOcrMs`
|
||||
dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
|
||||
queue. If `averageCaptureRoundTripOverheadMs` is high, native capture encode,
|
||||
Base64 transport, Electron image decode, or IPC/render scheduling is the next
|
||||
bottleneck. The current 3 artifacts/second target requires `averageMsPerParsed`
|
||||
at or below `333 ms` on a clean 20-artifact iteration.
|
||||
```text
|
||||
outputs/native-live-smoke/<timestamp>/
|
||||
```
|
||||
|
||||
Latest live timing evidence on 2026-07-08:
|
||||
Use this native smoke path before enabling any artifact-store promotion from
|
||||
native `scan-results.json`.
|
||||
|
||||
- Probe: `/automation/probe-click?index=1` returned `clicked: true`,
|
||||
`inputBlocked: false`, `changed: true`, and `captureTarget:
|
||||
"genshin-client"`.
|
||||
- Baseline after helper/hot-loop cleanup:
|
||||
`/scanner/start?entry=visible-inventory&limit=50&engine=current` completed
|
||||
`50/50` parsed and stored with `0` review, `0` duplicates, `0` misses,
|
||||
`2` pages, `elapsedMs: 61765`, `averageMsPerParsed: 1235`,
|
||||
`averageCaptureMs: 186`, `averageOcrMs: 162`, and
|
||||
`averageScrollReadyMs: 844`.
|
||||
- Deferred-write experiment:
|
||||
the same 50-artifact run completed `50/50` with `0` misses but regressed to
|
||||
`elapsedMs: 63616` because 50 single-record writes produced
|
||||
`writeFlushMs: 8163`.
|
||||
- Current source replaces that experiment with batch persist and quiet
|
||||
auto-scan UI captures. Later direct-GDI live runs validated the batch/quiet
|
||||
path at limits 20, 45, and 100 with 0 misses in the current environment.
|
||||
- Direct GDI hot-path validation:
|
||||
after skipping `desktopCapturer.getSources()` in auto-scan artifact captures,
|
||||
the 20-artifact iteration baseline improved to `20/20` parsed, `19` stored,
|
||||
`0` review, `1` duplicate, `0` misses, `7966 ms` elapsed,
|
||||
`398 ms/artifact`, `averageCaptureMs: 193`, `averageOcrMs: 167`,
|
||||
`averageClickMs: 2`, and `writeFlushMs: 4`. This is roughly
|
||||
`2.5 artifacts/second` on the first visible page.
|
||||
- Scroll-path validation with the same direct GDI hot path:
|
||||
`/scanner/start?entry=visible-inventory&limit=45&engine=current` completed
|
||||
`45/45` parsed, `42` stored, `0` review, `3` duplicates, `0` misses,
|
||||
`2` pages, `18625 ms` elapsed, `414 ms/artifact`, `averageCaptureMs: 187`,
|
||||
`averageOcrMs: 162`, and one scroll readiness wait of `173 ms`.
|
||||
- 100-artifact direct-GDI validation:
|
||||
`/scanner/start?entry=visible-inventory&limit=100&engine=current` completed
|
||||
on runtime signature `2026-07-08-direct-gdi-hotpath` with `100/100` parsed,
|
||||
`97` stored, `0` review, `3` duplicates, `0` misses, `4` pages,
|
||||
`42064 ms` elapsed, `421 ms/artifact`, `averageCaptureMs: 179`,
|
||||
`averageOcrMs: 154`, `averageClickMs: 2`, `writeFlushMs: 6`, and `3`
|
||||
scroll readiness waits averaging `176 ms`.
|
||||
- OCR/parser eval after this speed pass: `npm run eval` passed with `23/23`
|
||||
exact-match cases, `100%` field accuracy, and `100%` critical fields. This is
|
||||
a regression gate, not a substitute for manually checking live artifact values.
|
||||
- Ownership and lock proof on 2026-07-09:
|
||||
`/scanner/start?entry=visible-inventory&limit=20&engine=current` completed
|
||||
`20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and
|
||||
`0` misses in `8047 ms`. Smart Capture parsed equipped footers for `Citlali`
|
||||
and `Linnea`, reported an unlocked artifact as `locked: false`, then reported
|
||||
a visibly locked artifact as `locked: true` with `lockSignal.ratio:
|
||||
0.14797913950456323` over threshold `0.06`. A follow-up bounded scan
|
||||
persisted that locked artifact with `equipped: "Citlali"` and `locked: true`.
|
||||
- 3 artifacts/second preparation:
|
||||
auto-scan artifact captures now also omit the detail-preview payload and
|
||||
expose `averageCaptureRoundTripMs` plus
|
||||
`averageCaptureRoundTripOverheadMs`. The first live run exposed a false
|
||||
`missing-crops-or-ocr` review trigger because the hot path intentionally omits
|
||||
`detailDataUrl`; this is fixed in `getAutoReviewReason`.
|
||||
- 3 artifacts/second live attempts:
|
||||
after the review fix, a clean `limit=20` run completed `20/20` parsed,
|
||||
`19` stored, `0` review, `1` duplicate, `0` misses, `7285 ms` elapsed,
|
||||
or `364 ms/artifact` (`2.75 artifacts/second`). The stable final run on
|
||||
signature `2026-07-08-direct-gdi-reviewfix` completed `20/20`, `18` stored,
|
||||
`0` review, `2` duplicates, `0` misses, `7973 ms` elapsed, or
|
||||
`399 ms/artifact`. 3 artifacts/second is not proven.
|
||||
- Follow-up 3 artifacts/second attempts on 2026-07-09:
|
||||
after the distinctive partial piece parser fix, the best clean repeatability
|
||||
run reached `336 ms/artifact` with `20/20` parsed, `0` review, `0` misses,
|
||||
`318 ms` average capture roundtrip, and `138 ms` roundtrip overhead. Later
|
||||
runs with crop priority and image-payload cleanup stayed clean but ranged
|
||||
around `346-351 ms/artifact`; the strict `333 ms/artifact` budget remains
|
||||
unproven.
|
||||
- Rejected speed experiments:
|
||||
detail-region capture, `GAA_OCR_WORKERS=5`, DataURL-to-buffer decode, and
|
||||
substat OCR `PSM.SINGLE_COLUMN` were all live/benchmark tested and were slower
|
||||
than the direct-GDI baseline. Later checks also rejected skipping
|
||||
`analyzePaimonMenu`, skipping lock-state as a production shortcut, reducing
|
||||
the artifact-level crop scale, and `GAA_OCR_WORKERS=6` as the default.
|
||||
Keep `GAA_OCR_WORKERS=4` for current runs.
|
||||
- Quality-gated current-vs-IK comparison:
|
||||
`npm run scan:goal:compare:validated` produced
|
||||
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`
|
||||
with `createdAt: 2026-07-08T18:41:11.6120957+02:00`.
|
||||
The final validator summary passed at `limit=100` with winner `current`,
|
||||
`activeAvg: 378 ms/artifact`, `projected100: 37800 ms`, `missRate: 0`, and
|
||||
`reviewRate: 0`. The `current` 100-artifact run parsed `100/100`, stored `97`,
|
||||
had `0` review, `0` misses, and crossed `4` pages. The `ik-traineddata`
|
||||
100-artifact run parsed `97/100`, had `5` review and `3` misses, and was not
|
||||
qualified because it parsed fewer artifacts than requested.
|
||||
|
||||
The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
|
||||
temporary scan limit. It does not change the normal UI setting. The normal
|
||||
hotkeys and buttons still use the UI's configured scan limit.
|
||||
|
||||
Lookup and benchmark utility endpoints:
|
||||
The native preflight and start endpoints both accept an explicit category:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/status"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/regenerate"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup?engine=ik-traineddata"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=ik-traineddata"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=compare"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&profile=full"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/native/preflight?category=artifacts"
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2&category=artifacts"
|
||||
```
|
||||
|
||||
The benchmark endpoint measures the current Tesseract.js engine and the
|
||||
Inventory-Kamera-traineddata Tesseract.js path against the artifact crop set and
|
||||
returns timing/field counts, min/p50/p90/max timing, OCR p50/p90 timing,
|
||||
20/45/100-artifact projections, skipped-OCR count, and the active OCR worker
|
||||
pool size. It also returns per-field OCR timings under
|
||||
`ocrFieldAverages`, which is the first place to look before changing crop or
|
||||
parser behavior. Individual captures also report whether the artifact was
|
||||
detected as `sanctified`; level/substat crops are shifted in that state to match
|
||||
Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
|
||||
which omits the low-value set-effect crop and the main-stat-value crop that can
|
||||
be derived from slot, main-stat label, and level. The slot crop remains enabled
|
||||
in the fast profile because it improved live-read quality. The fast profile also uses
|
||||
Inventory Kamera's tighter substat crop height; full/manual captures keep the
|
||||
larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
|
||||
captures while keeping the detail screenshot, OCR text, crop rect metadata, and
|
||||
timings. OCR crops are passed to Tesseract as PNG buffers internally, not as
|
||||
Base64 DataURLs, to avoid encode/decode overhead in batch scans. When
|
||||
`skipOcrUnlessArtifactDetail` blocks OCR because no artifact detail card is
|
||||
visible, OCR crop preprocessing is skipped too. Auto-scan readiness and scroll
|
||||
checks use native detail/inventory fingerprints and omit preview DataURLs in
|
||||
poll captures. Fast preflight/poll captures also omit crop list construction, crop images, and lock-state
|
||||
detection unless a caller explicitly overrides that option; add
|
||||
`profile=full` to OCR every artifact detail crop for debugging. It uses the same
|
||||
artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
|
||||
skipped and the response shows `skippedOcrCaptures` instead of burning time on
|
||||
invalid crops.
|
||||
The fast auto-scan profile now keeps the optional Equipped footer OCR on real
|
||||
artifact-read captures when the footer marker is visible, so stored artifacts
|
||||
can record the equipped character without requiring a separate manual capture.
|
||||
Name, level, main-stat label, footer, and substats remain in the OCR hot path;
|
||||
slot, set, and main-stat value are derived when the lookup/parser can validate
|
||||
them. Preflight and readiness poll captures still skip OCR/crops/lock-state
|
||||
work because they only prove surface and fingerprint changes.
|
||||
Local store/review writes are serialized through an internal queue but no longer
|
||||
block the next inventory click. The scan still flushes the queue before it
|
||||
returns its final summary, so `stored` and `review` counts remain final-state
|
||||
numbers.
|
||||
The app warms the default OCR worker pool in the background after startup; check
|
||||
`/scanner/status` -> `ocrWarmup.current` before timing the first artifact. Use
|
||||
`/scanner/ocr/warmup?engine=ik-traineddata` before comparing Inventory
|
||||
Kamera-traineddata timings so the benchmark is not dominated by worker creation.
|
||||
`engine=ik-traineddata` uses Inventory Kamera's local
|
||||
`genshin_fast_09_04_21.traineddata` through Tesseract.js when the file is found
|
||||
in `data/tessdata`, `IK_TESSDATA_DIR`, `work/Inventory_Kamera`, `work/refs`,
|
||||
or the local `_ik_ref*` folders.
|
||||
`engine=compare` runs `current` and `ik-traineddata` against the same visible
|
||||
artifact detail state. The auto-scan default must stay `current` until the IK
|
||||
traineddata path wins on the same captures. For a controlled live comparison,
|
||||
start the scanner with `engine=ik-traineddata`; this only changes the OCR
|
||||
worker language for that run and leaves the default UI/hotkey path on
|
||||
`current`.
|
||||
The OCR pool defaults to four workers because the fast artifact crop set has
|
||||
four useful OCR parameter groups; set `GAA_OCR_WORKERS=1..8` before startup to
|
||||
benchmark a different worker count. Inventory Kamera's native engine pool is
|
||||
still the reference design, but the current app path remains Tesseract.js until
|
||||
native OCR is integrated and measured. Crops are scheduled across the whole
|
||||
worker pool and each worker caches its last Tesseract parameter profile; this is
|
||||
closer to Inventory Kamera's multi-engine field OCR than the earlier
|
||||
parameter-group-serial scheduler.
|
||||
Only `artifacts` is implemented as native capture today. `weapons`,
|
||||
`characters`, and `materials` are expected to block with a catalog-only message
|
||||
at preflight/start time until their category-specific capture flows are
|
||||
implemented and validated. They are not part of the active UI scope while those
|
||||
values are not scanned.
|
||||
Artifact preflight also blocks when the Genshin client capture is blank,
|
||||
almost entirely white/black, or too visually uniform to be trusted.
|
||||
The probe-click endpoint additionally requires a detected Genshin-client
|
||||
artifact inventory grid and visible artifact detail card before sending input.
|
||||
|
||||
## Diagnostic Evidence
|
||||
|
||||
The Diagnose page contains a compact evidence timeline for scanner work. It logs
|
||||
runtime pings, focus attempts, key presses, entry captures, artifact-tab clicks,
|
||||
preflight failures, grid/count metadata, detail fingerprints, and detail/inventory
|
||||
screenshots. The same last events are also published through:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
|
||||
ConvertTo-Json -Depth 18
|
||||
```
|
||||
|
||||
Use this before changing scanner behavior: run the smallest failing action, read
|
||||
the evidence timeline, then decide whether the failure is focus/input, entry
|
||||
navigation, grid detection, capture quality, OCR, or parser validation.
|
||||
|
||||
If Paimon entry shows `entry key ESC` or `entry key B` with `eventsSent: 0`, the
|
||||
running `InputHelper.exe` probably predates keyboard support or is blocked. Stop
|
||||
the elevated app/helper, run `npm run helper:build`, then restart with
|
||||
`npm run dev:admin` so the app loads the rebuilt helper.
|
||||
|
||||
## Soak-Test Helper
|
||||
## Evidence Commands
|
||||
|
||||
After the elevated app is running and Genshin is open on the artifact inventory,
|
||||
the non-elevated terminal can drive the local dev-control endpoints and save a
|
||||
@@ -407,130 +173,49 @@ The helper writes timestamped JSON snapshots and a transcript to:
|
||||
outputs/live-soak/<timestamp>/
|
||||
```
|
||||
|
||||
Default sequence:
|
||||
|
||||
1. `/health`
|
||||
2. `/scanner/status`
|
||||
3. `/capture/smart?skipOcr=1`
|
||||
4. `/automation/probe-click?index=1`
|
||||
5. `/automation/probe-click?index=3`
|
||||
6. `/scanner/start?entry=visible-inventory&limit=2`
|
||||
7. `/scanner/start?entry=visible-inventory&limit=5`
|
||||
8. `/scanner/start?entry=visible-inventory&limit=10`
|
||||
9. `/scanner/start?entry=visible-inventory&limit=20`
|
||||
10. `/review/samples?limit=30`
|
||||
|
||||
For the actual Inventory-Kamera speed target, use the explicit goal run after
|
||||
`/health` shows the current `appBuild`:
|
||||
For short iteration:
|
||||
|
||||
```powershell
|
||||
npm run scan:live:preflight
|
||||
npm run scan:live:preflight:wait
|
||||
npm run scan:goal
|
||||
npm run scan:goal:current
|
||||
npm run scan:goal:ik
|
||||
npm run scan:iterate:compare:validated
|
||||
npm run scan:iterate:compare:validated:wait
|
||||
npm run scan:goal:compare
|
||||
npm run scan:goal:compare:validated
|
||||
npm run scan:goal:compare:validated:wait
|
||||
npm run scan:iterate:validated
|
||||
npm run scan:iterate:validated:wait
|
||||
```
|
||||
|
||||
`scan:live:preflight` checks `/health`, `/scanner/status`, the current
|
||||
`APP_RUNTIME_SIGNATURE`, elevation, and whether Genshin is visible to the helper
|
||||
before a long live scan is attempted.
|
||||
Use `npm run scan:live:preflight:wait` during manual startup after `npm run
|
||||
dev:admin`; it waits up to 120 seconds for the elevated dev-control server and
|
||||
runtime checks to become ready. The non-waiting command remains the default for
|
||||
validated scan chains so automation fails fast on a missing runtime.
|
||||
|
||||
Use `npm run scan:iterate:compare:validated` for fast iteration while tuning OCR,
|
||||
parser, capture, or readiness behavior. It runs the same preflight, compares
|
||||
`current` vs. `ik-traineddata` at `limit=20`, and validates the newest assessment
|
||||
with `--limit=20 --summary`. This is the preferred loop while debugging because
|
||||
it gives quality-gated feedback without waiting for the full `2, 5, 20, 45, 100`
|
||||
goal sequence. Use `npm run scan:iterate:compare:validated:wait` directly after
|
||||
UAC if the elevated runtime may still be starting.
|
||||
|
||||
The goal run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
|
||||
limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
|
||||
`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
|
||||
uses the default `current` scan engine; use `scan:goal:ik` for a native
|
||||
IK-traineddata scan pass. Use `scan:goal:compare` to run both scan engines
|
||||
back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for
|
||||
`averageMsPerParsed`, `activeAverageMsPerParsed`, `averageCaptureMs`,
|
||||
`averageCaptureRoundTripMs`, `averageCaptureRoundTripOverheadMs`,
|
||||
`captureP50Ms`, `captureP90Ms`, `averageOcrMs`, `ocrP50Ms`, `ocrP90Ms`,
|
||||
`averageCardReadyMs`, `averageScrollReadyMs`, `artifactsPerMinute`, and
|
||||
`projectedMsFor100`.
|
||||
The run also writes `scan-performance-assessment.json`, which groups results by
|
||||
limit, picks the best qualified engine, and labels the dominant bottleneck as
|
||||
OCR, capture-roundtrip-overhead, capture, card-ready, or scroll-ready. A
|
||||
qualified winner must finish the run, parse the requested count, keep miss rate
|
||||
under 2%, and keep review rate at or below 15%; review and miss rates are
|
||||
penalized before active average speed is used as the tie-breaker. For IK-target
|
||||
claims, check `goal100Decision`; it
|
||||
must read `qualified-comparison: winner=<engine>`, and
|
||||
`goal100.comparisonComplete` must be `true` so a single-engine 100-artifact run
|
||||
is not mistaken for a current-vs-IK comparison.
|
||||
|
||||
Validate the saved assessment before using it as final evidence:
|
||||
For a full 2, 5, 20, 45, 100 current-run evidence chain:
|
||||
|
||||
```powershell
|
||||
npm run scan:assessment:validate -- --latest
|
||||
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json
|
||||
npm run scan:goal:validated
|
||||
npm run scan:goal:validated:wait
|
||||
```
|
||||
|
||||
`--latest` searches `outputs/live-soak/` for the newest
|
||||
`scan-performance-assessment.json`. Use explicit `--input` when comparing older
|
||||
or archived runs. Add `--expect-winner=current` or
|
||||
`--expect-winner=ik-traineddata` when validating a specific engine claim instead
|
||||
of accepting any qualified winner. Add `--limit=20` for a short iteration run
|
||||
instead of the final 100-artifact proof. Add `--summary` when you want a short
|
||||
report-ready PASS/FAIL output that includes the input assessment path and
|
||||
assessment `createdAt` timestamp.
|
||||
|
||||
Optional budget flags are useful for the current speed work:
|
||||
|
||||
```powershell
|
||||
npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120
|
||||
```
|
||||
|
||||
`--max-active-average-ms=333` is the strict 3 artifacts/second check. Use a
|
||||
separate `--max-capture-roundtrip-overhead-ms` budget when deciding whether the
|
||||
next optimization belongs in native capture transport instead of OCR.
|
||||
Single-engine repeatability runs may be validated with `--allow-single-engine`,
|
||||
but that mode is only for repeatability evidence and must not be used for IK
|
||||
parity claims.
|
||||
|
||||
`npm run scan:goal:compare:validated` is the preferred final command: it runs
|
||||
the live preflight first, then the full comparison, and then validates the
|
||||
newest assessment with `--summary`. Use
|
||||
`npm run scan:goal:compare:validated:wait` for the same final flow when starting
|
||||
immediately after UAC.
|
||||
|
||||
For later-session repeatability without changing OCR engines, use:
|
||||
For later-session repeatability without changing OCR engines:
|
||||
|
||||
```powershell
|
||||
npm run scan:repeatability:wait
|
||||
```
|
||||
|
||||
That command runs the current visible-inventory engine at 20, 45, and 100
|
||||
artifacts, then validates the 100-artifact result as single-engine evidence.
|
||||
It uses the `live-soak.ps1 -RepeatabilityRun` switch instead of passing a
|
||||
comma-separated `-Limits` value through npm/cmd, because Windows argument
|
||||
parsing can collapse `20,45,100` into one unsafe number. The script also refuses
|
||||
limits above 1800 as a final guard.
|
||||
Validate a saved assessment before using it as final evidence:
|
||||
|
||||
The assessment ranking can be verified without Genshin or the Electron app:
|
||||
```powershell
|
||||
npm run scan:assessment:validate -- --latest --summary
|
||||
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json --summary
|
||||
```
|
||||
|
||||
Optional budget flags:
|
||||
|
||||
```powershell
|
||||
npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120
|
||||
```
|
||||
|
||||
`--max-active-average-ms=333` is the strict 3 artifacts/second check. A passing
|
||||
100-artifact current run is valid scanner evidence; it is not evidence that the
|
||||
3 artifacts/second budget was met unless this budget check also passes.
|
||||
|
||||
The assessment self-test does not need Genshin:
|
||||
|
||||
```powershell
|
||||
npm run scan:assessment:test
|
||||
```
|
||||
|
||||
This self-test rejects synthetic runs that are fast but have too many misses or
|
||||
too many review samples, so the final IK comparison cannot be won by speed alone.
|
||||
|
||||
## Review-To-Eval Quality Loop
|
||||
|
||||
After any live scan that creates review samples, export candidates before adding
|
||||
@@ -541,13 +226,9 @@ npm run eval:review-candidates -- --limit=80
|
||||
```
|
||||
|
||||
Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
|
||||
worklist, not ground truth. Only after the expected fields are confirmed or
|
||||
corrected against the real artifact should a case be moved into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`. This prevents the parser from
|
||||
grading itself and keeps `npm run eval` meaningful. The exporter deduplicates
|
||||
samples, puts complete modern OCR captures first, and marks missing fast-profile
|
||||
fields so stale or partial captures are easier to ignore. Unconfirmed exporter
|
||||
output must stay in `outputs/review-eval-candidates/`.
|
||||
worklist, not ground truth. Only after expected fields are confirmed or
|
||||
corrected against the real artifact should a case move into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
For a manually checked candidate, generate a paste-ready confirmed-case snippet:
|
||||
|
||||
@@ -555,54 +236,12 @@ For a manually checked candidate, generate a paste-ready confirmed-case snippet:
|
||||
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
|
||||
```
|
||||
|
||||
The command requires explicit labels and writes only to the ignored outputs
|
||||
folder. Review the snippet before adding it to
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
For the current implementation summary and IK comparison rationale, see
|
||||
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
|
||||
|
||||
Use the readiness timings to compare against Inventory Kamera's fixed waits:
|
||||
IK waits about 200 ms after selecting the next inventory item and about 100 ms
|
||||
after fast scrolls. If `averageCardReadyMs` or `averageScrollReadyMs` dominates
|
||||
the active average while OCR is already low, tune the fingerprint gate before
|
||||
touching OCR again.
|
||||
|
||||
The runner reads `APP_RUNTIME_SIGNATURE` from `electron/main.ts` and refuses
|
||||
to run against a stale Electron process when `/health.appBuild.signature` does
|
||||
not match the current source. Use `-AllowStaleBuild` only for deliberate
|
||||
debugging of an older instance.
|
||||
Current dev builds also expose `/dev/shutdown` on localhost. The start cleanup
|
||||
script calls it before falling back to `Stop-Process`, so a previous elevated
|
||||
app can shut itself down cleanly even when the caller cannot terminate an
|
||||
administrator process directly. Older builds without that endpoint still need
|
||||
manual close or a confirmed `npm run dev:admin` restart.
|
||||
|
||||
Review samples are saved as a compact summary by default so Vite does not try to
|
||||
watch large Base64 payloads under `outputs/`. Full review payloads can be saved
|
||||
with `-SaveFullReviewSamples` when needed.
|
||||
|
||||
It stops on a failed probe, blocked scan, stopped scan, or timeout unless
|
||||
`-ContinueAfterBlocked` is supplied directly:
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\live-soak.ps1 -Limits 2,5 -ContinueAfterBlocked
|
||||
```
|
||||
|
||||
## Anti-Cheat And Safety Boundary
|
||||
|
||||
Do not describe the current implementation as bypassing anti-cheat. The app
|
||||
does not read memory, hook the process, inject code, modify game files, inspect
|
||||
packets, or interact with kernel drivers. It uses normal Windows screen capture,
|
||||
focus, cursor movement, wheel, and click input.
|
||||
|
||||
The practical finding is narrower:
|
||||
|
||||
- A non-elevated app can be blocked by Windows integrity/UIPI when the target
|
||||
process is elevated or protected.
|
||||
- Running the app elevated fixed input delivery in the tested environment.
|
||||
- Genshin's anti-cheat may still affect behavior on other machines, game modes,
|
||||
overlays, or future versions. Re-run the probe before trusting broad scans.
|
||||
Do not describe the implementation as bypassing anti-cheat. The app does not
|
||||
read memory, hook the process, inject code, modify game files, inspect packets,
|
||||
or interact with kernel drivers. It uses normal Windows screen capture, focus,
|
||||
cursor movement, wheel, and click input.
|
||||
|
||||
Never add automation that deletes, feeds, enhances, locks/unlocks, spends
|
||||
resources, reads memory, hooks, injects, or modifies Genshin.
|
||||
@@ -613,12 +252,9 @@ The current 16:9 layout profile is calibrated from a 1920x1080 English
|
||||
artifact-inventory capture:
|
||||
|
||||
- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
|
||||
- inventory grid: `8 x 4` safe automated targets, matching Inventory Kamera's
|
||||
32-artifact full-page model. The apparent lower fifth row is in the bottom
|
||||
control band and is intentionally not clicked during auto-scan.
|
||||
- inventory grid: `8 x 4` safe automated targets
|
||||
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
|
||||
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`
|
||||
- inventory count crop successfully read `2059/2400` in the live session
|
||||
|
||||
The profile is resolution-scaled for 16:9. Off-profile setups should be treated
|
||||
as higher risk and validated with Smart Capture plus the probe.
|
||||
@@ -635,5 +271,3 @@ Before marking an automation change done:
|
||||
6. For scan-loop changes, run
|
||||
`/scanner/start?entry=visible-inventory&limit=2` before any broader scan.
|
||||
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
|
||||
8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
|
||||
a non-stale `npm run scan:goal:compare:validated` run.
|
||||
|
||||
Reference in New Issue
Block a user