genshin-assistant/docs/AUTOMATION_LIVE_SCAN.md

# Automation Live Scan Runbook

This document is the durable reference for automatic artifact scanning, mouse
movement, click input, elevation, and live validation status.

## Current Known-Good State

Validated live on 2026-07-07 with Genshin open in the artifact inventory at
1920x1080, English UI:

- `npm run dev:admin` starts the app elevated after the user confirms UAC.
- Runtime status reported `isElevated: true`, `genshinFound: true`, and
  `targetProcess: "GenshinImpact"`.
- The safe probe endpoint `/automation/probe-click?index=1` focused Genshin,
  moved the cursor to the second visible inventory tile, clicked it, and changed
  the artifact detail panel fingerprint.
- Probe result: `clicked: true`, `inputBlocked: false`,
  `foregroundProcess: "GenshinImpact"`, and `changed: true`.
- A bounded live auto-scan via `/scanner/start?limit=2` completed with:
  `clicked: 2`, `attempted: 2`, `verified: 2`, `parsed: 2`, `stored: 2`,
  `review: 2`, `misses: 0`, `status: "done"`.

This proves that the current elevated app plus helper path can deliver mouse
movement and click input to the focused Genshin client in this environment.

Latest-source timing is not proven while `/health.appBuild.signature` differs
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`. On 2026-07-07 the port
was still owned by an older elevated runtime, so goal scans were intentionally
blocked by the stale-build gate. Restart the elevated app through
`npm run dev:admin` and confirm UAC before collecting new 100-artifact evidence.

## Elevation And UAC

Use:

```powershell
npm run dev:admin
```

The command runs `scripts/dev-admin.ps1`, which launches a new elevated
PowerShell window running `scripts/dev-admin-start.ps1`. The elevated start is
logged to:

```text
outputs/admin-start/admin-dev.log
```

The user must confirm the Windows UAC prompt. The app cannot and must not click
the Secure Desktop UAC prompt for itself. After confirmation, the app can verify
its own runtime through the dev status endpoint.

Useful checks:

```powershell
Invoke-RestMethod http://127.0.0.1:17317/health
Invoke-RestMethod http://127.0.0.1:17317/scanner/status
```

Expected runtime facts before automatic scan:

- `isElevated: true`
- `genshinFound: true`
- `targetProcess: "GenshinImpact"`
- hotkeys registered

## Mouse And Click Validation

Use the probe before broad auto-scan work:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
  ConvertTo-Json -Depth 12
```

The probe performs one read-only inventory selection click. It does not delete,
feed, enhance, lock, unlock, spend, or modify game resources.

Interpretation:

- `click.ok: true`, `clicked: true`, `inputBlocked: false` means Windows did not
  block SendInput/UIPI in the current configuration.
- `focused: true` and `foregroundProcess: "GenshinImpact"` means the click was
  sent while Genshin was foreground.
- `changed: true` means the detail panel changed after the click.
- `changed: false` can be benign if the target tile was already selected or two
  neighboring artifacts render identically; retry with another `index`, `row`,
  or `col`.

Examples:

```powershell
# Second visible tile
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1"

# Specific grid cell
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?row=0&col=3"
```

## Bounded Live Auto-Scan

For live validation, prefer a bounded scan first:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
```

The visible-inventory path remains the safest first check. The normal guided
entry tries the read-only direct world path first:
`B -> artifact tab -> first artifact tile`. If that does not produce a visible
artifact detail card, it falls back to the Inventory Kamera-compatible sequence:
`ESC -> B -> artifact tab -> first artifact tile`.

The normal Auto-Scan button uses a guided start. It first takes one lightweight
preflight capture without OCR, full-frame payload, review scoring, or storing.
If an artifact detail card is already visible, it starts the visible-inventory
scan. Otherwise it runs the guided entry above. OCR/review/store work starts
only after the artifact-detail preflight passes.
Guided entry uses short state polling for the Inventory screen, artifact grid,
and first detail card instead of waiting the full fixed delay every time; if the
state never appears, the same timeout budget returns the last diagnostic capture.

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=auto-entry&limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2&engine=ik-traineddata"
```

Those paths send only read-only navigation. `ESC` is not a universal "go to
world" command: from the world it opens the Paimon menu, while from the
already-open Paimon menu it returns to the world. This is why the normal
`auto-entry` path first tries `B` directly and uses the IK-style `ESC -> B`
fallback only when direct entry did not reach an artifact detail card.

The scan starts only after a valid lookup package, supported 16:9 layout,
detected artifact grid, Genshin-client capture, and visual artifact-detail
markers are all present. If any preflight check fails, keep using the
visible-inventory path while tuning the entry step.

The visual preflight also classifies the Paimon menu. The Paimon profile/card
grid can look like an inventory grid if only fixed 16:9 coordinates are used, so
the scanner must reject `paimonMenu.present` before any artifact OCR, review
sample creation, store write, or grid scan starts. The guided entry may still
take lightweight skip-OCR captures while navigating, but those captures are only
state evidence.

The same guard also runs inside the scan loop. If the app is on the main game
screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
without an artifact detail card, auto-scan must block instead of clicking tiles
or trying OCR.
After each click the loop polls the detail fingerprint with a short bounded
budget instead of sleeping blindly. The current budget is 420 ms with 60 ms
polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it
does not change the loop retries or stops through the normal miss guards. If the
card changed but remains animated, the loop now proceeds after 200 ms, matching
Inventory Kamera's select-next-item wait more closely without removing the
detail-change guard.
The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
not re-run the focus helper before every tile, which avoids an OS focus ping on
each artifact while still relying on click readback, foreground checks, and the
detail-card guard for safety.
After a scroll, the loop now uses the same cheap fingerprint polling model for
the inventory pane: it proceeds as soon as the next page fingerprint changed and
stabilized instead of always sleeping the old fixed 760 ms settle delay. Changed
but still animated inventory pages may proceed after 100 ms, again matching IK's
fast-scroll wait while still blocking unchanged pages.

Then poll:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12
```

Before live timing, verify that the endpoint is the current app instance:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/health" |
  ConvertTo-Json -Depth 6
```

The response must include `appBuild.signature` and
`appBuild.expectedOcrWorkerPoolSize`. If `appBuild` is missing, or
`/scanner/status` still reports the old OCR warmup start time, the local port is
still owned by a stale elevated Electron process. Close the old Administrator
window/app and restart with `npm run dev:admin` before running scanner probes.

Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
`activeAverageMsPerParsed`, `averageCaptureMs`, `averageOcrMs`,
`artifactsPerMinute`, and `projectedMsFor100`. `elapsedMs` is end-to-end
including queued writes; `activeScanMs` is the click/capture/OCR loop before
the final store/review flush. A run only counts as speed
evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
together; raw click count alone is not scanner throughput. If `averageOcrMs`
dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
queue. If `averageCaptureMs` dominates, crop payload/capture work is the
bottleneck.

The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
temporary scan limit. It does not change the normal UI setting. The normal
hotkeys and buttons still use the UI's configured scan limit.

Lookup and benchmark utility endpoints:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/status"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/regenerate"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup?engine=ik-traineddata"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=ik-traineddata"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=compare"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&profile=full"
```

The benchmark endpoint measures the current Tesseract.js engine and the
Inventory-Kamera-traineddata Tesseract.js path against the artifact crop set and
returns timing/field counts, min/p50/p90/max timing, OCR p50/p90 timing,
20/45/100-artifact projections, skipped-OCR count, and the active OCR worker
pool size. It also returns per-field OCR timings under
`ocrFieldAverages`, which is the first place to look before changing crop or
parser behavior. Individual captures also report whether the artifact was
detected as `sanctified`; level/substat crops are shifted in that state to match
Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
which omits the low-value set-effect crop, the slot crop that can be derived
from the matched artifact piece name, and the main-stat-value crop that can be
derived from slot, main-stat label, and level. The fast profile also uses
Inventory Kamera's tighter substat crop height; full/manual captures keep the
larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
captures while keeping the detail screenshot, OCR text, crop rect metadata, and
timings. OCR crops are passed to Tesseract as PNG buffers internally, not as
Base64 DataURLs, to avoid encode/decode overhead in batch scans. When
`skipOcrUnlessArtifactDetail` blocks OCR because no artifact detail card is
visible, OCR crop preprocessing is skipped too. Auto-scan readiness and scroll
checks use native detail/inventory fingerprints and omit preview DataURLs in
poll captures. Fast preflight/poll captures also omit crop list construction, crop images, and lock-state
detection unless a caller explicitly overrides that option; add
`profile=full` to OCR every artifact detail crop for debugging. It uses the same
artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
skipped and the response shows `skippedOcrCaptures` instead of burning time on
invalid crops.
For speed, the fast auto-scan profile also skips the optional Equipped footer
OCR. Name, level, main-stat label, and substats remain in the OCR hot path;
slot, set, and main-stat value are derived when the lookup/parser can validate
them. Use a full/manual capture when equipped ownership or every debug crop matters.
Local store/review writes are serialized through an internal queue but no longer
block the next inventory click. The scan still flushes the queue before it
returns its final summary, so `stored` and `review` counts remain final-state
numbers.
The app warms the default OCR worker pool in the background after startup; check
`/scanner/status` -> `ocrWarmup.current` before timing the first artifact. Use
`/scanner/ocr/warmup?engine=ik-traineddata` before comparing Inventory
Kamera-traineddata timings so the benchmark is not dominated by worker creation.
`engine=ik-traineddata` uses Inventory Kamera's local
`genshin_fast_09_04_21.traineddata` through Tesseract.js when the file is found
in `data/tessdata`, `IK_TESSDATA_DIR`, `work/Inventory_Kamera`, `work/refs`,
or the local `_ik_ref*` folders.
`engine=compare` runs `current` and `ik-traineddata` against the same visible
artifact detail state. The auto-scan default must stay `current` until the IK
traineddata path wins on the same captures. For a controlled live comparison,
start the scanner with `engine=ik-traineddata`; this only changes the OCR
worker language for that run and leaves the default UI/hotkey path on
`current`.
The OCR pool defaults to four workers because the fast artifact crop set has
four useful OCR parameter groups; set `GAA_OCR_WORKERS=1..8` before startup to
benchmark a different worker count. Inventory Kamera's native engine pool is
still the reference design, but the current app path remains Tesseract.js until
native OCR is integrated and measured. Crops are scheduled across the whole
worker pool and each worker caches its last Tesseract parameter profile; this is
closer to Inventory Kamera's multi-engine field OCR than the earlier
parameter-group-serial scheduler.

## Diagnostic Evidence

The Diagnose page contains a compact evidence timeline for scanner work. It logs
runtime pings, focus attempts, key presses, entry captures, artifact-tab clicks,
preflight failures, grid/count metadata, detail fingerprints, and detail/inventory
screenshots. The same last events are also published through:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 18
```

Use this before changing scanner behavior: run the smallest failing action, read
the evidence timeline, then decide whether the failure is focus/input, entry
navigation, grid detection, capture quality, OCR, or parser validation.

If Paimon entry shows `entry key ESC` or `entry key B` with `eventsSent: 0`, the
running `InputHelper.exe` probably predates keyboard support or is blocked. Stop
the elevated app/helper, run `npm run helper:build`, then restart with
`npm run dev:admin` so the app loads the rebuilt helper.

## Soak-Test Helper

After the elevated app is running and Genshin is open on the artifact inventory,
the non-elevated terminal can drive the local dev-control endpoints and save a
full evidence bundle:

```powershell
npm run scan:soak
```

The helper writes timestamped JSON snapshots and a transcript to:

```text
outputs/live-soak/<timestamp>/
```

Default sequence:

1. `/health`
2. `/scanner/status`
3. `/capture/smart?skipOcr=1`
4. `/automation/probe-click?index=1`
5. `/automation/probe-click?index=3`
6. `/scanner/start?limit=2`
7. `/scanner/start?limit=5`
8. `/scanner/start?limit=10`
9. `/scanner/start?limit=20`
10. `/review/samples?limit=30`

For the actual Inventory-Kamera speed target, use the explicit goal run after
`/health` shows the current `appBuild`:

```powershell
npm run scan:goal
npm run scan:goal:current
npm run scan:goal:ik
npm run scan:goal:compare
```

That run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
uses the default `current` scan engine; use `scan:goal:ik` for a native
IK-traineddata scan pass. Use `scan:goal:compare` to run both scan engines
back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for
`averageMsPerParsed`, `activeAverageMsPerParsed`, `averageCaptureMs`,
`captureP50Ms`, `captureP90Ms`, `averageOcrMs`, `ocrP50Ms`, `ocrP90Ms`,
`averageCardReadyMs`, `averageScrollReadyMs`, `artifactsPerMinute`, and
`projectedMsFor100`.
The run also writes `scan-performance-assessment.json`, which groups results by
limit, picks the best qualified engine, and labels the dominant bottleneck as
OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
run, parse the requested count, keep miss rate under 2%, and keep review rate
at or below 15%; review and miss rates are penalized before active average speed
is used as the tie-breaker.

The assessment ranking can be verified without Genshin or the Electron app:

```powershell
npm run scan:assessment:test
```

This self-test rejects synthetic runs that are fast but have too many misses or
too many review samples, so the final IK comparison cannot be won by speed alone.

For the current implementation summary and IK comparison rationale, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).

Use the readiness timings to compare against Inventory Kamera's fixed waits:
IK waits about 200 ms after selecting the next inventory item and about 100 ms
after fast scrolls. If `averageCardReadyMs` or `averageScrollReadyMs` dominates
the active average while OCR is already low, tune the fingerprint gate before
touching OCR again.

The runner reads `APP_RUNTIME_SIGNATURE` from `electron/main.ts` and refuses
to run against a stale Electron process when `/health.appBuild.signature` does
not match the current source. Use `-AllowStaleBuild` only for deliberate
debugging of an older instance.
Current dev builds also expose `/dev/shutdown` on localhost. The start cleanup
script calls it before falling back to `Stop-Process`, so a previous elevated
app can shut itself down cleanly even when the caller cannot terminate an
administrator process directly. Older builds without that endpoint still need
manual close or a confirmed `npm run dev:admin` restart.

Review samples are saved as a compact summary by default so Vite does not try to
watch large Base64 payloads under `outputs/`. Full review payloads can be saved
with `-SaveFullReviewSamples` when needed.

It stops on a failed probe, blocked scan, stopped scan, or timeout unless
`-ContinueAfterBlocked` is supplied directly:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\live-soak.ps1 -Limits 2,5 -ContinueAfterBlocked
```

## Anti-Cheat And Safety Boundary

Do not describe the current implementation as bypassing anti-cheat. The app
does not read memory, hook the process, inject code, modify game files, inspect
packets, or interact with kernel drivers. It uses normal Windows screen capture,
focus, cursor movement, wheel, and click input.

The practical finding is narrower:

- A non-elevated app can be blocked by Windows integrity/UIPI when the target
  process is elevated or protected.
- Running the app elevated fixed input delivery in the tested environment.
- Genshin's anti-cheat may still affect behavior on other machines, game modes,
  overlays, or future versions. Re-run the probe before trusting broad scans.

Never add automation that deletes, feeds, enhances, locks/unlocks, spends
resources, reads memory, hooks, injects, or modifies Genshin.

## Live Layout Facts

The current 16:9 layout profile is calibrated from a 1920x1080 English
artifact-inventory capture:

- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
- inventory grid: `8 x 4` safe automated targets, matching Inventory Kamera's
  32-artifact full-page model. The apparent lower fifth row is in the bottom
  control band and is intentionally not clicked during auto-scan.
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`
- inventory count crop successfully read `2059/2400` in the live session

The profile is resolution-scaled for 16:9. Off-profile setups should be treated
as higher risk and validated with Smart Capture plus the probe.

## Validation Checklist

Before marking an automation change done:

1. Run `npm run lint`.
2. Run `npx tsc -p tsconfig.electron.json` when Electron/preload/main changed.
3. Run `npm test`.
4. Run `npm run build`.
5. If Genshin is available, run `/automation/probe-click?index=1`.
6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
   a non-stale `npm run scan:goal:compare` run.