genshin-assistant/docs/AUTOMATION_LIVE_SCAN.md

# Automation Live Scan Runbook

This document is the durable reference for automatic artifact scanning, mouse
movement, click input, elevation, and live validation status.

## Current Known-Good State

Validated live with Genshin open in the artifact inventory at 1920x1080,
English UI:

- `npm run dev:admin` starts the app elevated after the user confirms UAC.
- Runtime status reports `isElevated: true`, `genshinFound: true`, and
  `targetProcess: "GenshinImpact"`.
- `/automation/probe-click?index=1` performs one read-only inventory tile
  selection click.
- The visible-inventory auto-scan path is the production baseline. It requires
  the Artifact inventory to already be open with a visible artifact detail card.
- On 2026-07-09, `/scanner/start?entry=visible-inventory&limit=100` completed
  with `100/100` attempted, verified, and parsed, `98` stored, `2` duplicates,
  `0` review samples, `0` misses, `4` pages, and `393 ms/artifact`.
- On 2026-07-09, `npm run scan:native:smoke` completed against runtime
  signature `2026-07-09-native-ik-visual-probe`: native visual preflight passed
  at 1920x1080 with `32` grid targets, guarded probe changed the detail panel,
  native capture wrote `2/2` artifact card crops in `510 ms`, and post-capture
  processing parsed `2/2` with `0` review, `0` errors, `queueConcurrency: 2`,
  and `persisted: false`.
- The same native path later completed dry 20/50/100-item runs. The final run
  captured `100/100` crops across 4 pages in `24,673 ms`, parsed `100/100`,
  produced `0` errors, routed 3 implausible OCR values into Review, and wrote
  nothing to the artifact store. Full evidence and the parser/timestamp fixes
  are recorded in
  [NATIVE_SCANNER_VALIDATION_2026-07-09.md](NATIVE_SCANNER_VALIDATION_2026-07-09.md).
- On 2026-07-10, a fresh later-session chain passed with the hardened native
  helper: `5/20/50/100`, all parsed/evaluated, `0` Review, `0` processing
  errors, and `0` store writes. The new 100er captured `100/100` over four
  pages at `4.37/s` and processed at `3.15/s`, with no duplicate PNG hash group,
  repeated page, or identical page boundary.
- A later real Settings-UI acceptance selected `Eigenes Limit -> Artefakte ->
  5`. Genshin, visible detail, and the 4x8 grid passed preflight;
  target/captured/processed/parsed were all 5, Review and processing errors were
  0, persistence stayed disabled, and the current-session result rail contained
  five entries. A later packaged row-limit run resolved one detected row to
  eight Artifacts and passed 8/8 with the same non-persistence boundary.
- The packaged full-owned-inventory run `20260710-223233` resolved the target
  from the OCR-confirmed owned count `2,211`, captured all 2,211 crops over 70
  pages, streamed results during capture, and produced zero processing errors
  and zero store writes. Its first parser pass correctly failed the 15% Review
  gate at 490 Review. After the structurally safe five-star +0 repair, the
  current package reprocessed the same complete corpus at 136 Review (6.15%);
  the saved-run validator then passed with zero issues.
- The bounded inventory-top reset now completes in about `1.7 s` instead of
  about `50.7 s`; the 3,200-event range and per-event foreground/stop guards
  remain intact.
- A packaged build started from `%TEMP%` and completed a native 5er through
  renderer, preload, IPC, packaged helper, and packaged IK resources with
  `persist=false`. See
  [NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md](NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md).

The old alternate OCR-engine and comparison paths have been retired. The app now
runs the current OCR/capture path only. Performance claims should be stated as
current-run repeatability evidence, not as external-tool parity.

## Elevation And UAC

Use:

```powershell
npm run dev:admin
```

The command runs `scripts/dev-admin.ps1`, which launches a new elevated
PowerShell window running `scripts/dev-admin-start.ps1`. The elevated start is
logged to:

```text
outputs/admin-start/admin-dev.log
```

The user must confirm the Windows UAC prompt. The app cannot and must not click
the Secure Desktop UAC prompt for itself.

Useful checks:

```powershell
Invoke-RestMethod http://127.0.0.1:17317/health
Invoke-RestMethod http://127.0.0.1:17317/scanner/status
```

Expected runtime facts before automatic scan:

- `isElevated: true`
- `genshinFound: true`
- `targetProcess: "GenshinImpact"`
- `/health.appBuild.signature` matches `APP_RUNTIME_SIGNATURE` in
  `electron/main.ts`

## Mouse And Click Validation

Use a probe before broad auto-scan work:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
  ConvertTo-Json -Depth 12
```

The probe performs one read-only inventory selection click. It does not delete,
feed, enhance, lock, unlock, spend, or modify game resources.

Interpretation:

- `click.ok: true`, `clicked: true`, and `inputBlocked: false` means Windows did
  not block SendInput/UIPI in the current configuration.
- `focused: true` and `foregroundProcess: "GenshinImpact"` means the click was
  sent while Genshin was foreground.
- `changed: true` means the detail panel changed after the click.
- `changed: false` can be benign if the target tile was already selected or two
  neighboring artifacts render identically; retry with another `index`, `row`,
  or `col`.

## Bounded Live Auto-Scan

Start with a tiny bounded run:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"
```

Then poll:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12
```

The scan starts only after a valid lookup package, supported 16:9 layout,
detected artifact grid, Genshin-client capture, and visual artifact-detail
markers are all present. If any preflight check fails, keep using the
visible-inventory path and fix the blocking evidence before trying broader
runs.

The loop verifies detail fingerprints after clicks, retries one unchanged detail
read, stores parsed artifacts in a batch, and flushes pending writes before the
final summary. Stats separate clicked, attempted, verified, parsed, stored,
review, duplicates, and misses so progress is not confused with successful
reads.

For the native IK-style path, prefer the dedicated smoke runner:

```powershell
npm run scan:native:smoke
npm run scan:native:smoke:5
```

For the gated live scale chain, run and validate each size before continuing:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\native-live-smoke.ps1 -Limit 20
npm run scan:native:validate:saved -- --targets=20
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\native-live-smoke.ps1 -Limit 50
npm run scan:native:validate:saved -- --targets=50
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\native-live-smoke.ps1 -Limit 100
npm run scan:native:validate:saved -- --targets=100
```

It checks `/health`, `/scanner/native/data`,
`/scanner/native/preflight?category=artifacts` including the native visual
blank-capture guard, runs the guarded
`/automation/probe-click?index=1`, starts `/scanner/start?limit=N&category=artifacts`,
waits for the native helper to finish, runs `/scanner/native/process` with
`persist=0`, and loads `/scanner/native/results`. Evidence is written to:

```text
outputs/native-live-smoke/<timestamp>/
```

Use this native smoke path before enabling any artifact-store promotion from
native `scan-results.json`.

For a deliberate packaged-runtime acceptance, first build with
`npm run package:offline-check`, then launch the unpacked EXE from a foreign
working directory with a loopback-only temporary CDP port:

```powershell
$exe = (Resolve-Path "outputs\dist\win-unpacked\Genshin Artifact Assistant.exe").Path
Start-Process -FilePath $exe `
  -ArgumentList "--remote-debugging-address=127.0.0.1", "--remote-debugging-port=9223" `
  -WorkingDirectory $env:TEMP -Verb RunAs
npm run package:live:inspect
```

`package:live:inspect` is read-only. The following command is a real bounded
five-card Genshin scan and therefore requires the same live authorization and
preconditions as every other input test:

```powershell
npm run package:live:scan5
```

The runner calls packaged renderer/preload/IPC methods serially, validates the
5-result non-persistence contract, and writes JSON plus a screenshot below
`outputs/packaged-live/`. Close the packaged app after testing so the temporary
CDP port disappears.

For the streaming pipeline, use the real Settings-UI path with a bounded target
large enough to observe overlap:

```powershell
npm run package:live:ui-stream20
```

This acceptance is stricter than a correct final count. It requires native
capture to be observed as running while at least one processing worker has
finished and at least one current-session result card is already visible. It
also records the maximum evaluated/result counts seen during capture, verifies
all 20 final entries, and checks the insertion animation unless the operating
system requests reduced motion. Persistence remains disabled.

The same packaged Settings-UI runner exposes all accepted scope paths:

```powershell
npm run package:live:ui-scan5
npm run package:live:ui-row1
npm run package:live:ui-stream20
npm run package:live:ui-full-inventory
```

`package:live:ui-full-inventory` explicitly selects `Gesamtes Inventar` and
never supplies a numeric fallback. It accepts the target only from the running
scanner status after the app has read the current owned Artifact count. The
runner rejects targets outside `1..2400`, requires exact captured/processed/
parsed/result/UI-row counts, checks `pages = ceil(target / 32)`, enforces at
most 15% Review with zero processing errors, verifies streaming overlap and
document fit, and requires `persist=false` with zero store writes. Its timeout
is 15 minutes. The real elevated run `20260710-223233` reached 2,211/2,211 over
70 pages. Because the initial parser produced 490 Review, the gate rejected the
first result honestly; current-package reprocessing reduced that to 136 Review
(6.15%), and `scan:native:validate:saved -- --targets=2211` passed with zero
issues. This two-stage evidence is the accepted full-inventory baseline.

The native preflight and start endpoints both accept an explicit category:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/native/preflight?category=artifacts"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2&category=artifacts"
```

Only `artifacts` is implemented as native capture today. `weapons`,
`characters`, and `materials` are expected to block with a catalog-only message
at preflight/start time until their category-specific capture flows are
implemented and validated. They are not part of the active UI scope while those
values are not scanned.
Artifact preflight also blocks when the Genshin client capture is blank,
almost entirely white/black, or too visually uniform to be trusted.
The probe-click endpoint additionally requires a detected Genshin-client
artifact inventory grid and visible artifact detail card before sending input.

At native Artifact-scan startup, the helper first resets the inventory to the
top with a bounded upward wheel sequence at the calibrated grid anchor and then
waits for the UI to stabilize. Genshin is focused only for initial startup. The
native loop checks that the same Genshin HWND is still foreground immediately
before every click and every wheel event; focus loss stops the run without an
automatic refocus. Stop, ESC, Enter, and F9 remain active during the reset,
stabilization, clicks, captures, and page scrolling. Native status reports
`initialTopResetMs` and `totalMs` separately; `activeMs` starts after the reset
and remains the capture-rate denominator.

Do not replace this production reset with `B` plus an Artifact-tab click based
on event count alone. Reopening depends on the current menu/inventory state and
adds a key toggle plus a coordinate target. It has not been live benchmarked
against the accepted bounded wheel reset. If evaluated later, run it as a
separate guarded A/B probe and compare successful top-state detection, focus/
stop behavior, coordinate robustness, and elapsed reset time before changing
the default.

## Evidence Commands

The commands in this first subsection exercise the older visible-inventory
engine and its assessment format. They remain useful for regression work on
that engine, but they do not post-process or validate the current native C#
capture path. Use `native-live-smoke.ps1` plus
`scan:native:validate:saved` for current native acceptance.

After the elevated app is running and Genshin is open on the artifact inventory,
the non-elevated terminal can drive the local dev-control endpoints and save a
full evidence bundle:

```powershell
npm run scan:soak
```

The helper writes timestamped JSON snapshots and a transcript to:

```text
outputs/live-soak/<timestamp>/
```

For short iteration:

```powershell
npm run scan:iterate:validated
npm run scan:iterate:validated:wait
```

For a full 2, 5, 20, 45, 100 current-run evidence chain:

```powershell
npm run scan:goal:validated
npm run scan:goal:validated:wait
```

For later-session repeatability without changing OCR engines:

```powershell
npm run scan:repeatability:wait
```

Validate a saved assessment before using it as final evidence:

```powershell
npm run scan:assessment:validate -- --latest --summary
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json --summary
```

Optional budget flags:

```powershell
npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120
```

`--max-active-average-ms=333` is the strict 3 artifacts/second check. A passing
100-artifact current run is valid scanner evidence; it is not evidence that the
3 artifacts/second budget was met unless this budget check also passes.

The historical `native-live-smoke.ps1` path is intentionally serial: it waits
for native capture to finish and only then invokes post-capture processing.
Do not divide a target by the sum of those two durations and describe the
result as the current streaming pipeline's end-to-end rate. The 2026-07-11
read-only streaming-20 check (`20260711-090019`) proved overlap (a result was
visible before capture completed) and observed 20/20 clean results at 4.43
capture/s and 2.39/s from request issue to result reconciliation. That bounded
observation is not a general performance claim: native start, capture start/end,
processing start/end, and final reconciliation still need one persisted shared
clock before publishing an end-to-end budget result.

The assessment self-test does not need Genshin:

```powershell
npm run scan:assessment:test
```

Saved native results can be re-evaluated without Genshin, focus changes, or
input automation:

```powershell
npm run scan:native:replay -- --run-dir=<native-run-dir> --repeats=5
```

This is an offline determinism and result-contract check. It does not replace a
live capture smoke test. The ordered checklist was executed on 2026-07-10; its
results are in
[NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md](NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md).

## Review-To-Eval Quality Loop

After any live scan that creates review samples, export candidates before adding
anything to the permanent eval corpus:

```powershell
npm run eval:review-candidates -- --limit=80
```

Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
worklist, not ground truth. Only choose a candidate marked
`visualEvidence: available`: it had a retrievable local PNG at export time.
The export exposes only safe capture/run references, never the local crop path.
After expected fields are confirmed or corrected against the real artifact, a
case may move into `src/eval/corpus/confirmedReviewCorpus.ts`.

For a manually checked candidate, generate a paste-ready confirmed-case snippet:

```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
```

The preparer refuses candidates without explicit labels or retrievable visual
evidence. Re-export after retaining a crop rather than promoting an OCR-only
legacy record.

## Anti-Cheat And Safety Boundary

Do not describe the implementation as bypassing anti-cheat. The app does not
read memory, hook the process, inject code, modify game files, inspect packets,
or interact with kernel drivers. It uses normal Windows screen capture, focus,
cursor movement, wheel, and click input.

Never add automation that deletes, feeds, enhances, locks/unlocks, spends
resources, reads memory, hooks, injects, or modifies Genshin.

## Live Layout Facts

The current 16:9 layout profile is calibrated from a 1920x1080 English
artifact-inventory capture:

- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
- inventory grid: `8 x 4` safe automated targets
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`

The profile is resolution-scaled for 16:9. Off-profile setups should be treated
as higher risk and validated with Smart Capture plus the probe.

## Validation Checklist

Before marking an automation change done:

1. Run `npm run helper:test` when native input safety changed.
2. Run `npm run lint`.
3. Run `npx tsc -p tsconfig.electron.json` when Electron/preload/main changed.
4. Run `npm test`.
5. Run `npm run build`.
6. If Genshin is available, run `/automation/probe-click?index=1`.
7. For native scan-loop changes, run `npm run scan:native:smoke:5` before any
   20/50/100 progression. Use the explicit visible-inventory endpoint only
   when that legacy engine itself changed.
8. Record new live findings in this file and in `docs/scanner-rework-status.md`.