genshin-assistant/docs/AUTOMATION_LIVE_SCAN.md

# Automation Live Scan Runbook

This document is the durable reference for automatic artifact scanning, mouse
movement, click input, elevation, and live validation status.

## Current Known-Good State

Validated live with Genshin open in the artifact inventory at 1920x1080,
English UI:

- `npm run dev:admin` starts the app elevated after the user confirms UAC.
- Runtime status reports `isElevated: true`, `genshinFound: true`, and
  `targetProcess: "GenshinImpact"`.
- `/automation/probe-click?index=1` performs one read-only inventory tile
  selection click.
- The visible-inventory auto-scan path is the production baseline. It requires
  the Artifact inventory to already be open with a visible artifact detail card.
- On 2026-07-09, `/scanner/start?entry=visible-inventory&limit=100` completed
  with `100/100` attempted, verified, and parsed, `98` stored, `2` duplicates,
  `0` review samples, `0` misses, `4` pages, and `393 ms/artifact`.
- On 2026-07-09, `npm run scan:native:smoke` completed against runtime
  signature `2026-07-09-native-ik-visual-probe`: native visual preflight passed
  at 1920x1080 with `32` grid targets, guarded probe changed the detail panel,
  native capture wrote `2/2` artifact card crops in `510 ms`, and post-capture
  processing parsed `2/2` with `0` review, `0` errors, `queueConcurrency: 2`,
  and `persisted: false`.
- The same native path later completed dry 20/50/100-item runs. The final run
  captured `100/100` crops across 4 pages in `24,673 ms`, parsed `100/100`,
  produced `0` errors, routed 3 implausible OCR values into Review, and wrote
  nothing to the artifact store. Full evidence and the parser/timestamp fixes
  are recorded in
  [NATIVE_SCANNER_VALIDATION_2026-07-09.md](NATIVE_SCANNER_VALIDATION_2026-07-09.md).

The old alternate OCR-engine and comparison paths have been retired. The app now
runs the current OCR/capture path only. Performance claims should be stated as
current-run repeatability evidence, not as external-tool parity.

## Elevation And UAC

Use:

```powershell
npm run dev:admin
```

The command runs `scripts/dev-admin.ps1`, which launches a new elevated
PowerShell window running `scripts/dev-admin-start.ps1`. The elevated start is
logged to:

```text
outputs/admin-start/admin-dev.log
```

The user must confirm the Windows UAC prompt. The app cannot and must not click
the Secure Desktop UAC prompt for itself.

Useful checks:

```powershell
Invoke-RestMethod http://127.0.0.1:17317/health
Invoke-RestMethod http://127.0.0.1:17317/scanner/status
```

Expected runtime facts before automatic scan:

- `isElevated: true`
- `genshinFound: true`
- `targetProcess: "GenshinImpact"`
- `/health.appBuild.signature` matches `APP_RUNTIME_SIGNATURE` in
  `electron/main.ts`

## Mouse And Click Validation

Use a probe before broad auto-scan work:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
  ConvertTo-Json -Depth 12
```

The probe performs one read-only inventory selection click. It does not delete,
feed, enhance, lock, unlock, spend, or modify game resources.

Interpretation:

- `click.ok: true`, `clicked: true`, and `inputBlocked: false` means Windows did
  not block SendInput/UIPI in the current configuration.
- `focused: true` and `foregroundProcess: "GenshinImpact"` means the click was
  sent while Genshin was foreground.
- `changed: true` means the detail panel changed after the click.
- `changed: false` can be benign if the target tile was already selected or two
  neighboring artifacts render identically; retry with another `index`, `row`,
  or `col`.

## Bounded Live Auto-Scan

Start with a tiny bounded run:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"
```

Then poll:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12
```

The scan starts only after a valid lookup package, supported 16:9 layout,
detected artifact grid, Genshin-client capture, and visual artifact-detail
markers are all present. If any preflight check fails, keep using the
visible-inventory path and fix the blocking evidence before trying broader
runs.

The loop verifies detail fingerprints after clicks, retries one unchanged detail
read, stores parsed artifacts in a batch, and flushes pending writes before the
final summary. Stats separate clicked, attempted, verified, parsed, stored,
review, duplicates, and misses so progress is not confused with successful
reads.

For the native IK-style path, prefer the dedicated smoke runner:

```powershell
npm run scan:native:smoke
npm run scan:native:smoke:5
```

It checks `/health`, `/scanner/native/data`,
`/scanner/native/preflight?category=artifacts` including the native visual
blank-capture guard, runs the guarded
`/automation/probe-click?index=1`, starts `/scanner/start?limit=N&category=artifacts`,
waits for the native helper to finish, runs `/scanner/native/process` with
`persist=0`, and loads `/scanner/native/results`. Evidence is written to:

```text
outputs/native-live-smoke/<timestamp>/
```

Use this native smoke path before enabling any artifact-store promotion from
native `scan-results.json`.

The native preflight and start endpoints both accept an explicit category:

```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/native/preflight?category=artifacts"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2&category=artifacts"
```

Only `artifacts` is implemented as native capture today. `weapons`,
`characters`, and `materials` are expected to block with a catalog-only message
at preflight/start time until their category-specific capture flows are
implemented and validated. They are not part of the active UI scope while those
values are not scanned.
Artifact preflight also blocks when the Genshin client capture is blank,
almost entirely white/black, or too visually uniform to be trusted.
The probe-click endpoint additionally requires a detected Genshin-client
artifact inventory grid and visible artifact detail card before sending input.

## Evidence Commands

After the elevated app is running and Genshin is open on the artifact inventory,
the non-elevated terminal can drive the local dev-control endpoints and save a
full evidence bundle:

```powershell
npm run scan:soak
```

The helper writes timestamped JSON snapshots and a transcript to:

```text
outputs/live-soak/<timestamp>/
```

For short iteration:

```powershell
npm run scan:iterate:validated
npm run scan:iterate:validated:wait
```

For a full 2, 5, 20, 45, 100 current-run evidence chain:

```powershell
npm run scan:goal:validated
npm run scan:goal:validated:wait
```

For later-session repeatability without changing OCR engines:

```powershell
npm run scan:repeatability:wait
```

Validate a saved assessment before using it as final evidence:

```powershell
npm run scan:assessment:validate -- --latest --summary
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json --summary
```

Optional budget flags:

```powershell
npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120
```

`--max-active-average-ms=333` is the strict 3 artifacts/second check. A passing
100-artifact current run is valid scanner evidence; it is not evidence that the
3 artifacts/second budget was met unless this budget check also passes.

The assessment self-test does not need Genshin:

```powershell
npm run scan:assessment:test
```

## Review-To-Eval Quality Loop

After any live scan that creates review samples, export candidates before adding
anything to the permanent eval corpus:

```powershell
npm run eval:review-candidates -- --limit=80
```

Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
worklist, not ground truth. Only after expected fields are confirmed or
corrected against the real artifact should a case move into
`src/eval/corpus/confirmedReviewCorpus.ts`.

For a manually checked candidate, generate a paste-ready confirmed-case snippet:

```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
```

## Anti-Cheat And Safety Boundary

Do not describe the implementation as bypassing anti-cheat. The app does not
read memory, hook the process, inject code, modify game files, inspect packets,
or interact with kernel drivers. It uses normal Windows screen capture, focus,
cursor movement, wheel, and click input.

Never add automation that deletes, feeds, enhances, locks/unlocks, spends
resources, reads memory, hooks, injects, or modifies Genshin.

## Live Layout Facts

The current 16:9 layout profile is calibrated from a 1920x1080 English
artifact-inventory capture:

- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
- inventory grid: `8 x 4` safe automated targets
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`

The profile is resolution-scaled for 16:9. Off-profile setups should be treated
as higher risk and validated with Smart Capture plus the probe.

## Validation Checklist

Before marking an automation change done:

1. Run `npm run lint`.
2. Run `npx tsc -p tsconfig.electron.json` when Electron/preload/main changed.
3. Run `npm test`.
4. Run `npm run build`.
5. If Genshin is available, run `/automation/probe-click?index=1`.
6. For scan-loop changes, run
   `/scanner/start?entry=visible-inventory&limit=2` before any broader scan.
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.