genshin-assistant/docs/AUTOMATION_LIVE_SCAN.md

Automation Live Scan Runbook

This document is the durable reference for automatic artifact scanning, mouse movement, click input, elevation, and live validation status.

Current Known-Good State

Validated live with Genshin open in the artifact inventory at 1920x1080, English UI:

• npm run dev:admin starts the app elevated after the user confirms UAC.
• Runtime status reports isElevated: true, genshinFound: true, and targetProcess: "GenshinImpact".
• /automation/probe-click?index=1 performs one read-only inventory tile selection click.
• The visible-inventory auto-scan path is the production baseline. It requires the Artifact inventory to already be open with a visible artifact detail card.
• On 2026-07-09, /scanner/start?entry=visible-inventory&limit=100 completed with 100/100 attempted, verified, and parsed, 98 stored, 2 duplicates, 0 review samples, 0 misses, 4 pages, and 393 ms/artifact.
• On 2026-07-09, npm run scan:native:smoke completed against runtime signature 2026-07-09-native-ik-visual-probe: native visual preflight passed at 1920x1080 with 32 grid targets, guarded probe changed the detail panel, native capture wrote 2/2 artifact card crops in 510 ms, and post-capture processing parsed 2/2 with 0 review, 0 errors, queueConcurrency: 2, and persisted: false.
• The same native path later completed dry 20/50/100-item runs. The final run captured 100/100 crops across 4 pages in 24,673 ms, parsed 100/100, produced 0 errors, routed 3 implausible OCR values into Review, and wrote nothing to the artifact store. Full evidence and the parser/timestamp fixes are recorded in NATIVE_SCANNER_VALIDATION_2026-07-09.md.

The old alternate OCR-engine and comparison paths have been retired. The app now runs the current OCR/capture path only. Performance claims should be stated as current-run repeatability evidence, not as external-tool parity.

Elevation And UAC

Use:

npm run dev:admin

The command runs scripts/dev-admin.ps1, which launches a new elevated PowerShell window running scripts/dev-admin-start.ps1. The elevated start is logged to:

outputs/admin-start/admin-dev.log

The user must confirm the Windows UAC prompt. The app cannot and must not click the Secure Desktop UAC prompt for itself.

Useful checks:

Invoke-RestMethod http://127.0.0.1:17317/health
Invoke-RestMethod http://127.0.0.1:17317/scanner/status

Expected runtime facts before automatic scan:

• isElevated: true
• genshinFound: true
• targetProcess: "GenshinImpact"
• /health.appBuild.signature matches APP_RUNTIME_SIGNATURE in electron/main.ts

Mouse And Click Validation

Use a probe before broad auto-scan work:

Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
  ConvertTo-Json -Depth 12

The probe performs one read-only inventory selection click. It does not delete, feed, enhance, lock, unlock, spend, or modify game resources.

Interpretation:

• click.ok: true, clicked: true, and inputBlocked: false means Windows did not block SendInput/UIPI in the current configuration.
• focused: true and foregroundProcess: "GenshinImpact" means the click was sent while Genshin was foreground.
• changed: true means the detail panel changed after the click.
• changed: false can be benign if the target tile was already selected or two neighboring artifacts render identically; retry with another index, row, or col.

Bounded Live Auto-Scan

Start with a tiny bounded run:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"

Then poll:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12

The scan starts only after a valid lookup package, supported 16:9 layout, detected artifact grid, Genshin-client capture, and visual artifact-detail markers are all present. If any preflight check fails, keep using the visible-inventory path and fix the blocking evidence before trying broader runs.

The loop verifies detail fingerprints after clicks, retries one unchanged detail read, stores parsed artifacts in a batch, and flushes pending writes before the final summary. Stats separate clicked, attempted, verified, parsed, stored, review, duplicates, and misses so progress is not confused with successful reads.

For the native IK-style path, prefer the dedicated smoke runner:

npm run scan:native:smoke
npm run scan:native:smoke:5

It checks /health, /scanner/native/data, /scanner/native/preflight?category=artifacts including the native visual blank-capture guard, runs the guarded /automation/probe-click?index=1, starts /scanner/start?limit=N&category=artifacts, waits for the native helper to finish, runs /scanner/native/process with persist=0, and loads /scanner/native/results. Evidence is written to:

outputs/native-live-smoke/<timestamp>/

Use this native smoke path before enabling any artifact-store promotion from native scan-results.json.

The native preflight and start endpoints both accept an explicit category:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/native/preflight?category=artifacts"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2&category=artifacts"

Only artifacts is implemented as native capture today. weapons, characters, and materials are expected to block with a catalog-only message at preflight/start time until their category-specific capture flows are implemented and validated. They are not part of the active UI scope while those values are not scanned. Artifact preflight also blocks when the Genshin client capture is blank, almost entirely white/black, or too visually uniform to be trusted. The probe-click endpoint additionally requires a detected Genshin-client artifact inventory grid and visible artifact detail card before sending input.

Evidence Commands

After the elevated app is running and Genshin is open on the artifact inventory, the non-elevated terminal can drive the local dev-control endpoints and save a full evidence bundle:

npm run scan:soak

The helper writes timestamped JSON snapshots and a transcript to:

outputs/live-soak/<timestamp>/

For short iteration:

npm run scan:iterate:validated
npm run scan:iterate:validated:wait

For a full 2, 5, 20, 45, 100 current-run evidence chain:

npm run scan:goal:validated
npm run scan:goal:validated:wait

For later-session repeatability without changing OCR engines:

npm run scan:repeatability:wait

Validate a saved assessment before using it as final evidence:

npm run scan:assessment:validate -- --latest --summary
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json --summary

Optional budget flags:

npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120

--max-active-average-ms=333 is the strict 3 artifacts/second check. A passing 100-artifact current run is valid scanner evidence; it is not evidence that the 3 artifacts/second budget was met unless this budget check also passes.

The assessment self-test does not need Genshin:

npm run scan:assessment:test

Review-To-Eval Quality Loop

After any live scan that creates review samples, export candidates before adding anything to the permanent eval corpus:

npm run eval:review-candidates -- --limit=80

Read outputs/review-eval-candidates/review-eval-candidates.md. It is a review worklist, not ground truth. Only after expected fields are confirmed or corrected against the real artifact should a case move into src/eval/corpus/confirmedReviewCorpus.ts.

For a manually checked candidate, generate a paste-ready confirmed-case snippet:

npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json

Anti-Cheat And Safety Boundary

Do not describe the implementation as bypassing anti-cheat. The app does not read memory, hook the process, inject code, modify game files, inspect packets, or interact with kernel drivers. It uses normal Windows screen capture, focus, cursor movement, wheel, and click input.

Never add automation that deletes, feeds, enhances, locks/unlocks, spends resources, reads memory, hooks, injects, or modifies Genshin.

Live Layout Facts

The current 16:9 layout profile is calibrated from a 1920x1080 English artifact-inventory capture:

• detail rect approximately x=1308, y=120, width=492, height=838
• inventory grid: 8 x 4 safe automated targets
• first tile center: x=179, y=254, row=0, col=0
• second tile center: x=325, y=254, row=0, col=1

The profile is resolution-scaled for 16:9. Off-profile setups should be treated as higher risk and validated with Smart Capture plus the probe.

Validation Checklist

Before marking an automation change done:

1. Run npm run lint.
2. Run npx tsc -p tsconfig.electron.json when Electron/preload/main changed.
3. Run npm test.
4. Run npm run build.
5. If Genshin is available, run /automation/probe-click?index=1.
6. For scan-loop changes, run /scanner/start?entry=visible-inventory&limit=2 before any broader scan.
7. Record new live findings in this file and in docs/scanner-rework-status.md.