genshin-assistant/docs/AUTOMATION_LIVE_SCAN.md

Automation Live Scan Runbook

This document is the durable reference for automatic artifact scanning, mouse movement, click input, elevation, and live validation status.

Current Known-Good State

Validated live with Genshin open in the artifact inventory at 1920x1080, English UI:

• npm run dev:admin starts the app elevated after the user confirms UAC.
• Runtime status reports isElevated: true, genshinFound: true, and targetProcess: "GenshinImpact".
• /automation/probe-click?index=1 performs one read-only inventory tile selection click.
• The visible-inventory auto-scan path is the production baseline. It requires the Artifact inventory to already be open with a visible artifact detail card.
• On 2026-07-09, /scanner/start?entry=visible-inventory&limit=100 completed with 100/100 attempted, verified, and parsed, 98 stored, 2 duplicates, 0 review samples, 0 misses, 4 pages, and 393 ms/artifact.
• On 2026-07-09, npm run scan:native:smoke completed against runtime signature 2026-07-09-native-ik-visual-probe: native visual preflight passed at 1920x1080 with 32 grid targets, guarded probe changed the detail panel, native capture wrote 2/2 artifact card crops in 510 ms, and post-capture processing parsed 2/2 with 0 review, 0 errors, queueConcurrency: 2, and persisted: false.
• The same native path later completed dry 20/50/100-item runs. The final run captured 100/100 crops across 4 pages in 24,673 ms, parsed 100/100, produced 0 errors, routed 3 implausible OCR values into Review, and wrote nothing to the artifact store. Full evidence and the parser/timestamp fixes are recorded in NATIVE_SCANNER_VALIDATION_2026-07-09.md.
• On 2026-07-10, a fresh later-session chain passed with the hardened native helper: 5/20/50/100, all parsed/evaluated, 0 Review, 0 processing errors, and 0 store writes. The new 100er captured 100/100 over four pages at 4.37/s and processed at 3.15/s, with no duplicate PNG hash group, repeated page, or identical page boundary.
• A later real Settings-UI acceptance selected Eigenes Limit -> Artefakte -> 5. Genshin, visible detail, and the 4x8 grid passed preflight; target/captured/processed/parsed were all 5, Review and processing errors were 0, persistence stayed disabled, and the current-session result rail contained five entries. A later packaged row-limit run resolved one detected row to eight Artifacts and passed 8/8 with the same non-persistence boundary.
• The packaged full-owned-inventory run 20260710-223233 resolved the target from the OCR-confirmed owned count 2,211, captured all 2,211 crops over 70 pages, streamed results during capture, and produced zero processing errors and zero store writes. Its first parser pass correctly failed the 15% Review gate at 490 Review. After the structurally safe five-star +0 repair, the current package reprocessed the same complete corpus at 136 Review (6.15%); the saved-run validator then passed with zero issues.
• The bounded inventory-top reset now completes in about 1.7 s instead of about 50.7 s; the 3,200-event range and per-event foreground/stop guards remain intact.
• A packaged build started from %TEMP% and completed a native 5er through renderer, preload, IPC, packaged helper, and packaged IK resources with persist=false. See NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md.

The old alternate OCR-engine and comparison paths have been retired. The app now runs the current OCR/capture path only. Performance claims should be stated as current-run repeatability evidence, not as external-tool parity.

Elevation And UAC

Use:

npm run dev:admin

The command runs scripts/dev-admin.ps1, which launches a new elevated PowerShell window running scripts/dev-admin-start.ps1. The elevated start is logged to:

outputs/admin-start/admin-dev.log

The user must confirm the Windows UAC prompt. The app cannot and must not click the Secure Desktop UAC prompt for itself.

Useful checks:

Invoke-RestMethod http://127.0.0.1:17317/health
Invoke-RestMethod http://127.0.0.1:17317/scanner/status

Expected runtime facts before automatic scan:

• isElevated: true
• genshinFound: true
• targetProcess: "GenshinImpact"
• /health.appBuild.signature matches APP_RUNTIME_SIGNATURE in electron/main.ts

Mouse And Click Validation

Use a probe before broad auto-scan work:

Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
  ConvertTo-Json -Depth 12

The probe performs one read-only inventory selection click. It does not delete, feed, enhance, lock, unlock, spend, or modify game resources.

Interpretation:

• click.ok: true, clicked: true, and inputBlocked: false means Windows did not block SendInput/UIPI in the current configuration.
• focused: true and foregroundProcess: "GenshinImpact" means the click was sent while Genshin was foreground.
• changed: true means the detail panel changed after the click.
• changed: false can be benign if the target tile was already selected or two neighboring artifacts render identically; retry with another index, row, or col.

Bounded Live Auto-Scan

Start with a tiny bounded run:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"

Then poll:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12

The scan starts only after a valid lookup package, supported 16:9 layout, detected artifact grid, Genshin-client capture, and visual artifact-detail markers are all present. If any preflight check fails, keep using the visible-inventory path and fix the blocking evidence before trying broader runs.

The loop verifies detail fingerprints after clicks, retries one unchanged detail read, stores parsed artifacts in a batch, and flushes pending writes before the final summary. Stats separate clicked, attempted, verified, parsed, stored, review, duplicates, and misses so progress is not confused with successful reads.

For the native IK-style path, prefer the dedicated smoke runner:

npm run scan:native:smoke
npm run scan:native:smoke:5

For the gated live scale chain, run and validate each size before continuing:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\native-live-smoke.ps1 -Limit 20
npm run scan:native:validate:saved -- --targets=20
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\native-live-smoke.ps1 -Limit 50
npm run scan:native:validate:saved -- --targets=50
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\native-live-smoke.ps1 -Limit 100
npm run scan:native:validate:saved -- --targets=100

It checks /health, /scanner/native/data, /scanner/native/preflight?category=artifacts including the native visual blank-capture guard, runs the guarded /automation/probe-click?index=1, starts /scanner/start?limit=N&category=artifacts, waits for the native helper to finish, runs /scanner/native/process with persist=0, and loads /scanner/native/results. Evidence is written to:

outputs/native-live-smoke/<timestamp>/

Use this native smoke path before enabling any artifact-store promotion from native scan-results.json.

For a deliberate packaged-runtime acceptance, first build with npm run package:offline-check, then launch the unpacked EXE from a foreign working directory with a loopback-only temporary CDP port:

$exe = (Resolve-Path "outputs\dist\win-unpacked\Genshin Artifact Assistant.exe").Path
Start-Process -FilePath $exe `
  -ArgumentList "--remote-debugging-address=127.0.0.1", "--remote-debugging-port=9223" `
  -WorkingDirectory $env:TEMP -Verb RunAs
npm run package:live:inspect

package:live:inspect is read-only. The following command is a real bounded five-card Genshin scan and therefore requires the same live authorization and preconditions as every other input test:

npm run package:live:scan5

The runner calls packaged renderer/preload/IPC methods serially, validates the 5-result non-persistence contract, and writes JSON plus a screenshot below outputs/packaged-live/. Close the packaged app after testing so the temporary CDP port disappears.

For the streaming pipeline, use the real Settings-UI path with a bounded target large enough to observe overlap:

npm run package:live:ui-stream20

This acceptance is stricter than a correct final count. It requires native capture to be observed as running while at least one processing worker has finished and at least one current-session result card is already visible. It also records the maximum evaluated/result counts seen during capture, verifies all 20 final entries, and checks the insertion animation unless the operating system requests reduced motion. Persistence remains disabled.

The same packaged Settings-UI runner exposes all accepted scope paths:

npm run package:live:ui-scan5
npm run package:live:ui-row1
npm run package:live:ui-stream20
npm run package:live:ui-full-inventory

package:live:ui-full-inventory explicitly selects Gesamtes Inventar and never supplies a numeric fallback. It accepts the target only from the running scanner status after the app has read the current owned Artifact count. The runner rejects targets outside 1..2400, requires exact captured/processed/ parsed/result/UI-row counts, checks pages = ceil(target / 32), enforces at most 15% Review with zero processing errors, verifies streaming overlap and document fit, and requires persist=false with zero store writes. Its timeout is 15 minutes. The real elevated run 20260710-223233 reached 2,211/2,211 over 70 pages. Because the initial parser produced 490 Review, the gate rejected the first result honestly; current-package reprocessing reduced that to 136 Review (6.15%), and scan:native:validate:saved -- --targets=2211 passed with zero issues. This two-stage evidence is the accepted full-inventory baseline.

The native preflight and start endpoints both accept an explicit category:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/native/preflight?category=artifacts"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2&category=artifacts"

Only artifacts is implemented as native capture today. weapons, characters, and materials are expected to block with a catalog-only message at preflight/start time until their category-specific capture flows are implemented and validated. They are not part of the active UI scope while those values are not scanned. Artifact preflight also blocks when the Genshin client capture is blank, almost entirely white/black, or too visually uniform to be trusted. The probe-click endpoint additionally requires a detected Genshin-client artifact inventory grid and visible artifact detail card before sending input.

At native Artifact-scan startup, the helper first resets the inventory to the top with a bounded upward wheel sequence at the calibrated grid anchor and then waits for the UI to stabilize. Genshin is focused only for initial startup. The native loop checks that the same Genshin HWND is still foreground immediately before every click and every wheel event; focus loss stops the run without an automatic refocus. Stop, ESC, Enter, and F9 remain active during the reset, stabilization, clicks, captures, and page scrolling. Native status reports initialTopResetMs and totalMs separately; activeMs starts after the reset and remains the capture-rate denominator.

Do not replace this production reset with B plus an Artifact-tab click based on event count alone. Reopening depends on the current menu/inventory state and adds a key toggle plus a coordinate target. It has not been live benchmarked against the accepted bounded wheel reset. If evaluated later, run it as a separate guarded A/B probe and compare successful top-state detection, focus/ stop behavior, coordinate robustness, and elapsed reset time before changing the default.

Evidence Commands

The commands in this first subsection exercise the older visible-inventory engine and its assessment format. They remain useful for regression work on that engine, but they do not post-process or validate the current native C# capture path. Use native-live-smoke.ps1 plus scan:native:validate:saved for current native acceptance.

After the elevated app is running and Genshin is open on the artifact inventory, the non-elevated terminal can drive the local dev-control endpoints and save a full evidence bundle:

npm run scan:soak

The helper writes timestamped JSON snapshots and a transcript to:

outputs/live-soak/<timestamp>/

For short iteration:

npm run scan:iterate:validated
npm run scan:iterate:validated:wait

For a full 2, 5, 20, 45, 100 current-run evidence chain:

npm run scan:goal:validated
npm run scan:goal:validated:wait

For later-session repeatability without changing OCR engines:

npm run scan:repeatability:wait

Validate a saved assessment before using it as final evidence:

npm run scan:assessment:validate -- --latest --summary
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json --summary

Optional budget flags:

npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120

--max-active-average-ms=333 is the strict 3 artifacts/second check. A passing 100-artifact current run is valid scanner evidence; it is not evidence that the 3 artifacts/second budget was met unless this budget check also passes.

The historical native-live-smoke.ps1 path is intentionally serial: it waits for native capture to finish and only then invokes post-capture processing. Do not divide a target by the sum of those two durations and describe the result as the current streaming pipeline's end-to-end rate. The 2026-07-11 read-only streaming-20 check (20260711-090019) proved overlap (a result was visible before capture completed) and observed 20/20 clean results at 4.43 capture/s and 2.39/s from request issue to result reconciliation. That bounded observation is not a general performance claim: native start, capture start/end, processing start/end, and final reconciliation still need one persisted shared clock before publishing an end-to-end budget result.

The assessment self-test does not need Genshin:

npm run scan:assessment:test

Saved native results can be re-evaluated without Genshin, focus changes, or input automation:

npm run scan:native:replay -- --run-dir=<native-run-dir> --repeats=5

This is an offline determinism and result-contract check. It does not replace a live capture smoke test. The ordered checklist was executed on 2026-07-10; its results are in NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md.

Review-To-Eval Quality Loop

After any live scan that creates review samples, export candidates before adding anything to the permanent eval corpus:

npm run eval:review-candidates -- --limit=80

Read outputs/review-eval-candidates/review-eval-candidates.md. It is a review worklist, not ground truth. Only choose a candidate marked visualEvidence: available: it had a retrievable local PNG at export time. The export exposes only safe capture/run references, never the local crop path. After expected fields are confirmed or corrected against the real artifact, a case may move into src/eval/corpus/confirmedReviewCorpus.ts.

For a manually checked candidate, generate a paste-ready confirmed-case snippet:

npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json

The preparer refuses candidates without explicit labels or retrievable visual evidence. Re-export after retaining a crop rather than promoting an OCR-only legacy record.

Anti-Cheat And Safety Boundary

Do not describe the implementation as bypassing anti-cheat. The app does not read memory, hook the process, inject code, modify game files, inspect packets, or interact with kernel drivers. It uses normal Windows screen capture, focus, cursor movement, wheel, and click input.

Never add automation that deletes, feeds, enhances, locks/unlocks, spends resources, reads memory, hooks, injects, or modifies Genshin.

Live Layout Facts

The current 16:9 layout profile is calibrated from a 1920x1080 English artifact-inventory capture:

• detail rect approximately x=1308, y=120, width=492, height=838
• inventory grid: 8 x 4 safe automated targets
• first tile center: x=179, y=254, row=0, col=0
• second tile center: x=325, y=254, row=0, col=1

The profile is resolution-scaled for 16:9. Off-profile setups should be treated as higher risk and validated with Smart Capture plus the probe.

Validation Checklist

Before marking an automation change done:

1. Run npm run helper:test when native input safety changed.
2. Run npm run lint.
3. Run npx tsc -p tsconfig.electron.json when Electron/preload/main changed.
4. Run npm test.
5. Run npm run build.
6. If Genshin is available, run /automation/probe-click?index=1.
7. For native scan-loop changes, run npm run scan:native:smoke:5 before any 20/50/100 progression. Use the explicit visible-inventory endpoint only when that legacy engine itself changed.
8. Record new live findings in this file and in docs/scanner-rework-status.md.