genshin-assistant/docs/AUTOMATION_LIVE_SCAN.md

Automation Live Scan Runbook

This document is the durable reference for automatic artifact scanning, mouse movement, click input, elevation, and live validation status.

Current Known-Good State

Validated live on 2026-07-07 with Genshin open in the artifact inventory at 1920x1080, English UI:

• npm run dev:admin starts the app elevated after the user confirms UAC.
• Runtime status reported isElevated: true, genshinFound: true, and targetProcess: "GenshinImpact".
• The safe probe endpoint /automation/probe-click?index=1 focused Genshin, moved the cursor to the second visible inventory tile, clicked it, and changed the artifact detail panel fingerprint.
• Probe result: clicked: true, inputBlocked: false, foregroundProcess: "GenshinImpact", and changed: true.
• A bounded live auto-scan via /scanner/start?limit=2 completed with: clicked: 2, attempted: 2, verified: 2, parsed: 2, stored: 2, review: 2, misses: 0, status: "done".

This proves that the current elevated app plus helper path can deliver mouse movement and click input to the focused Genshin client in this environment.

Latest-source timing is not proven while /health.appBuild.signature differs from the APP_RUNTIME_SIGNATURE in electron/main.ts. On 2026-07-07 the port was still owned by an older elevated runtime, so goal scans were intentionally blocked by the stale-build gate. Restart the elevated app through npm run dev:admin and confirm UAC before collecting new 100-artifact evidence.

Elevation And UAC

Use:

npm run dev:admin

The command runs scripts/dev-admin.ps1, which launches a new elevated PowerShell window running scripts/dev-admin-start.ps1. The elevated start is logged to:

outputs/admin-start/admin-dev.log

The user must confirm the Windows UAC prompt. The app cannot and must not click the Secure Desktop UAC prompt for itself. After confirmation, the app can verify its own runtime through the dev status endpoint.

Useful checks:

Invoke-RestMethod http://127.0.0.1:17317/health
Invoke-RestMethod http://127.0.0.1:17317/scanner/status

Expected runtime facts before automatic scan:

• isElevated: true
• genshinFound: true
• targetProcess: "GenshinImpact"
• hotkeys registered

Mouse And Click Validation

Use the probe before broad auto-scan work:

Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
  ConvertTo-Json -Depth 12

The probe performs one read-only inventory selection click. It does not delete, feed, enhance, lock, unlock, spend, or modify game resources.

Interpretation:

• click.ok: true, clicked: true, inputBlocked: false means Windows did not block SendInput/UIPI in the current configuration.
• focused: true and foregroundProcess: "GenshinImpact" means the click was sent while Genshin was foreground.
• changed: true means the detail panel changed after the click.
• changed: false can be benign if the target tile was already selected or two neighboring artifacts render identically; retry with another index, row, or col.

Examples:

# Second visible tile
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1"

# Specific grid cell
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?row=0&col=3"

Bounded Live Auto-Scan

For live validation, prefer a bounded scan first:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"

The visible-inventory path remains the safest first check. The normal guided entry tries the read-only direct world path first: B -> artifact tab -> first artifact tile. If that does not produce a visible artifact detail card, it falls back to the Inventory Kamera-compatible sequence: ESC -> B -> artifact tab -> first artifact tile.

The normal Auto-Scan button uses a guided start. It first takes one lightweight preflight capture without OCR, full-frame payload, review scoring, or storing. If an artifact detail card is already visible, it starts the visible-inventory scan. Otherwise it runs the guided entry above. OCR/review/store work starts only after the artifact-detail preflight passes. Guided entry uses short state polling for the Inventory screen, artifact grid, and first detail card instead of waiting the full fixed delay every time; if the state never appears, the same timeout budget returns the last diagnostic capture.

Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=auto-entry&limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2&engine=ik-traineddata"

Those paths send only read-only navigation. ESC is not a universal "go to world" command: from the world it opens the Paimon menu, while from the already-open Paimon menu it returns to the world. This is why the normal auto-entry path first tries B directly and uses the IK-style ESC -> B fallback only when direct entry did not reach an artifact detail card.

The scan starts only after a valid lookup package, supported 16:9 layout, detected artifact grid, Genshin-client capture, and visual artifact-detail markers are all present. If any preflight check fails, keep using the visible-inventory path while tuning the entry step.

The visual preflight also classifies the Paimon menu. The Paimon profile/card grid can look like an inventory grid if only fixed 16:9 coordinates are used, so the scanner must reject paimonMenu.present before any artifact OCR, review sample creation, store write, or grid scan starts. The guided entry may still take lightweight skip-OCR captures while navigating, but those captures are only state evidence.

The same guard also runs inside the scan loop. If the app is on the main game screen, a Paimon/menu screen, a generic primary-screen capture, or any screen without an artifact detail card, auto-scan must block instead of clicking tiles or trying OCR. After each click the loop polls the detail fingerprint with a short bounded budget instead of sleeping blindly. The current budget is 420 ms with 60 ms polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it does not change the loop retries or stops through the normal miss guards. If the card changed but remains animated, the loop now proceeds after 200 ms, matching Inventory Kamera's select-next-item wait more closely without removing the detail-change guard. The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do not re-run the focus helper before every tile, which avoids an OS focus ping on each artifact while still relying on click readback, foreground checks, and the detail-card guard for safety. After a scroll, the loop now uses the same cheap fingerprint polling model for the inventory pane: it proceeds as soon as the next page fingerprint changed and stabilized instead of always sleeping the old fixed 760 ms settle delay. Changed but still animated inventory pages may proceed after 100 ms, again matching IK's fast-scroll wait while still blocking unchanged pages.

Then poll:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 12

Before live timing, verify that the endpoint is the current app instance:

Invoke-RestMethod "http://127.0.0.1:17317/health" |
  ConvertTo-Json -Depth 6

The response must include appBuild.signature and appBuild.expectedOcrWorkerPoolSize. If appBuild is missing, or /scanner/status still reports the old OCR warmup start time, the local port is still owned by a stale elevated Electron process. Close the old Administrator window/app and restart with npm run dev:admin before running scanner probes.

Use the status stats timing fields for IK comparisons: elapsedMs, activeScanMs, writeFlushMs, averageMsPerParsed, activeAverageMsPerParsed, averageCaptureMs, averageOcrMs, artifactsPerMinute, and projectedMsFor100. elapsedMs is end-to-end including queued writes; activeScanMs is the click/capture/OCR loop before the final store/review flush. A run only counts as speed evidence when parsed, stored, review, duplicates, and misses are read together; raw click count alone is not scanner throughput. If averageOcrMs dominates averageMsPerParsed, the next speed lever is an IK-style OCR worker queue. If averageCaptureMs dominates, crop payload/capture work is the bottleneck.

The /scanner/start?limit=N endpoint sends a renderer command payload with a temporary scan limit. It does not change the normal UI setting. The normal hotkeys and buttons still use the UI's configured scan limit.

Lookup and benchmark utility endpoints:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/status"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/regenerate"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup?engine=ik-traineddata"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=ik-traineddata"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=compare"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&profile=full"

The benchmark endpoint measures the current Tesseract.js engine and the Inventory-Kamera-traineddata Tesseract.js path against the artifact crop set and returns timing/field counts, min/p50/p90/max timing, OCR p50/p90 timing, 20/45/100-artifact projections, skipped-OCR count, and the active OCR worker pool size. It also returns per-field OCR timings under ocrFieldAverages, which is the first place to look before changing crop or parser behavior. Individual captures also report whether the artifact was detected as sanctified; level/substat crops are shifted in that state to match Inventory Kamera's crop model. By default it uses the auto-scan fast OCR profile, which omits the low-value set-effect crop, the slot crop that can be derived from the matched artifact piece name, and the main-stat-value crop that can be derived from slot, main-stat label, and level. The fast profile also uses Inventory Kamera's tighter substat crop height; full/manual captures keep the larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR captures while keeping the detail screenshot, OCR text, crop rect metadata, and timings. OCR crops are passed to Tesseract as PNG buffers internally, not as Base64 DataURLs, to avoid encode/decode overhead in batch scans. When skipOcrUnlessArtifactDetail blocks OCR because no artifact detail card is visible, OCR crop preprocessing is skipped too. Auto-scan readiness and scroll checks use native detail/inventory fingerprints and omit preview DataURLs in poll captures. Fast preflight/poll captures also omit crop list construction, crop images, and lock-state detection unless a caller explicitly overrides that option; add profile=full to OCR every artifact detail crop for debugging. It uses the same artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is skipped and the response shows skippedOcrCaptures instead of burning time on invalid crops. For speed, the fast auto-scan profile also skips the optional Equipped footer OCR. Name, level, main-stat label, and substats remain in the OCR hot path; slot, set, and main-stat value are derived when the lookup/parser can validate them. Use a full/manual capture when equipped ownership or every debug crop matters. Local store/review writes are serialized through an internal queue but no longer block the next inventory click. The scan still flushes the queue before it returns its final summary, so stored and review counts remain final-state numbers. The app warms the default OCR worker pool in the background after startup; check /scanner/status -> ocrWarmup.current before timing the first artifact. Use /scanner/ocr/warmup?engine=ik-traineddata before comparing Inventory Kamera-traineddata timings so the benchmark is not dominated by worker creation. engine=ik-traineddata uses Inventory Kamera's local genshin_fast_09_04_21.traineddata through Tesseract.js when the file is found in data/tessdata, IK_TESSDATA_DIR, work/Inventory_Kamera, work/refs, or the local _ik_ref* folders. engine=compare runs current and ik-traineddata against the same visible artifact detail state. The auto-scan default must stay current until the IK traineddata path wins on the same captures. For a controlled live comparison, start the scanner with engine=ik-traineddata; this only changes the OCR worker language for that run and leaves the default UI/hotkey path on current. The OCR pool defaults to four workers because the fast artifact crop set has four useful OCR parameter groups; set GAA_OCR_WORKERS=1..8 before startup to benchmark a different worker count. Inventory Kamera's native engine pool is still the reference design, but the current app path remains Tesseract.js until native OCR is integrated and measured. Crops are scheduled across the whole worker pool and each worker caches its last Tesseract parameter profile; this is closer to Inventory Kamera's multi-engine field OCR than the earlier parameter-group-serial scheduler.

Diagnostic Evidence

The Diagnose page contains a compact evidence timeline for scanner work. It logs runtime pings, focus attempts, key presses, entry captures, artifact-tab clicks, preflight failures, grid/count metadata, detail fingerprints, and detail/inventory screenshots. The same last events are also published through:

Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
  ConvertTo-Json -Depth 18

Use this before changing scanner behavior: run the smallest failing action, read the evidence timeline, then decide whether the failure is focus/input, entry navigation, grid detection, capture quality, OCR, or parser validation.

If Paimon entry shows entry key ESC or entry key B with eventsSent: 0, the running InputHelper.exe probably predates keyboard support or is blocked. Stop the elevated app/helper, run npm run helper:build, then restart with npm run dev:admin so the app loads the rebuilt helper.

Soak-Test Helper

After the elevated app is running and Genshin is open on the artifact inventory, the non-elevated terminal can drive the local dev-control endpoints and save a full evidence bundle:

npm run scan:soak

The helper writes timestamped JSON snapshots and a transcript to:

outputs/live-soak/<timestamp>/

Default sequence:

1. /health
2. /scanner/status
3. /capture/smart?skipOcr=1
4. /automation/probe-click?index=1
5. /automation/probe-click?index=3
6. /scanner/start?limit=2
7. /scanner/start?limit=5
8. /scanner/start?limit=10
9. /scanner/start?limit=20
10. /review/samples?limit=30

For the actual Inventory-Kamera speed target, use the explicit goal run after /health shows the current appBuild:

npm run scan:goal
npm run scan:goal:current
npm run scan:goal:ik
npm run scan:goal:compare

That run first warms/benchmarks current vs. ik-traineddata, then scans limits 2, 5, 20, 45, 100 with the selected scan engine, and writes scan-run-summary.json plus scan-run-summary.csv. npm run scan:goal uses the default current scan engine; use scan:goal:ik for a native IK-traineddata scan pass. Use scan:goal:compare to run both scan engines back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for averageMsPerParsed, activeAverageMsPerParsed, averageCaptureMs, captureP50Ms, captureP90Ms, averageOcrMs, ocrP50Ms, ocrP90Ms, averageCardReadyMs, averageScrollReadyMs, artifactsPerMinute, and projectedMsFor100. The run also writes scan-performance-assessment.json, which groups results by limit, picks the best qualified engine, and labels the dominant bottleneck as OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the run, parse the requested count, keep miss rate under 2%, and keep review rate at or below 15%; review and miss rates are penalized before active average speed is used as the tie-breaker.

The assessment ranking can be verified without Genshin or the Electron app:

npm run scan:assessment:test

This self-test rejects synthetic runs that are fast but have too many misses or too many review samples, so the final IK comparison cannot be won by speed alone.

For the current implementation summary and IK comparison rationale, see scanner-ik-progress-report.md.

Use the readiness timings to compare against Inventory Kamera's fixed waits: IK waits about 200 ms after selecting the next inventory item and about 100 ms after fast scrolls. If averageCardReadyMs or averageScrollReadyMs dominates the active average while OCR is already low, tune the fingerprint gate before touching OCR again.

The runner reads APP_RUNTIME_SIGNATURE from electron/main.ts and refuses to run against a stale Electron process when /health.appBuild.signature does not match the current source. Use -AllowStaleBuild only for deliberate debugging of an older instance. Current dev builds also expose /dev/shutdown on localhost. The start cleanup script calls it before falling back to Stop-Process, so a previous elevated app can shut itself down cleanly even when the caller cannot terminate an administrator process directly. Older builds without that endpoint still need manual close or a confirmed npm run dev:admin restart.

Review samples are saved as a compact summary by default so Vite does not try to watch large Base64 payloads under outputs/. Full review payloads can be saved with -SaveFullReviewSamples when needed.

It stops on a failed probe, blocked scan, stopped scan, or timeout unless -ContinueAfterBlocked is supplied directly:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\live-soak.ps1 -Limits 2,5 -ContinueAfterBlocked

Anti-Cheat And Safety Boundary

Do not describe the current implementation as bypassing anti-cheat. The app does not read memory, hook the process, inject code, modify game files, inspect packets, or interact with kernel drivers. It uses normal Windows screen capture, focus, cursor movement, wheel, and click input.

The practical finding is narrower:

• A non-elevated app can be blocked by Windows integrity/UIPI when the target process is elevated or protected.
• Running the app elevated fixed input delivery in the tested environment.
• Genshin's anti-cheat may still affect behavior on other machines, game modes, overlays, or future versions. Re-run the probe before trusting broad scans.

Never add automation that deletes, feeds, enhances, locks/unlocks, spends resources, reads memory, hooks, injects, or modifies Genshin.

Live Layout Facts

The current 16:9 layout profile is calibrated from a 1920x1080 English artifact-inventory capture:

• detail rect approximately x=1308, y=120, width=492, height=838
• inventory grid: 8 x 4 safe automated targets, matching Inventory Kamera's 32-artifact full-page model. The apparent lower fifth row is in the bottom control band and is intentionally not clicked during auto-scan.
• first tile center: x=179, y=254, row=0, col=0
• second tile center: x=325, y=254, row=0, col=1
• inventory count crop successfully read 2059/2400 in the live session

The profile is resolution-scaled for 16:9. Off-profile setups should be treated as higher risk and validated with Smart Capture plus the probe.

Validation Checklist

Before marking an automation change done:

1. Run npm run lint.
2. Run npx tsc -p tsconfig.electron.json when Electron/preload/main changed.
3. Run npm test.
4. Run npm run build.
5. If Genshin is available, run /automation/probe-click?index=1.
6. For scan-loop changes, run /scanner/start?limit=2 before any broader scan.
7. Record new live findings in this file and in docs/scanner-rework-status.md.
8. For IK-target claims, attach or cite scan-performance-assessment.json from a non-stale npm run scan:goal:compare run.