bao/genshin-assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(scanner): validate elevated live automation

Browse Source
This commit is contained in:
AzuTear
2026-07-07 07:49:22 +02:00
parent 7930e369a7
commit ef65c3e6a0
37 changed files with 826 additions and 217 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+17 -2
View File
@@ -49,7 +49,7 @@ flowchart LR
| Module | Responsibility |
| --- | --- |
| `electron/main.ts` | Window lifecycle, capture source listing, Smart Capture, OCR crop generation, overlay window IPC, persistent PowerShell input/capture helper, JSON artifact store |
| `electron/main.ts` | Window lifecycle, capture source listing, Smart Capture, OCR crop generation, overlay window IPC, input/capture sidecar orchestration, JSON artifact store, dev-only scanner control endpoints |
| `electron/preload.cjs` | Safe renderer bridge exposed as `window.assistantApi` |
| `src/lib/artifactStore.ts` | Pure signature/id/record helpers for the persistent artifact store |
| `src/App.tsx` | Main app shell, scan view, triage view, build view, overlay preview |
@@ -111,9 +111,24 @@ sequenceDiagram
**Automatic grid scan** is user-triggered input automation limited to clicking detected inventory tiles and wheel-scrolling the inventory. Safety and reliability rules:
- All input goes through one persistent PowerShell helper process (`input-helper.ps1` in userData) that compiles the Win32 interop once and speaks JSON over stdin/stdout (ops: ping, focus, cursor, click, scroll, capture). Mouse movement is sent as iterated relative SendInput deltas (what a real mouse produces): Genshin tracks the cursor via raw input and snaps the OS cursor back to its own position every frame, so SetCursorPos/absolute moves silently stop working once the game owns the cursor. The helper verifies the cursor reached the target and refuses to click otherwise.
- All input goes through the helper service boundary (currently a C# sidecar with
fallback support behind the same JSON protocol). The helper owns focus, cursor
movement, click, scroll, guard-state polling, elevation detection, and GDI
capture. Mouse movement is sent as iterated relative input deltas instead of
relying on a single absolute cursor jump. The helper verifies the cursor
reached the target and refuses to click otherwise.
- `npm run dev:admin` is the validated dev path for automation when elevated
input is required. The elevated PowerShell startup is handled by
`scripts/dev-admin.ps1` and logged to `outputs/admin-start/admin-dev.log`.
The user must approve UAC manually; the app cannot approve the Secure Desktop
prompt itself.
- Failsafe: before every click and scroll the renderer polls cursor position and ESC state. Holding ESC or moving the mouse away from the last automated position aborts the scan immediately; the Stop button also aborts. Only the `GetAsyncKeyState` held-down bit (0x8000) is used - the "pressed since last call" bit fires for stale ESC presses from normal Genshin menu navigation and caused false aborts.
- SendInput's return value is checked: zero injected events (UIPI, e.g. elevated Genshin vs. non-elevated app) aborts with an explicit hint instead of silently clicking into nothing.
- Dev-only probes under `http://127.0.0.1:17317` are used for live validation:
`/automation/probe-click?index=N` tests one read-only tile selection, and
`/scanner/start?limit=N` starts an auto-scan with a temporary limit payload.
The live known-good result on 2026-07-07 is documented in
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
- Click verification: after each click the parsed detail-panel signature should change. An unchanged signature is a soft miss (it can also mean two OCR-identical neighbor pieces, common among +0 artifacts), so it is retried once with a small offset, logged with the stuck artifact name, and then skipped - never fatal on its own. The scan aborts only when the first ~6 clicks of page 1 produce nothing new (diagnosis hint: elevated Genshin blocks SendInput via UIPI, or grid coordinates are wrong) or a later page yields zero new artifacts.
- Scan stats separate clicked (click attempts), parsed (readable captures), stored (persisted), review (review samples), duplicates, and misses, so "scanned" cannot be mistaken for "successfully read".
- Scrolling sends one wheel notch per grid row with the cursor anchored over the inventory (assumption: roughly one row per notch; overlap is absorbed by dedupe, and a page without new artifacts stops the scan).