feat(scanner): validate elevated live automation
This commit is contained in:
AzuTear
+17
-2
@@ -49,7 +49,7 @@ flowchart LR
|
||||
|
||||
| Module | Responsibility |
|
||||
| --- | --- |
|
||||
| `electron/main.ts` | Window lifecycle, capture source listing, Smart Capture, OCR crop generation, overlay window IPC, persistent PowerShell input/capture helper, JSON artifact store |
|
||||
| `electron/main.ts` | Window lifecycle, capture source listing, Smart Capture, OCR crop generation, overlay window IPC, input/capture sidecar orchestration, JSON artifact store, dev-only scanner control endpoints |
|
||||
| `electron/preload.cjs` | Safe renderer bridge exposed as `window.assistantApi` |
|
||||
| `src/lib/artifactStore.ts` | Pure signature/id/record helpers for the persistent artifact store |
|
||||
| `src/App.tsx` | Main app shell, scan view, triage view, build view, overlay preview |
|
||||
@@ -111,9 +111,24 @@ sequenceDiagram
|
||||
|
||||
**Automatic grid scan** is user-triggered input automation limited to clicking detected inventory tiles and wheel-scrolling the inventory. Safety and reliability rules:
|
||||
|
||||
- All input goes through one persistent PowerShell helper process (`input-helper.ps1` in userData) that compiles the Win32 interop once and speaks JSON over stdin/stdout (ops: ping, focus, cursor, click, scroll, capture). Mouse movement is sent as iterated relative SendInput deltas (what a real mouse produces): Genshin tracks the cursor via raw input and snaps the OS cursor back to its own position every frame, so SetCursorPos/absolute moves silently stop working once the game owns the cursor. The helper verifies the cursor reached the target and refuses to click otherwise.
|
||||
- All input goes through the helper service boundary (currently a C# sidecar with
|
||||
fallback support behind the same JSON protocol). The helper owns focus, cursor
|
||||
movement, click, scroll, guard-state polling, elevation detection, and GDI
|
||||
capture. Mouse movement is sent as iterated relative input deltas instead of
|
||||
relying on a single absolute cursor jump. The helper verifies the cursor
|
||||
reached the target and refuses to click otherwise.
|
||||
- `npm run dev:admin` is the validated dev path for automation when elevated
|
||||
input is required. The elevated PowerShell startup is handled by
|
||||
`scripts/dev-admin.ps1` and logged to `outputs/admin-start/admin-dev.log`.
|
||||
The user must approve UAC manually; the app cannot approve the Secure Desktop
|
||||
prompt itself.
|
||||
- Failsafe: before every click and scroll the renderer polls cursor position and ESC state. Holding ESC or moving the mouse away from the last automated position aborts the scan immediately; the Stop button also aborts. Only the `GetAsyncKeyState` held-down bit (0x8000) is used - the "pressed since last call" bit fires for stale ESC presses from normal Genshin menu navigation and caused false aborts.
|
||||
- SendInput's return value is checked: zero injected events (UIPI, e.g. elevated Genshin vs. non-elevated app) aborts with an explicit hint instead of silently clicking into nothing.
|
||||
- Dev-only probes under `http://127.0.0.1:17317` are used for live validation:
|
||||
`/automation/probe-click?index=N` tests one read-only tile selection, and
|
||||
`/scanner/start?limit=N` starts an auto-scan with a temporary limit payload.
|
||||
The live known-good result on 2026-07-07 is documented in
|
||||
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
- Click verification: after each click the parsed detail-panel signature should change. An unchanged signature is a soft miss (it can also mean two OCR-identical neighbor pieces, common among +0 artifacts), so it is retried once with a small offset, logged with the stuck artifact name, and then skipped - never fatal on its own. The scan aborts only when the first ~6 clicks of page 1 produce nothing new (diagnosis hint: elevated Genshin blocks SendInput via UIPI, or grid coordinates are wrong) or a later page yields zero new artifacts.
|
||||
- Scan stats separate clicked (click attempts), parsed (readable captures), stored (persisted), review (review samples), duplicates, and misses, so "scanned" cannot be mistaken for "successfully read".
|
||||
- Scrolling sends one wheel notch per grid row with the cursor anchored over the inventory (assumption: roughly one row per notch; overlap is absorbed by dedupe, and a page without new artifacts stops the scan).
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
# Automation Live Scan Runbook
|
||||
|
||||
This document is the durable reference for automatic artifact scanning, mouse
|
||||
movement, click input, elevation, and live validation status.
|
||||
|
||||
## Current Known-Good State
|
||||
|
||||
Validated live on 2026-07-07 with Genshin open in the artifact inventory at
|
||||
1920x1080, English UI:
|
||||
|
||||
- `npm run dev:admin` starts the app elevated after the user confirms UAC.
|
||||
- Runtime status reported `isElevated: true`, `genshinFound: true`, and
|
||||
`targetProcess: "GenshinImpact"`.
|
||||
- The safe probe endpoint `/automation/probe-click?index=1` focused Genshin,
|
||||
moved the cursor to the second visible inventory tile, clicked it, and changed
|
||||
the artifact detail panel fingerprint.
|
||||
- Probe result: `clicked: true`, `inputBlocked: false`,
|
||||
`foregroundProcess: "GenshinImpact"`, and `changed: true`.
|
||||
- A bounded live auto-scan via `/scanner/start?limit=2` completed with:
|
||||
`clicked: 2`, `attempted: 2`, `verified: 2`, `parsed: 2`, `stored: 2`,
|
||||
`review: 2`, `misses: 0`, `status: "done"`.
|
||||
|
||||
This proves that the current elevated app plus helper path can deliver mouse
|
||||
movement and click input to the focused Genshin client in this environment.
|
||||
|
||||
## Elevation And UAC
|
||||
|
||||
Use:
|
||||
|
||||
```powershell
|
||||
npm run dev:admin
|
||||
```
|
||||
|
||||
The command runs `scripts/dev-admin.ps1`, which launches a new elevated
|
||||
PowerShell window running `scripts/dev-admin-start.ps1`. The elevated start is
|
||||
logged to:
|
||||
|
||||
```text
|
||||
outputs/admin-start/admin-dev.log
|
||||
```
|
||||
|
||||
The user must confirm the Windows UAC prompt. The app cannot and must not click
|
||||
the Secure Desktop UAC prompt for itself. After confirmation, the app can verify
|
||||
its own runtime through the dev status endpoint.
|
||||
|
||||
Useful checks:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod http://127.0.0.1:17317/health
|
||||
Invoke-RestMethod http://127.0.0.1:17317/scanner/status
|
||||
```
|
||||
|
||||
Expected runtime facts before automatic scan:
|
||||
|
||||
- `isElevated: true`
|
||||
- `genshinFound: true`
|
||||
- `targetProcess: "GenshinImpact"`
|
||||
- hotkeys registered
|
||||
|
||||
## Mouse And Click Validation
|
||||
|
||||
Use the probe before broad auto-scan work:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1" |
|
||||
ConvertTo-Json -Depth 12
|
||||
```
|
||||
|
||||
The probe performs one read-only inventory selection click. It does not delete,
|
||||
feed, enhance, lock, unlock, spend, or modify game resources.
|
||||
|
||||
Interpretation:
|
||||
|
||||
- `click.ok: true`, `clicked: true`, `inputBlocked: false` means Windows did not
|
||||
block SendInput/UIPI in the current configuration.
|
||||
- `focused: true` and `foregroundProcess: "GenshinImpact"` means the click was
|
||||
sent while Genshin was foreground.
|
||||
- `changed: true` means the detail panel changed after the click.
|
||||
- `changed: false` can be benign if the target tile was already selected or two
|
||||
neighboring artifacts render identically; retry with another `index`, `row`,
|
||||
or `col`.
|
||||
|
||||
Examples:
|
||||
|
||||
```powershell
|
||||
# Second visible tile
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?index=1"
|
||||
|
||||
# Specific grid cell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?row=0&col=3"
|
||||
```
|
||||
|
||||
## Bounded Live Auto-Scan
|
||||
|
||||
For live validation, prefer a bounded scan first:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
|
||||
```
|
||||
|
||||
Then poll:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
|
||||
ConvertTo-Json -Depth 12
|
||||
```
|
||||
|
||||
The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
|
||||
temporary scan limit. It does not change the normal UI setting. The normal
|
||||
hotkeys and buttons still use the UI's configured scan limit.
|
||||
|
||||
## Anti-Cheat And Safety Boundary
|
||||
|
||||
Do not describe the current implementation as bypassing anti-cheat. The app
|
||||
does not read memory, hook the process, inject code, modify game files, inspect
|
||||
packets, or interact with kernel drivers. It uses normal Windows screen capture,
|
||||
focus, cursor movement, wheel, and click input.
|
||||
|
||||
The practical finding is narrower:
|
||||
|
||||
- A non-elevated app can be blocked by Windows integrity/UIPI when the target
|
||||
process is elevated or protected.
|
||||
- Running the app elevated fixed input delivery in the tested environment.
|
||||
- Genshin's anti-cheat may still affect behavior on other machines, game modes,
|
||||
overlays, or future versions. Re-run the probe before trusting broad scans.
|
||||
|
||||
Never add automation that deletes, feeds, enhances, locks/unlocks, spends
|
||||
resources, reads memory, hooks, injects, or modifies Genshin.
|
||||
|
||||
## Live Layout Facts
|
||||
|
||||
The current 16:9 layout profile is calibrated from a 1920x1080 English
|
||||
artifact-inventory capture:
|
||||
|
||||
- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
|
||||
- inventory grid: `8 x 5`
|
||||
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
|
||||
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`
|
||||
- inventory count crop successfully read `2059/2400` in the live session
|
||||
|
||||
The profile is resolution-scaled for 16:9. Off-profile setups should be treated
|
||||
as higher risk and validated with Smart Capture plus the probe.
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
Before marking an automation change done:
|
||||
|
||||
1. Run `npm run lint`.
|
||||
2. Run `npx tsc -p tsconfig.electron.json` when Electron/preload/main changed.
|
||||
3. Run `npm test`.
|
||||
4. Run `npm run build`.
|
||||
5. If Genshin is available, run `/automation/probe-click?index=1`.
|
||||
6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
|
||||
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
|
||||
@@ -15,6 +15,7 @@ This document contains Architecture Decision Records.
|
||||
| ADR-007 | Measure OCR accuracy with a labeled eval harness before reworking the scanner | Accepted | 2026-07-05 |
|
||||
| ADR-008 | Replace the PowerShell input/capture helper with a C# sidecar | Accepted | 2026-07-05 |
|
||||
| ADR-009 | Resolution-anchored layout profiles and OCR preprocessing over color detection | Accepted | 2026-07-05 |
|
||||
| ADR-010 | Elevated dev runner and bounded live automation probes | Accepted | 2026-07-07 |
|
||||
|
||||
## ADR-001: Build A Local Electron App First
|
||||
|
||||
@@ -235,3 +236,53 @@ validated against the ADR-007 eval harness.
|
||||
insufficient.
|
||||
- Non-16:9 or non-borderless setups are explicitly unsupported for the auto
|
||||
scanner; the app should detect and warn rather than silently misread.
|
||||
|
||||
## ADR-010: Elevated Dev Runner And Bounded Live Automation Probes
|
||||
|
||||
### Status
|
||||
|
||||
Accepted
|
||||
|
||||
### Context
|
||||
|
||||
Automatic grid scanning needs read-only mouse movement, click, and wheel input
|
||||
to reach the focused Genshin window. A lower-integrity app can fail to deliver
|
||||
input to an elevated or protected target because of Windows UIPI/integrity
|
||||
boundaries. During live testing, `npm run dev:admin` originally printed that a
|
||||
new Administrator window was started, but the elevated PowerShell received no
|
||||
arguments, so the intended dev process did not reliably start.
|
||||
|
||||
The project also needed a smaller live validation path than a full inventory
|
||||
scan. A full scan is too risky as the first proof of input delivery because it
|
||||
can click many tiles before a bad coordinate, focus issue, or blocked input is
|
||||
understood.
|
||||
|
||||
### Decision
|
||||
|
||||
Keep automatic scan input automation read-only and require an elevated runtime
|
||||
when Windows reports that automation would otherwise be blocked. Replace the
|
||||
old `dev-admin.cmd` entry with `scripts/dev-admin.ps1`, quote the elevated
|
||||
PowerShell arguments explicitly, and log elevated startup to
|
||||
`outputs/admin-start/admin-dev.log`.
|
||||
|
||||
Add dev-only HTTP checks:
|
||||
|
||||
- `/automation/probe-click?index=N` or `?row=R&col=C` performs one safe
|
||||
inventory selection click and verifies whether the detail panel changed.
|
||||
- `/scanner/start?limit=N` sends a temporary scan-limit payload to the renderer,
|
||||
so live auto-scan validation can start with two items instead of the UI
|
||||
default.
|
||||
|
||||
Document the workflow in [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
|
||||
### Consequences
|
||||
|
||||
- The user still has to approve Windows UAC manually; the app must not try to
|
||||
click the Secure Desktop prompt.
|
||||
- We can distinguish input delivery from OCR/parser quality with a one-click
|
||||
probe before running any broader scan.
|
||||
- Live validation now has a low-risk path: check elevation and Genshin
|
||||
detection, run a single probe click, then run a bounded `limit=2` scan.
|
||||
- The implementation remains inside the allowed safety boundary: no memory
|
||||
reads, hooks, injection, game-file modification, deleting, feeding, enhancing,
|
||||
locking/unlocking, or spending resources.
|
||||
|
||||
+9
-5
@@ -72,7 +72,7 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
|
||||
| Styling | CSS with dark purple glassmorphism system | Premium fintech-inspired visual direction |
|
||||
| OCR | Tesseract.js prototype plus deterministic normalization/derivation | OCR alone is not trusted as the decision source |
|
||||
| Capture | Electron desktopCapturer plus Windows GDI Smart Capture | GDI path is used for Genshin Smart Capture reliability |
|
||||
| Input automation | PowerShell sidecar prototype now, native sidecar planned | Current sidecar is good for proving behavior, not the final production path |
|
||||
| Input automation | C# sidecar with elevated dev runner when needed | Live-validated for read-only inventory selection clicks; see `docs/AUTOMATION_LIVE_SCAN.md` |
|
||||
| Tests | Vitest + TypeScript checks | Current validation baseline; regression samples must expand |
|
||||
| Packaging | electron-builder | Configured in `package.json` |
|
||||
|
||||
@@ -95,11 +95,15 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
|
||||
- The parser already uses known sets, pieces, slots, stat aliases, set aliases, character aliases, and derived slot/set mapping.
|
||||
- Review samples, learned replacements, parser notes, and stored artifacts already persist locally.
|
||||
- The auto-scan loop is no longer a naive click spammer: it has preflight, verification, miss handling, page fingerprinting, and stop conditions.
|
||||
- Elevated live automation is validated in the current dev environment:
|
||||
`/automation/probe-click?index=1` changed the selected artifact and
|
||||
`/scanner/start?limit=2` completed with 2/2 verified reads and 0 misses.
|
||||
|
||||
### What is still structurally weak
|
||||
|
||||
- The scan experience is still partly orchestrated from `src/App.tsx`, which makes behavior changes harder than they should be.
|
||||
- The current PowerShell input sidecar is serviceable for experimentation but not a strong production base for long-running, low-jitter auto-scan.
|
||||
- Broader scan soak testing still needs to increase the live limit gradually and
|
||||
validate scroll/page transitions beyond the first visible row.
|
||||
- OCR quality is still inconsistent enough that some fields are recovered by fallback and derivation more often than they should be.
|
||||
- Learned fixes currently focus on text replacements; they do not yet update crop offsets, UI profile variants, or scanner targeting rules in a structured way.
|
||||
- The scan page is cleaner than before, but it still exposes too much operator/debug state in the main flow.
|
||||
@@ -196,7 +200,7 @@ Outcome:
|
||||
- Auto-scan never starts on a session that cannot prove one successful detail-card change.
|
||||
|
||||
Status:
|
||||
- Planned
|
||||
- First live path validated; broader soak testing still needed
|
||||
|
||||
### Phase 5 - Learning loop that actually compounds
|
||||
|
||||
@@ -228,7 +232,7 @@ Status:
|
||||
1. Finish scan-page cleanup so the main operator view is no longer noisy.
|
||||
2. Tighten the game data generator and parser contract, then backfill regression tests from real bad samples.
|
||||
3. Continue moving auto-scan behavior out of `App.tsx` and into isolated scanner modules.
|
||||
4. Replace or wrap the current PowerShell sidecar with a more stable long-lived automation process.
|
||||
4. Soak-test the elevated C# helper automation path with gradually larger scan limits and page scroll transitions.
|
||||
5. Extend the learning system from text-only fixes into crop/UI profile tuning.
|
||||
6. Resume recommendation work only when scan accuracy is consistently trustworthy.
|
||||
|
||||
@@ -236,7 +240,7 @@ Status:
|
||||
|
||||
| Question | Status |
|
||||
| --- | --- |
|
||||
| Should the production input sidecar be Rust/C++ first, or a transitional Node native addon, for the next iteration? | Open |
|
||||
| Is the current C# helper sufficient for production packaging, or does a later Rust/C++ sidecar still materially reduce latency or packaging risk? | Open |
|
||||
| When should UI-profile learning be allowed to change crop geometry automatically versus requiring review approval? | Open |
|
||||
| What scan-quality threshold is high enough before recommendations should be considered user-facing again? | Open |
|
||||
| Which Genshin UI languages should be supported after English once the scanner contract is stable? | Open |
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
# Scanner rework status
|
||||
|
||||
Progress on the approved scanner/OCR rework. See ADR-007/008/009 in
|
||||
[DECISIONS.md](DECISIONS.md) for the decisions behind these.
|
||||
Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in
|
||||
[DECISIONS.md](DECISIONS.md) for the decisions behind these. For the current
|
||||
live automation runbook, see
|
||||
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
|
||||
## Done (implemented, unit-tested, build green)
|
||||
|
||||
@@ -12,34 +14,44 @@ Progress on the approved scanner/OCR rework. See ADR-007/008/009 in
|
||||
fallback. Verified end-to-end (spawn, runtime, base64 capture).
|
||||
- **Layout profiles + OCR preprocessing** — `src/lib/layoutProfile.ts` (pure
|
||||
geometry, 16:9 detection), `src/lib/ocrPreprocess.ts` (grayscale + Otsu
|
||||
binarize). main.ts crops via the profile and OCRs an upscaled + binarized copy.
|
||||
binarize). main.ts now uses calibrated 16:9 detail/count/grid coordinates
|
||||
first and OCRs an upscaled + binarized copy.
|
||||
- **Card-ready gating** — `src/lib/cardReadyGate.ts` replaces the fixed 280 ms
|
||||
settle with change+stability polling; robust to animation.
|
||||
- **GOOD interop** — `src/lib/goodInterop.ts` (export + best-effort import for
|
||||
scanned records).
|
||||
scanned records), Electron file-picker import/export, and store merge.
|
||||
- **Rescan-merge** — `src/lib/artifactMerge.ts` collapses leveled re-scan
|
||||
duplicates by a level-independent identity.
|
||||
- **Data staleness warning** — `src/lib/dataPackageStatus.ts`, surfaced in the
|
||||
Scanner Diagnose data-package line.
|
||||
- **Lock detection (experimental)** — `src/lib/lockDetection.ts`, pure heuristic,
|
||||
not yet wired into capture.
|
||||
- **Lock detection (experimental)** — `src/lib/lockDetection.ts`, wired into
|
||||
live capture as a read-only `locked` flag and persisted with scanned records.
|
||||
- **Elevated live automation path** — `npm run dev:admin` now starts through
|
||||
`scripts/dev-admin.ps1` and logs to `outputs/admin-start/admin-dev.log`.
|
||||
Live status confirmed `isElevated: true`, `genshinFound: true`, and
|
||||
`targetProcess: "GenshinImpact"`.
|
||||
- **Read-only click probe** — `/automation/probe-click?index=1` verified that
|
||||
the app can focus Genshin, move to a visible inventory tile, click it, and
|
||||
observe a changed detail panel fingerprint (`clicked: true`,
|
||||
`inputBlocked: false`, `changed: true`).
|
||||
- **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live
|
||||
with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records,
|
||||
2 review samples, and 0 misses.
|
||||
|
||||
## Remaining — needs the live environment or a UI pass
|
||||
|
||||
These cannot be finished/validated without Genshin running at the user's
|
||||
resolution or without UI work best tested live:
|
||||
|
||||
1. **Calibrate IK-style fixed crop coordinates** (ADR-009). The layout module is
|
||||
the structure; the exact per-field fractions still come from a
|
||||
colour-detected/fallback detail rect. A reference 16:9 screenshot of the
|
||||
artifact screen lets us pin exact client-relative crop coordinates.
|
||||
2. **Validate/tune OCR preprocessing** on real captures — confirm invert +
|
||||
1. **Validate/tune OCR preprocessing** on more real captures — confirm invert +
|
||||
threshold + upscale factor help (not hurt) actual Tesseract reads. The
|
||||
text-level eval harness cannot measure image preprocessing.
|
||||
3. **Wire GOOD import** — file-picker IPC + merge imported records into the store
|
||||
(the conversion engine is done and tested).
|
||||
4. **Wire live lock detection** — calibrate crop position/threshold against a
|
||||
reference screenshot, then populate a `locked` flag during capture.
|
||||
2. **Validate locked=true** against a known locked artifact — unlocked/grey lock
|
||||
was live-checked; a gold locked icon still needs a positive sample.
|
||||
|
||||
3. **Broader scan soak test** — after the bounded two-item live scan passed,
|
||||
the next automation validation should increase the limit gradually and watch
|
||||
for repeated pages, scroll behavior, duplicate handling, and OCR review rate.
|
||||
|
||||
## Grow the eval corpus
|
||||
|
||||
|
||||
Reference in New Issue
Block a user