bao/genshin-assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare scanner branch for merge

Browse Source
This commit is contained in:
AzuTear
2026-07-09 08:44:50 +02:00
parent f791d1464c
commit 8b73c01e46
69 changed files with 6700 additions and 3773 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/AUTOMATION_LIVE_SCAN.md
+203 -44
View File
@@ -5,8 +5,8 @@ movement, click input, elevation, and live validation status.
## Current Known-Good State
Validated live on 2026-07-07 with Genshin open in the artifact inventory at
1920x1080, English UI:
Validated live with Genshin open in the artifact inventory at 1920x1080,
English UI:
- `npm run dev:admin` starts the app elevated after the user confirms UAC.
- Runtime status reported `isElevated: true`, `genshinFound: true`, and
@@ -19,15 +19,24 @@ Validated live on 2026-07-07 with Genshin open in the artifact inventory at
- A bounded live auto-scan via `/scanner/start?limit=2` completed with:
`clicked: 2`, `attempted: 2`, `verified: 2`, `parsed: 2`, `stored: 2`,
`review: 2`, `misses: 0`, `status: "done"`.
- On 2026-07-08, a visible-inventory 50-artifact run completed with `50/50`
parsed and stored, `0` review, `0` duplicates, and `0` misses. Throughput
was still slow at `61765 ms` elapsed (`1235 ms/artifact`).
- On 2026-07-09, the current-engine visible-inventory path completed
`/scanner/start?entry=visible-inventory&limit=20&engine=current` with
`20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and
`0` misses in `8047 ms` elapsed (`402 ms/artifact`). A same-session artifact
detail capture also persisted an equipped footer as `equipped: "Citlali"` and
an unlocked grey lock as `locked: false`.
This proves that the current elevated app plus helper path can deliver mouse
movement and click input to the focused Genshin client in this environment.
Latest-source timing is not proven while `/health.appBuild.signature` differs
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`. On 2026-07-07 the port
was still owned by an older elevated runtime, so goal scans were intentionally
blocked by the stale-build gate. Restart the elevated app through
`npm run dev:admin` and confirm UAC before collecting new 100-artifact evidence.
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`, or after source changes
that have not been loaded by a fresh elevated runtime. Restart the elevated app
through `npm run dev:admin` and confirm UAC before collecting new 50/100
artifact evidence.
## Elevation And UAC
@@ -101,23 +110,24 @@ Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?row=0&col=3"
For live validation, prefer a bounded scan first:
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"
```
The visible-inventory path remains the safest first check. The normal guided
entry tries the read-only direct world path first:
`B -> artifact tab -> first artifact tile`. If that does not produce a visible
artifact detail card, it falls back to the Inventory Kamera-compatible sequence:
`ESC -> B -> artifact tab -> first artifact tile`.
The visible-inventory path is the merge-relevant safe path. It requires the
Artifact inventory to already be open with a visible artifact detail card.
The normal Auto-Scan button uses a guided start. It first takes one lightweight
preflight capture without OCR, full-frame payload, review scoring, or storing.
If an artifact detail card is already visible, it starts the visible-inventory
scan. Otherwise it runs the guided entry above. OCR/review/store work starts
only after the artifact-detail preflight passes.
Guided entry uses short state polling for the Inventory screen, artifact grid,
and first detail card instead of waiting the full fixed delay every time; if the
state never appears, the same timeout budget returns the last diagnostic capture.
scan. Otherwise it blocks with an operator-facing status and asks the user to
open the Artifact inventory with a visible detail card. OCR/review/store work
starts only after the artifact-detail preflight passes.
The explicit Dev-Control entry modes below remain available for targeted
experiments only. They send read-only navigation, but they are not the
merge-ready default because live testing showed that `auto-entry` can leave the
app in the Paimon menu when the starting state is not what the choreography
expects.
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
@@ -147,13 +157,11 @@ The same guard also runs inside the scan loop. If the app is on the main game
screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
without an artifact detail card, auto-scan must block instead of clicking tiles
or trying OCR.
After each click the loop polls the detail fingerprint with a short bounded
budget instead of sleeping blindly. The current budget is 420 ms with 60 ms
polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it
does not change the loop retries or stops through the normal miss guards. If the
card changed but remains animated, the loop now proceeds after 200 ms, matching
Inventory Kamera's select-next-item wait more closely without removing the
detail-change guard.
After each click the loop now performs one fast artifact capture/OCR pass and
uses that capture's detail fingerprint to verify that the selected artifact
changed. This removes the old separate card-ready capture from the hot path. If
the detail fingerprint is unchanged, the loop retries once and then follows the
normal miss/block guards.
The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
not re-run the focus helper before every tile, which avoids an OS focus ping on
each artifact while still relying on click readback, foreground checks, and the
@@ -186,15 +194,87 @@ window/app and restart with `npm run dev:admin` before running scanner probes.
Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
`activeAverageMsPerParsed`, `averageCaptureMs`, `averageOcrMs`,
`artifactsPerMinute`, and `projectedMsFor100`. `elapsedMs` is end-to-end
including queued writes; `activeScanMs` is the click/capture/OCR loop before
the final store/review flush. A run only counts as speed
`activeAverageMsPerParsed`, `averageCaptureMs`,
`averageCaptureRoundTripMs`, `averageCaptureRoundTripOverheadMs`,
`averageOcrMs`, `artifactsPerMinute`, and `projectedMsFor100`.
`elapsedMs` is end-to-end including queued writes; `activeScanMs` is the
click/capture/OCR loop before the final store/review flush. A run only counts as speed
evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
together; raw click count alone is not scanner throughput. If `averageOcrMs`
dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
queue. If `averageCaptureMs` dominates, crop payload/capture work is the
bottleneck.
queue. If `averageCaptureRoundTripOverheadMs` is high, native capture encode,
Base64 transport, Electron image decode, or IPC/render scheduling is the next
bottleneck. The current 3 artifacts/second target requires `averageMsPerParsed`
at or below `333 ms` on a clean 20-artifact iteration.
Latest live timing evidence on 2026-07-08:
- Probe: `/automation/probe-click?index=1` returned `clicked: true`,
`inputBlocked: false`, `changed: true`, and `captureTarget:
"genshin-client"`.
- Baseline after helper/hot-loop cleanup:
`/scanner/start?entry=visible-inventory&limit=50&engine=current` completed
`50/50` parsed and stored with `0` review, `0` duplicates, `0` misses,
`2` pages, `elapsedMs: 61765`, `averageMsPerParsed: 1235`,
`averageCaptureMs: 186`, `averageOcrMs: 162`, and
`averageScrollReadyMs: 844`.
- Deferred-write experiment:
the same 50-artifact run completed `50/50` with `0` misses but regressed to
`elapsedMs: 63616` because 50 single-record writes produced
`writeFlushMs: 8163`.
- Current source replaces that experiment with batch persist and quiet
auto-scan UI captures. This is code-validated, but the batch version still
needs a fresh elevated live run; the follow-up restart was blocked because the
admin runtime did not become reachable after shutdown/UAC.
- Direct GDI hot-path validation:
after skipping `desktopCapturer.getSources()` in auto-scan artifact captures,
the 20-artifact iteration baseline improved to `20/20` parsed, `19` stored,
`0` review, `1` duplicate, `0` misses, `7966 ms` elapsed,
`398 ms/artifact`, `averageCaptureMs: 193`, `averageOcrMs: 167`,
`averageClickMs: 2`, and `writeFlushMs: 4`. This is roughly
`2.5 artifacts/second` on the first visible page.
- Scroll-path validation with the same direct GDI hot path:
`/scanner/start?entry=visible-inventory&limit=45&engine=current` completed
`45/45` parsed, `42` stored, `0` review, `3` duplicates, `0` misses,
`2` pages, `18625 ms` elapsed, `414 ms/artifact`, `averageCaptureMs: 187`,
`averageOcrMs: 162`, and one scroll readiness wait of `173 ms`.
- 100-artifact direct-GDI validation:
`/scanner/start?entry=visible-inventory&limit=100&engine=current` completed
on runtime signature `2026-07-08-direct-gdi-hotpath` with `100/100` parsed,
`97` stored, `0` review, `3` duplicates, `0` misses, `4` pages,
`42064 ms` elapsed, `421 ms/artifact`, `averageCaptureMs: 179`,
`averageOcrMs: 154`, `averageClickMs: 2`, `writeFlushMs: 6`, and `3`
scroll readiness waits averaging `176 ms`.
- OCR/parser eval after this speed pass: `npm run eval` passed with `23/23`
exact-match cases, `100%` field accuracy, and `100%` critical fields. This is
a regression gate, not a substitute for manually checking live artifact values.
- 3 artifacts/second preparation:
auto-scan artifact captures now also omit the detail-preview payload and
expose `averageCaptureRoundTripMs` plus
`averageCaptureRoundTripOverheadMs`. The first live run exposed a false
`missing-crops-or-ocr` review trigger because the hot path intentionally omits
`detailDataUrl`; this is fixed in `getAutoReviewReason`.
- 3 artifacts/second live attempts:
after the review fix, a clean `limit=20` run completed `20/20` parsed,
`19` stored, `0` review, `1` duplicate, `0` misses, `7285 ms` elapsed,
or `364 ms/artifact` (`2.75 artifacts/second`). The stable final run on
signature `2026-07-08-direct-gdi-reviewfix` completed `20/20`, `18` stored,
`0` review, `2` duplicates, `0` misses, `7973 ms` elapsed, or
`399 ms/artifact`. 3 artifacts/second is not proven.
- Rejected speed experiments:
detail-region capture, `GAA_OCR_WORKERS=5`, DataURL-to-buffer decode, and
substat OCR `PSM.SINGLE_COLUMN` were all live/benchmark tested and were slower
than the direct-GDI baseline. Keep `GAA_OCR_WORKERS=4` for current runs.
- Quality-gated current-vs-IK comparison:
`npm run scan:goal:compare:validated` produced
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`
with `createdAt: 2026-07-08T18:41:11.6120957+02:00`.
The final validator summary passed at `limit=100` with winner `current`,
`activeAvg: 378 ms/artifact`, `projected100: 37800 ms`, `missRate: 0`, and
`reviewRate: 0`. The `current` 100-artifact run parsed `100/100`, stored `97`,
had `0` review, `0` misses, and crossed `4` pages. The `ik-traineddata`
100-artifact run parsed `97/100`, had `5` review and `3` misses, and was not
qualified because it parsed fewer artifacts than requested.
The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
temporary scan limit. It does not change the normal UI setting. The normal
@@ -222,9 +302,9 @@ pool size. It also returns per-field OCR timings under
parser behavior. Individual captures also report whether the artifact was
detected as `sanctified`; level/substat crops are shifted in that state to match
Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
which omits the low-value set-effect crop, the slot crop that can be derived
from the matched artifact piece name, and the main-stat-value crop that can be
derived from slot, main-stat label, and level. The fast profile also uses
which omits the low-value set-effect crop and the main-stat-value crop that can
be derived from slot, main-stat label, and level. The slot crop remains enabled
in the fast profile because it improved live-read quality. The fast profile also uses
Inventory Kamera's tighter substat crop height; full/manual captures keep the
larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
captures while keeping the detail screenshot, OCR text, crop rect metadata, and
@@ -239,10 +319,13 @@ detection unless a caller explicitly overrides that option; add
artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
skipped and the response shows `skippedOcrCaptures` instead of burning time on
invalid crops.
For speed, the fast auto-scan profile also skips the optional Equipped footer
OCR. Name, level, main-stat label, and substats remain in the OCR hot path;
The fast auto-scan profile now keeps the optional Equipped footer OCR on real
artifact-read captures when the footer marker is visible, so stored artifacts
can record the equipped character without requiring a separate manual capture.
Name, level, main-stat label, footer, and substats remain in the OCR hot path;
slot, set, and main-stat value are derived when the lookup/parser can validate
them. Use a full/manual capture when equipped ownership or every debug crop matters.
them. Preflight and readiness poll captures still skip OCR/crops/lock-state
work because they only prove surface and fingerprint changes.
Local store/review writes are serialized through an internal queue but no longer
block the next inventory click. The scan still flushes the queue before it
returns its final summary, so `stored` and `review` counts remain final-state
@@ -314,23 +397,45 @@ Default sequence:
3. `/capture/smart?skipOcr=1`
4. `/automation/probe-click?index=1`
5. `/automation/probe-click?index=3`
6. `/scanner/start?limit=2`
7. `/scanner/start?limit=5`
8. `/scanner/start?limit=10`
9. `/scanner/start?limit=20`
6. `/scanner/start?entry=visible-inventory&limit=2`
7. `/scanner/start?entry=visible-inventory&limit=5`
8. `/scanner/start?entry=visible-inventory&limit=10`
9. `/scanner/start?entry=visible-inventory&limit=20`
10. `/review/samples?limit=30`
For the actual Inventory-Kamera speed target, use the explicit goal run after
`/health` shows the current `appBuild`:
```powershell
npm run scan:live:preflight
npm run scan:live:preflight:wait
npm run scan:goal
npm run scan:goal:current
npm run scan:goal:ik
npm run scan:iterate:compare:validated
npm run scan:iterate:compare:validated:wait
npm run scan:goal:compare
npm run scan:goal:compare:validated
npm run scan:goal:compare:validated:wait
```
That run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
`scan:live:preflight` checks `/health`, `/scanner/status`, the current
`APP_RUNTIME_SIGNATURE`, elevation, and whether Genshin is visible to the helper
before a long live scan is attempted.
Use `npm run scan:live:preflight:wait` during manual startup after `npm run
dev:admin`; it waits up to 120 seconds for the elevated dev-control server and
runtime checks to become ready. The non-waiting command remains the default for
validated scan chains so automation fails fast on a missing runtime.
Use `npm run scan:iterate:compare:validated` for fast iteration while tuning OCR,
parser, capture, or readiness behavior. It runs the same preflight, compares
`current` vs. `ik-traineddata` at `limit=20`, and validates the newest assessment
with `--limit=20 --summary`. This is the preferred loop while debugging because
it gives quality-gated feedback without waiting for the full `2, 5, 20, 45, 100`
goal sequence. Use `npm run scan:iterate:compare:validated:wait` directly after
UAC if the elevated runtime may still be starting.
The goal run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
uses the default `current` scan engine; use `scan:goal:ik` for a native
@@ -345,7 +450,32 @@ limit, picks the best qualified engine, and labels the dominant bottleneck as
OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
run, parse the requested count, keep miss rate under 2%, and keep review rate
at or below 15%; review and miss rates are penalized before active average speed
is used as the tie-breaker.
is used as the tie-breaker. For IK-target claims, check `goal100Decision`; it
must read `qualified-comparison: winner=<engine>`, and
`goal100.comparisonComplete` must be `true` so a single-engine 100-artifact run
is not mistaken for a current-vs-IK comparison.
Validate the saved assessment before using it as final evidence:
```powershell
npm run scan:assessment:validate -- --latest
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json
```
`--latest` searches `outputs/live-soak/` for the newest
`scan-performance-assessment.json`. Use explicit `--input` when comparing older
or archived runs. Add `--expect-winner=current` or
`--expect-winner=ik-traineddata` when validating a specific engine claim instead
of accepting any qualified winner. Add `--limit=20` for a short iteration run
instead of the final 100-artifact proof. Add `--summary` when you want a short
report-ready PASS/FAIL output that includes the input assessment path and
assessment `createdAt` timestamp.
`npm run scan:goal:compare:validated` is the preferred final command: it runs
the live preflight first, then the full comparison, and then validates the
newest assessment with `--summary`. Use
`npm run scan:goal:compare:validated:wait` for the same final flow when starting
immediately after UAC.
The assessment ranking can be verified without Genshin or the Electron app:
@@ -356,6 +486,34 @@ npm run scan:assessment:test
This self-test rejects synthetic runs that are fast but have too many misses or
too many review samples, so the final IK comparison cannot be won by speed alone.
## Review-To-Eval Quality Loop
After any live scan that creates review samples, export candidates before adding
anything to the permanent eval corpus:
```powershell
npm run eval:review-candidates -- --limit=80
```
Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
worklist, not ground truth. Only after the expected fields are confirmed or
corrected against the real artifact should a case be moved into
`src/eval/corpus/confirmedReviewCorpus.ts`. This prevents the parser from
grading itself and keeps `npm run eval` meaningful. The exporter deduplicates
samples, puts complete modern OCR captures first, and marks missing fast-profile
fields so stale or partial captures are easier to ignore. Unconfirmed exporter
output must stay in `outputs/review-eval-candidates/`.
For a manually checked candidate, generate a paste-ready confirmed-case snippet:
```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
```
The command requires explicit labels and writes only to the ignored outputs
folder. Review the snippet before adding it to
`src/eval/corpus/confirmedReviewCorpus.ts`.
For the current implementation summary and IK comparison rationale, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
@@ -429,7 +587,8 @@ Before marking an automation change done:
3. Run `npm test`.
4. Run `npm run build`.
5. If Genshin is available, run `/automation/probe-click?index=1`.
6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
6. For scan-loop changes, run
`/scanner/start?entry=visible-inventory&limit=2` before any broader scan.
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
a non-stale `npm run scan:goal:compare` run.
a non-stale `npm run scan:goal:compare:validated` run.