bao/genshin-assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare scanner branch for merge

Browse Source
This commit is contained in:
AzuTear
2026-07-09 08:44:50 +02:00
parent f791d1464c
commit 8b73c01e46
69 changed files with 6700 additions and 3773 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -8,6 +8,7 @@ outputs/dist/
outputs/admin-start/
outputs/live-capture/
outputs/live-soak/
outputs/review-eval-candidates/
# Logs
*.log
docs/ARCHITECTURE.md
+31 -8
View File
@@ -49,10 +49,19 @@ flowchart LR
| Module | Responsibility |
| --- | --- |
| `electron/main.ts` | Window lifecycle, capture source listing, Smart Capture, OCR crop generation, overlay window IPC, input/capture sidecar orchestration, JSON artifact store, dev-only scanner control endpoints |
| `electron/main.ts` | Electron app composition, dependency wiring, app lifecycle, hotkeys, and IPC/dev-control registration |
| `electron/appWindowManager.ts` | Main window and overlay window lifecycle, menu-bar removal, dashboard focus behavior, and renderer window command delivery |
| `electron/services/inputHelper.ts` | Stable JSON protocol client for the compiled C# input/capture sidecar plus PowerShell fallback startup |
| `electron/services/inputHelperPowerShellFallback.ts` | PowerShell fallback script body for environments where the compiled helper is unavailable |
| `electron/services/goodFileService.ts` | Local GOOD export file writing and GOOD import file dialog/read handling |
| `electron/preload.cjs` | Safe renderer bridge exposed as `window.assistantApi` |
| `src/lib/artifactStore.ts` | Pure signature/id/record helpers for the persistent artifact store |
| `src/App.tsx` | Main app shell, scan view, triage view, build view, overlay preview |
| `src/App.tsx` | Thin React entry that renders the app page |
| `src/pages/AppPage.tsx` and `src/pages/app/*` | App page composition and high-level layout routing |
| `src/features/scan/hooks/useScanViewController.ts` | Scan feature state composition and view-controller assembly |
| `src/features/scan/hooks/scanViewScanActions.ts` | Manual scan and visible-grid scan orchestration |
| `src/features/scan/hooks/scanViewEntryActions.ts` | Guided auto-entry choreography for visible inventory, direct inventory, and IK-style fallback paths |
| `src/features/scan/hooks/useScanGoodInterop.ts` | Scan-page GOOD import/export actions against renderer repository ports |
| `src/lib/artifactOcrParser.ts` | Converts OCR output into a parsed artifact candidate with confidence and notes |
| `src/lib/fuzzyMatch.ts` | Generic fuzzy string matching for OCR text against known game data |
| `src/lib/genshinLookup.ts` | Pure lookup and validation API for generated Genshin data |
@@ -63,6 +72,9 @@ flowchart LR
| `src/data/genshinGameData.json` | Generated local dictionary of characters, artifact sets, slots, and stats |
| `scripts/generate-genshin-data.cjs` | Regenerates the local Genshin dictionary from `genshin-db` |
| `src/types/*` | Shared app, capture, and domain contracts |
| `src/styles/global.css` | Stylesheet entrypoint importing split style modules |
| `src/styles/base.css` | Shared application, layout, scanner workspace, modal, triage, build, and overlay styles |
| `src/styles/diagnostics.css` | Diagnose/dev-view specific styles |
## Dependency Rules
@@ -129,7 +141,8 @@ sequenceDiagram
- SendInput's return value is checked: zero injected events (UIPI, e.g. elevated Genshin vs. non-elevated app) aborts with an explicit hint instead of silently clicking into nothing.
- Dev-only probes under `http://127.0.0.1:17317` are used for live validation:
`/automation/probe-click?index=N` tests one read-only tile selection, and
`/scanner/start?limit=N` starts an auto-scan with a temporary limit payload.
`/scanner/start?entry=visible-inventory&limit=N` starts an auto-scan with a
temporary limit payload from an already visible artifact detail view.
The live known-good result on 2026-07-07 is documented in
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
- Click verification: after each click the parsed detail-panel signature should change. An unchanged signature is a soft miss (it can also mean two OCR-identical neighbor pieces, common among +0 artifacts), so it is retried once with a small offset, logged with the stuck artifact name, and then skipped - never fatal on its own. The scan aborts only when the first ~6 clicks of page 1 produce nothing new (diagnosis hint: elevated Genshin blocks SendInput via UIPI, or grid coordinates are wrong) or a later page yields zero new artifacts.
@@ -140,15 +153,25 @@ sequenceDiagram
- Guided auto-entry is state gated. The normal scan button first performs a
lightweight no-OCR preflight; OCR/store/review work starts only after the
artifact inventory grid and artifact detail card are visually confirmed.
- The entry path tries direct inventory (`B`) first and uses the IK-style
ESC/B fallback only when direct entry does not reach an artifact detail card.
- Card and page waits are fingerprint based. The scanner can proceed as soon as
the expected visual state changes and stabilizes, while still accepting IK-like
200 ms item and 100 ms scroll readiness points.
- The normal scan button does not navigate into inventory when that preflight
fails; it blocks and asks the operator to open the Artifact inventory with a
visible detail card. Explicit Dev-Control entry modes can still test direct
inventory or Paimon-menu choreography, but they are not the merge-ready
default path.
- Item verification uses the artifact OCR capture's own detail fingerprint, so
the loop no longer performs a separate card-ready capture before OCR. Page
waits remain fingerprint based and can proceed as soon as the inventory pane
changes and stabilizes, while still accepting IK-like 100 ms scroll readiness
points.
- Scrolling sends one wheel notch per grid row with the cursor anchored over the inventory (assumption: roughly one row per notch; overlap is absorbed by dedupe, and a page without new artifacts stops the scan).
- The scan never deletes, enhances, feeds, locks, or spends anything; it only selects tiles to read them.
Parsed artifacts from both modes are persisted into `artifact-store.json` keyed by a content signature that excludes the equipped character, so re-equipping updates a record instead of duplicating it. Leveling an artifact currently creates a new record (documented limitation until rescan-merge exists).
During auto-scan, artifact store writes can be batched and flushed after the
click/capture/OCR loop to avoid per-artifact save/reload churn in the hot path.
Auto-scan artifact captures also bypass Electron source-list enumeration and use
the GDI capture helper directly once the selected source/Genshin state has been
preflighted. Manual captures and source refresh still use `desktopCapturer`.
## Security And Safety
docs/AUTOMATION_LIVE_SCAN.md
+203 -44
View File
@@ -5,8 +5,8 @@ movement, click input, elevation, and live validation status.
## Current Known-Good State
Validated live on 2026-07-07 with Genshin open in the artifact inventory at
1920x1080, English UI:
Validated live with Genshin open in the artifact inventory at 1920x1080,
English UI:
- `npm run dev:admin` starts the app elevated after the user confirms UAC.
- Runtime status reported `isElevated: true`, `genshinFound: true`, and
@@ -19,15 +19,24 @@ Validated live on 2026-07-07 with Genshin open in the artifact inventory at
- A bounded live auto-scan via `/scanner/start?limit=2` completed with:
`clicked: 2`, `attempted: 2`, `verified: 2`, `parsed: 2`, `stored: 2`,
`review: 2`, `misses: 0`, `status: "done"`.
- On 2026-07-08, a visible-inventory 50-artifact run completed with `50/50`
parsed and stored, `0` review, `0` duplicates, and `0` misses. Throughput
was still slow at `61765 ms` elapsed (`1235 ms/artifact`).
- On 2026-07-09, the current-engine visible-inventory path completed
`/scanner/start?entry=visible-inventory&limit=20&engine=current` with
`20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and
`0` misses in `8047 ms` elapsed (`402 ms/artifact`). A same-session artifact
detail capture also persisted an equipped footer as `equipped: "Citlali"` and
an unlocked grey lock as `locked: false`.
This proves that the current elevated app plus helper path can deliver mouse
movement and click input to the focused Genshin client in this environment.
Latest-source timing is not proven while `/health.appBuild.signature` differs
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`. On 2026-07-07 the port
was still owned by an older elevated runtime, so goal scans were intentionally
blocked by the stale-build gate. Restart the elevated app through
`npm run dev:admin` and confirm UAC before collecting new 100-artifact evidence.
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`, or after source changes
that have not been loaded by a fresh elevated runtime. Restart the elevated app
through `npm run dev:admin` and confirm UAC before collecting new 50/100
artifact evidence.
## Elevation And UAC
@@ -101,23 +110,24 @@ Invoke-RestMethod "http://127.0.0.1:17317/automation/probe-click?row=0&col=3"
For live validation, prefer a bounded scan first:
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2"
```
The visible-inventory path remains the safest first check. The normal guided
entry tries the read-only direct world path first:
`B -> artifact tab -> first artifact tile`. If that does not produce a visible
artifact detail card, it falls back to the Inventory Kamera-compatible sequence:
`ESC -> B -> artifact tab -> first artifact tile`.
The visible-inventory path is the merge-relevant safe path. It requires the
Artifact inventory to already be open with a visible artifact detail card.
The normal Auto-Scan button uses a guided start. It first takes one lightweight
preflight capture without OCR, full-frame payload, review scoring, or storing.
If an artifact detail card is already visible, it starts the visible-inventory
scan. Otherwise it runs the guided entry above. OCR/review/store work starts
only after the artifact-detail preflight passes.
Guided entry uses short state polling for the Inventory screen, artifact grid,
and first detail card instead of waiting the full fixed delay every time; if the
state never appears, the same timeout budget returns the last diagnostic capture.
scan. Otherwise it blocks with an operator-facing status and asks the user to
open the Artifact inventory with a visible detail card. OCR/review/store work
starts only after the artifact-detail preflight passes.
The explicit Dev-Control entry modes below remain available for targeted
experiments only. They send read-only navigation, but they are not the
merge-ready default because live testing showed that `auto-entry` can leave the
app in the Paimon menu when the starting state is not what the choreography
expects.
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
@@ -147,13 +157,11 @@ The same guard also runs inside the scan loop. If the app is on the main game
screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
without an artifact detail card, auto-scan must block instead of clicking tiles
or trying OCR.
After each click the loop polls the detail fingerprint with a short bounded
budget instead of sleeping blindly. The current budget is 420 ms with 60 ms
polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it
does not change the loop retries or stops through the normal miss guards. If the
card changed but remains animated, the loop now proceeds after 200 ms, matching
Inventory Kamera's select-next-item wait more closely without removing the
detail-change guard.
After each click the loop now performs one fast artifact capture/OCR pass and
uses that capture's detail fingerprint to verify that the selected artifact
changed. This removes the old separate card-ready capture from the hot path. If
the detail fingerprint is unchanged, the loop retries once and then follows the
normal miss/block guards.
The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
not re-run the focus helper before every tile, which avoids an OS focus ping on
each artifact while still relying on click readback, foreground checks, and the
@@ -186,15 +194,87 @@ window/app and restart with `npm run dev:admin` before running scanner probes.
Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
`activeAverageMsPerParsed`, `averageCaptureMs`, `averageOcrMs`,
`artifactsPerMinute`, and `projectedMsFor100`. `elapsedMs` is end-to-end
including queued writes; `activeScanMs` is the click/capture/OCR loop before
the final store/review flush. A run only counts as speed
`activeAverageMsPerParsed`, `averageCaptureMs`,
`averageCaptureRoundTripMs`, `averageCaptureRoundTripOverheadMs`,
`averageOcrMs`, `artifactsPerMinute`, and `projectedMsFor100`.
`elapsedMs` is end-to-end including queued writes; `activeScanMs` is the
click/capture/OCR loop before the final store/review flush. A run only counts as speed
evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
together; raw click count alone is not scanner throughput. If `averageOcrMs`
dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
queue. If `averageCaptureMs` dominates, crop payload/capture work is the
bottleneck.
queue. If `averageCaptureRoundTripOverheadMs` is high, native capture encode,
Base64 transport, Electron image decode, or IPC/render scheduling is the next
bottleneck. The current 3 artifacts/second target requires `averageMsPerParsed`
at or below `333 ms` on a clean 20-artifact iteration.
Latest live timing evidence on 2026-07-08:
- Probe: `/automation/probe-click?index=1` returned `clicked: true`,
`inputBlocked: false`, `changed: true`, and `captureTarget:
"genshin-client"`.
- Baseline after helper/hot-loop cleanup:
`/scanner/start?entry=visible-inventory&limit=50&engine=current` completed
`50/50` parsed and stored with `0` review, `0` duplicates, `0` misses,
`2` pages, `elapsedMs: 61765`, `averageMsPerParsed: 1235`,
`averageCaptureMs: 186`, `averageOcrMs: 162`, and
`averageScrollReadyMs: 844`.
- Deferred-write experiment:
the same 50-artifact run completed `50/50` with `0` misses but regressed to
`elapsedMs: 63616` because 50 single-record writes produced
`writeFlushMs: 8163`.
- Current source replaces that experiment with batch persist and quiet
auto-scan UI captures. This is code-validated, but the batch version still
needs a fresh elevated live run; the follow-up restart was blocked because the
admin runtime did not become reachable after shutdown/UAC.
- Direct GDI hot-path validation:
after skipping `desktopCapturer.getSources()` in auto-scan artifact captures,
the 20-artifact iteration baseline improved to `20/20` parsed, `19` stored,
`0` review, `1` duplicate, `0` misses, `7966 ms` elapsed,
`398 ms/artifact`, `averageCaptureMs: 193`, `averageOcrMs: 167`,
`averageClickMs: 2`, and `writeFlushMs: 4`. This is roughly
`2.5 artifacts/second` on the first visible page.
- Scroll-path validation with the same direct GDI hot path:
`/scanner/start?entry=visible-inventory&limit=45&engine=current` completed
`45/45` parsed, `42` stored, `0` review, `3` duplicates, `0` misses,
`2` pages, `18625 ms` elapsed, `414 ms/artifact`, `averageCaptureMs: 187`,
`averageOcrMs: 162`, and one scroll readiness wait of `173 ms`.
- 100-artifact direct-GDI validation:
`/scanner/start?entry=visible-inventory&limit=100&engine=current` completed
on runtime signature `2026-07-08-direct-gdi-hotpath` with `100/100` parsed,
`97` stored, `0` review, `3` duplicates, `0` misses, `4` pages,
`42064 ms` elapsed, `421 ms/artifact`, `averageCaptureMs: 179`,
`averageOcrMs: 154`, `averageClickMs: 2`, `writeFlushMs: 6`, and `3`
scroll readiness waits averaging `176 ms`.
- OCR/parser eval after this speed pass: `npm run eval` passed with `23/23`
exact-match cases, `100%` field accuracy, and `100%` critical fields. This is
a regression gate, not a substitute for manually checking live artifact values.
- 3 artifacts/second preparation:
auto-scan artifact captures now also omit the detail-preview payload and
expose `averageCaptureRoundTripMs` plus
`averageCaptureRoundTripOverheadMs`. The first live run exposed a false
`missing-crops-or-ocr` review trigger because the hot path intentionally omits
`detailDataUrl`; this is fixed in `getAutoReviewReason`.
- 3 artifacts/second live attempts:
after the review fix, a clean `limit=20` run completed `20/20` parsed,
`19` stored, `0` review, `1` duplicate, `0` misses, `7285 ms` elapsed,
or `364 ms/artifact` (`2.75 artifacts/second`). The stable final run on
signature `2026-07-08-direct-gdi-reviewfix` completed `20/20`, `18` stored,
`0` review, `2` duplicates, `0` misses, `7973 ms` elapsed, or
`399 ms/artifact`. 3 artifacts/second is not proven.
- Rejected speed experiments:
detail-region capture, `GAA_OCR_WORKERS=5`, DataURL-to-buffer decode, and
substat OCR `PSM.SINGLE_COLUMN` were all live/benchmark tested and were slower
than the direct-GDI baseline. Keep `GAA_OCR_WORKERS=4` for current runs.
- Quality-gated current-vs-IK comparison:
`npm run scan:goal:compare:validated` produced
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`
with `createdAt: 2026-07-08T18:41:11.6120957+02:00`.
The final validator summary passed at `limit=100` with winner `current`,
`activeAvg: 378 ms/artifact`, `projected100: 37800 ms`, `missRate: 0`, and
`reviewRate: 0`. The `current` 100-artifact run parsed `100/100`, stored `97`,
had `0` review, `0` misses, and crossed `4` pages. The `ik-traineddata`
100-artifact run parsed `97/100`, had `5` review and `3` misses, and was not
qualified because it parsed fewer artifacts than requested.
The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
temporary scan limit. It does not change the normal UI setting. The normal
@@ -222,9 +302,9 @@ pool size. It also returns per-field OCR timings under
parser behavior. Individual captures also report whether the artifact was
detected as `sanctified`; level/substat crops are shifted in that state to match
Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
which omits the low-value set-effect crop, the slot crop that can be derived
from the matched artifact piece name, and the main-stat-value crop that can be
derived from slot, main-stat label, and level. The fast profile also uses
which omits the low-value set-effect crop and the main-stat-value crop that can
be derived from slot, main-stat label, and level. The slot crop remains enabled
in the fast profile because it improved live-read quality. The fast profile also uses
Inventory Kamera's tighter substat crop height; full/manual captures keep the
larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
captures while keeping the detail screenshot, OCR text, crop rect metadata, and
@@ -239,10 +319,13 @@ detection unless a caller explicitly overrides that option; add
artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
skipped and the response shows `skippedOcrCaptures` instead of burning time on
invalid crops.
For speed, the fast auto-scan profile also skips the optional Equipped footer
OCR. Name, level, main-stat label, and substats remain in the OCR hot path;
The fast auto-scan profile now keeps the optional Equipped footer OCR on real
artifact-read captures when the footer marker is visible, so stored artifacts
can record the equipped character without requiring a separate manual capture.
Name, level, main-stat label, footer, and substats remain in the OCR hot path;
slot, set, and main-stat value are derived when the lookup/parser can validate
them. Use a full/manual capture when equipped ownership or every debug crop matters.
them. Preflight and readiness poll captures still skip OCR/crops/lock-state
work because they only prove surface and fingerprint changes.
Local store/review writes are serialized through an internal queue but no longer
block the next inventory click. The scan still flushes the queue before it
returns its final summary, so `stored` and `review` counts remain final-state
@@ -314,23 +397,45 @@ Default sequence:
3. `/capture/smart?skipOcr=1`
4. `/automation/probe-click?index=1`
5. `/automation/probe-click?index=3`
6. `/scanner/start?limit=2`
7. `/scanner/start?limit=5`
8. `/scanner/start?limit=10`
9. `/scanner/start?limit=20`
6. `/scanner/start?entry=visible-inventory&limit=2`
7. `/scanner/start?entry=visible-inventory&limit=5`
8. `/scanner/start?entry=visible-inventory&limit=10`
9. `/scanner/start?entry=visible-inventory&limit=20`
10. `/review/samples?limit=30`
For the actual Inventory-Kamera speed target, use the explicit goal run after
`/health` shows the current `appBuild`:
```powershell
npm run scan:live:preflight
npm run scan:live:preflight:wait
npm run scan:goal
npm run scan:goal:current
npm run scan:goal:ik
npm run scan:iterate:compare:validated
npm run scan:iterate:compare:validated:wait
npm run scan:goal:compare
npm run scan:goal:compare:validated
npm run scan:goal:compare:validated:wait
```
That run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
`scan:live:preflight` checks `/health`, `/scanner/status`, the current
`APP_RUNTIME_SIGNATURE`, elevation, and whether Genshin is visible to the helper
before a long live scan is attempted.
Use `npm run scan:live:preflight:wait` during manual startup after `npm run
dev:admin`; it waits up to 120 seconds for the elevated dev-control server and
runtime checks to become ready. The non-waiting command remains the default for
validated scan chains so automation fails fast on a missing runtime.
Use `npm run scan:iterate:compare:validated` for fast iteration while tuning OCR,
parser, capture, or readiness behavior. It runs the same preflight, compares
`current` vs. `ik-traineddata` at `limit=20`, and validates the newest assessment
with `--limit=20 --summary`. This is the preferred loop while debugging because
it gives quality-gated feedback without waiting for the full `2, 5, 20, 45, 100`
goal sequence. Use `npm run scan:iterate:compare:validated:wait` directly after
UAC if the elevated runtime may still be starting.
The goal run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
uses the default `current` scan engine; use `scan:goal:ik` for a native
@@ -345,7 +450,32 @@ limit, picks the best qualified engine, and labels the dominant bottleneck as
OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
run, parse the requested count, keep miss rate under 2%, and keep review rate
at or below 15%; review and miss rates are penalized before active average speed
is used as the tie-breaker.
is used as the tie-breaker. For IK-target claims, check `goal100Decision`; it
must read `qualified-comparison: winner=<engine>`, and
`goal100.comparisonComplete` must be `true` so a single-engine 100-artifact run
is not mistaken for a current-vs-IK comparison.
Validate the saved assessment before using it as final evidence:
```powershell
npm run scan:assessment:validate -- --latest
npm run scan:assessment:validate -- --input=<run-dir>\scan-performance-assessment.json
```
`--latest` searches `outputs/live-soak/` for the newest
`scan-performance-assessment.json`. Use explicit `--input` when comparing older
or archived runs. Add `--expect-winner=current` or
`--expect-winner=ik-traineddata` when validating a specific engine claim instead
of accepting any qualified winner. Add `--limit=20` for a short iteration run
instead of the final 100-artifact proof. Add `--summary` when you want a short
report-ready PASS/FAIL output that includes the input assessment path and
assessment `createdAt` timestamp.
`npm run scan:goal:compare:validated` is the preferred final command: it runs
the live preflight first, then the full comparison, and then validates the
newest assessment with `--summary`. Use
`npm run scan:goal:compare:validated:wait` for the same final flow when starting
immediately after UAC.
The assessment ranking can be verified without Genshin or the Electron app:
@@ -356,6 +486,34 @@ npm run scan:assessment:test
This self-test rejects synthetic runs that are fast but have too many misses or
too many review samples, so the final IK comparison cannot be won by speed alone.
## Review-To-Eval Quality Loop
After any live scan that creates review samples, export candidates before adding
anything to the permanent eval corpus:
```powershell
npm run eval:review-candidates -- --limit=80
```
Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
worklist, not ground truth. Only after the expected fields are confirmed or
corrected against the real artifact should a case be moved into
`src/eval/corpus/confirmedReviewCorpus.ts`. This prevents the parser from
grading itself and keeps `npm run eval` meaningful. The exporter deduplicates
samples, puts complete modern OCR captures first, and marks missing fast-profile
fields so stale or partial captures are easier to ignore. Unconfirmed exporter
output must stay in `outputs/review-eval-candidates/`.
For a manually checked candidate, generate a paste-ready confirmed-case snippet:
```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
```
The command requires explicit labels and writes only to the ignored outputs
folder. Review the snippet before adding it to
`src/eval/corpus/confirmedReviewCorpus.ts`.
For the current implementation summary and IK comparison rationale, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
@@ -429,7 +587,8 @@ Before marking an automation change done:
3. Run `npm test`.
4. Run `npm run build`.
5. If Genshin is available, run `/automation/probe-click?index=1`.
6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
6. For scan-loop changes, run
`/scanner/start?entry=visible-inventory&limit=2` before any broader scan.
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
a non-stale `npm run scan:goal:compare` run.
a non-stale `npm run scan:goal:compare:validated` run.
docs/CHECKLISTS.md
+9 -1
View File
@@ -16,9 +16,17 @@
## IK-Speed Or OCR-Engine Claim
- [ ] `/health.appBuild.signature` matches the current `APP_RUNTIME_SIGNATURE`.
- [ ] `npm run scan:live:preflight` passes, or use the validated comparison command that runs it first.
- [ ] During manual UAC startup, `npm run scan:live:preflight:wait` may be used before the scan chain.
- [ ] `npm run scan:assessment:test` passes.
- [ ] Use `npm run scan:iterate:compare:validated` or `npm run scan:iterate:compare:validated:wait` for short 20-artifact tuning loops.
- [ ] Prefer `npm run scan:goal:compare:validated` or `npm run scan:goal:compare:validated:wait` for the final live comparison because it runs preflight, comparison, and assessment validation in sequence.
- [ ] The run includes `scan-performance-assessment.json`.
- [ ] The 100-artifact run finishes cleanly.
- [ ] `npm run scan:assessment:validate -- --latest` or explicit `--input=<path>\scan-performance-assessment.json` passes.
- [ ] If claiming a specific winner, the validator is run with `--expect-winner=current` or `--expect-winner=ik-traineddata`.
- [ ] Final report cites the validator JSON or `--summary` output, including the assessment path.
- [ ] For iteration claims, the 20-artifact run finishes cleanly and is validated with `--limit=20`.
- [ ] For final claims, the 100-artifact run finishes cleanly.
- [ ] Parsed count is at least the requested count.
- [ ] Miss rate is at or below 2%.
- [ ] Review rate is at or below 15%.
docs/CONVENTIONS.md
+4
View File
@@ -11,10 +11,13 @@ This document defines project engineering standards.
## File Organization
- Keep Electron OS integration in `electron/`.
- Keep Electron `main.ts` as composition/wiring. Move durable window, file, helper, capture, OCR, or dev-control responsibilities into named modules under `electron/`.
- Keep React components in `src/`, with extraction when `App.tsx` becomes hard to review.
- Keep feature controller hooks small enough to review. If a hook owns persistence, import/export, entry choreography, scan-loop orchestration, and UI state at once, split those concerns into feature-local hooks or services.
- Keep pure domain logic in `src/lib/`.
- Keep shared contracts in `src/types/`.
- Keep generated outputs in `dist/`, `dist-electron/`, and `outputs/`.
- Keep `src/styles/global.css` as the stylesheet entrypoint. Put broad app styles in `base.css` and dev/diagnostic-only styling in `diagnostics.css` unless a more specific style module is introduced.
## UI Rules
@@ -30,6 +33,7 @@ This document defines project engineering standards.
- Confidence and raw OCR details must remain inspectable.
- Heuristics should fail safely into unknown fields or review notes.
- Do not add irreversible game actions.
- Keep auto-entry choreography separate from scan-loop execution. Entry code may navigate to a readable artifact detail state; loop code should process verified grid targets.
## TypeScript Rules
docs/DECISIONS.md
+7 -1
View File
@@ -323,7 +323,11 @@ engine or claiming IK parity. A qualified scan result must:
`scripts/live-soak.ps1` writes the evidence bundle and
`scan-performance-assessment.json`. `npm run scan:assessment:test` verifies that
the ranking logic rejects fast but low-quality synthetic runs without needing
Genshin.
Genshin. The assessment also records `goal100Decision` and
`goal100.comparisonComplete`; IK-target claims require a qualified 100-artifact
winner and a complete `current` vs. `ik-traineddata` comparison.
`npm run scan:assessment:validate -- --summary` prints the assessment path and
`createdAt` timestamp so reports can cite the exact evidence file.
### Consequences
@@ -332,4 +336,6 @@ Genshin.
closer to IK; it must win the same-capture benchmark and a qualified live run.
- Stale elevated Electron instances are treated as invalid evidence, not as a
harmless warning.
- The validated `:wait` scan scripts are acceptable for manual post-UAC startup;
non-waiting scripts remain useful when automation should fail fast.
- The goal remains open until the 100-artifact qualified comparison is captured.
docs/MERGE_READINESS.md
+58
View File
@@ -0,0 +1,58 @@
# Merge Readiness
Current branch: `codex/ik-scanner-progress`
This checklist records the evidence needed before merging this scanner branch
into `main`. It separates merge-relevant proof from optional follow-up work.
## Merge-Relevant Evidence
| Area | Status | Evidence |
| --- | --- | --- |
| TypeScript/lint gate | Passed | `npm run lint` |
| Unit/regression suite | Passed | `npm test` with 207 tests, including equipped footer, lock detection, and PNG lock-crop regression coverage |
| Production build | Passed | `npm run build` |
| Whitespace check | Passed | `git diff --check` |
| OCR eval gate | Passed | `npm run eval` with `23/23` exact-match cases and `100%` critical fields |
| Scan assessment self-test | Passed | `npm run scan:assessment:test`; the fixture intentionally expects `ik-traineddata` to win its synthetic `limit=100` case while real live-winner claims stay tied to archived live assessments |
| Live runtime preflight | Passed | `npm run scan:live:preflight`; signature `2026-07-08-direct-gdi-reviewfix`, elevated yes, Genshin found |
| Safe visible-inventory scan path | Passed | 2026-07-09 live run: `/scanner/start?entry=visible-inventory&limit=20&engine=current` completed `20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, `0` misses, `8047 ms` elapsed |
| Equipped character OCR/persist path | Passed for live smoke | 2026-07-09 live captures read equipped footers for `Citlali` and `Linnea`; parser regression covers `Equipped: Linnea l` -> `Linnea` |
| Unlocked lock state | Passed for live smoke | 2026-07-09 Smart Capture reported `locked: false` on an unlocked artifact; after restart, `lockSignal.ratio: 0` with threshold `0.06` |
| Positive locked lock state | Passed | 2026-07-09 Smart Capture on a visibly locked artifact reported `locked: true`, `lockSignal.ratio: 0.14797913950456323`, threshold `0.06` |
| Lock-state persistence | Passed | 2026-07-09 `/scanner/start?entry=visible-inventory&limit=1&engine=current` stored `A Note in Spring's Leich` with `equipped: "Citlali"` and `locked: true` |
| Lock-state diagnostics | Passed | Capture results include `lockSignal.ratio`, `lockSignal.threshold`, and the lock crop rect. Detection uses decoded PNG crop pixels to avoid native bitmap channel-order ambiguity |
| Unsafe auto-entry default | Mitigated | Normal guided Auto-Scan now blocks when no artifact detail card is visible instead of falling back to `auto-entry` |
| Review-to-eval export | Passed by tests | `npm run eval:review-candidates` script is covered by `src/eval/reviewEvalCandidatesScript.test.ts` and output is Git-ignored |
## Final Gate
```powershell
npm run lint
npm test
npm run build
git diff --check
```
Latest run after the lock-state fix: passed with `207` tests.
`npm run eval` and `npm run scan:assessment:test` also passed after the
lock-state fix.
## Explicitly Not Merge-Blocking
- `auto-entry`, `direct-inventory`, and `paimon-menu` are still experimental
Dev-Control entry modes. They can be tested later with low limits, but they
are no longer the normal merge path.
- The optional `3 artifacts/second` target is not proven. The stable current
path is closer to `2.5` to `2.75 artifacts/second` on clean 20-artifact live
runs.
- Native Tesseract/IK-traineddata is not the default. The latest documented
qualified live 100-artifact comparison in this environment had `current` as
winner; the assessment self-test is a synthetic validator fixture, not a live
winner claim.
## Merge Recommendation
Ready for final human diff review and then merge into `main`. The previously
open `locked: true` proof now passes for both Smart Capture and auto-scan
persistence.
docs/PROJECT.md
+36 -12
View File
@@ -5,6 +5,7 @@ This document is the source of truth for project intent, scope, runtime facts, a
For implementation structure, see [ARCHITECTURE.md](ARCHITECTURE.md). For engineering standards, see [CONVENTIONS.md](CONVENTIONS.md).
For the latest Inventory-Kamera comparison work, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
For the current branch merge checklist, see [MERGE_READINESS.md](MERGE_READINESS.md).
## Project Identity
@@ -60,7 +61,7 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
| --- | --- | --- |
| Safety | Never perform irreversible in-game actions. | Code review and manual test |
| Performance | Single artifact read should feel interactive and batch scan should not stall on false progress. | Capture latency monitored manually; auto-scan stops on blocked verification |
| IK target | First 100 artifacts should scan with accuracy at least as good as Inventory Kamera and equal or better speed. | `npm run scan:goal:compare` quality-gated report |
| IK target | First 100 artifacts should scan with accuracy at least as good as Inventory Kamera and equal or better speed. | `npm run scan:goal:compare:validated` or `npm run scan:goal:compare:validated:wait` quality-gated report |
| Privacy | Captures and parsed data stay local by default. | No remote upload in scanner path |
| Reliability | Uncertain OCR must be visible to the user. | Confidence and details view |
| Learning loop | Scanner mistakes should become reusable local review samples. | `review-samples.jsonl` |
@@ -104,19 +105,34 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
- Elevated live automation is validated in the current dev environment:
`/automation/probe-click?index=1` changed the selected artifact and
`/scanner/start?limit=2` completed with 2/2 verified reads and 0 misses.
- A 2026-07-08 visible-inventory 50-artifact run completed cleanly with
50/50 parsed and stored, 0 review, 0 duplicates, and 0 misses. It is stable
but still too slow for the 2-3 artifacts/second target.
- Later 2026-07-08 direct-GDI hot-path runs completed 20/20 parsed with
0 misses and 0 review. The best clean 20-artifact iteration reached
7285 ms, or roughly 2.75 artifacts/second; the final stable
`2026-07-08-direct-gdi-reviewfix` run completed in 7973 ms. The
3 artifacts/second target remains unproven.
- The same direct-GDI path completed a 100-artifact run with 100/100 parsed,
0 review, 0 misses, and 42064 ms elapsed across 4 pages.
- Review samples can now be exported with `npm run eval:review-candidates` into
a Git-ignored human-labeling worklist. This is the next quality phase before
adding more OCR corpus cases or trusting review queue data as labels.
### What is still structurally weak
- The scan experience is still partly orchestrated from `src/App.tsx`, which makes behavior changes harder than they should be.
- Broader scan soak testing still needs to increase the live limit gradually and
validate scroll/page transitions beyond the first visible row.
- Broader scan soak testing has reached clean 20-, 45-, and 100-artifact runs
with 0 misses on the current engine. The current-vs-IK-traineddata comparison
is now captured; `current` won the qualified 100-artifact comparison on
2026-07-08.
- OCR quality is still inconsistent enough that some fields are recovered by fallback and derivation more often than they should be.
- Learned fixes currently focus on text replacements; they do not yet update crop offsets, UI profile variants, or scanner targeting rules in a structured way.
- The scan page is cleaner than before, but it still exposes too much operator/debug state in the main flow.
- Recommendations and build logic exist, but the scanner is not yet reliable enough to make them the core focus.
- The latest source has not yet completed the final 100-artifact live comparison
because the current dev-control port is still owned by a stale elevated
Electron process. Live timing must wait for a UAC-approved restart.
- The latest source has completed the final current-vs-IK-traineddata live
comparison for this environment. Repeatability and 3 artifacts/second are
still open.
### Current product conclusion
@@ -171,6 +187,9 @@ Outcome:
5. safe derivation from piece/slot/value references
- Main stat/value inference is tightened with slot constraints and reference tables.
- Bad parses automatically generate structured review reasons.
- Equipped-character parsing is canonical-data constrained: noisy known names can
match through aliases/fuzzy lookup, but unknown footer fragments stay
`Not detected` instead of being stored as invented character names.
Status:
- In progress
@@ -191,6 +210,8 @@ Outcome:
- resume or stop
- Progress counts only when a new verified artifact or duplicate signature is confirmed.
- Repeated pages, unchanged detail cards, blocked cursor movement, and scroll failures stop the scan with diagnosis instead of producing fake progress.
- Fast artifact-read captures include the equipped footer when an equipped
marker is visible; preflight and polling captures still skip expensive OCR.
Status:
- In progress
@@ -225,7 +246,9 @@ Outcome:
- Review samples become both parser regression fixtures and learning inputs.
Status:
- Planned
- Prepared in code for text replacements, field aliases, constrained fixes,
crop adjustment proposals, and UI-profile adjustment proposals. Crop/profile
changes still require live review before being auto-applied.
### Phase 6 - Recommendations come back on top of a trusted scanner
@@ -241,11 +264,12 @@ Status:
1. Finish scan-page cleanup so the main operator view is no longer noisy.
2. Tighten the game data generator and parser contract, then backfill regression tests from real bad samples.
3. Continue moving auto-scan behavior out of `App.tsx` and into isolated scanner modules.
4. Soak-test the elevated C# helper automation path with gradually larger scan limits and page scroll transitions.
5. Run `npm run scan:goal:compare` after `/health.appBuild.signature` matches
the current source and use the quality-gated 100-artifact report as the IK
target evidence.
6. Extend the learning system from text-only fixes into crop/UI profile tuning.
4. Repeat the qualified current-vs-IK-traineddata comparison in a later live
session before making stronger speed/default-engine claims.
5. Validate equipped-character footer reads and a positive `locked=true` sample
from known artifacts; export candidates with `npm run eval:review-candidates`.
6. Grow the confirmed OCR corpus from review samples before tightening parser
thresholds further.
7. Resume recommendation work only when scan accuracy is consistently trustworthy.
## Open Questions
docs/ocr-eval.md
+47 -4
View File
@@ -3,14 +3,19 @@
Field-level accuracy measurement for the artifact OCR parser. This is the gate
every OCR, crop, layout, or parser change runs against (see ADR-007).
It is necessary but not sufficient for the IK target: live scan speed and
review/miss rates are measured by `npm run scan:goal:compare`.
review/miss rates are measured by `npm run scan:iterate:compare:validated` for
short iteration and `npm run scan:goal:compare:validated` for the final
100-artifact proof. Use the `:wait` variants directly after UAC startup.
## Run it
```powershell
npm run eval # full accuracy report for the seed corpus
npm run eval:review-candidates # export unconfirmed review samples for human labeling
npm test # runs the eval gate alongside the rest of the suite
npm run scan:assessment:test # verifies quality-first scan ranking logic
npm run scan:iterate:compare:validated:wait # 20-artifact live comparison
npm run scan:goal:compare:validated:wait # final 100-artifact live comparison
```
The report prints exact-match rate, overall field accuracy, a per-field
@@ -31,6 +36,35 @@ breakdown (critical fields marked with `*`), and every failing case with an
The review queue is the corpus source. A saved review sample carries the OCR
text plus the parser's *guess* - `reviewSampleToEvalCase` extracts both.
For the local Electron queue, run:
```powershell
npm run eval:review-candidates -- --limit=80
```
This writes:
- `outputs/review-eval-candidates/review-eval-candidates.json`
- `outputs/review-eval-candidates/review-eval-candidates.md`
The exporter deduplicates samples, puts complete modern OCR captures first,
marks missing fast-profile fields so stale/partial captures do not crowd out
useful cases, and surfaces ownership/lock evidence (`artifact-footer`,
`equipped`, and `locked=true/false`) for the next validation pass.
After manually checking one candidate against the real artifact, create a
confirmed corpus snippet with explicit expected labels:
```powershell
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
```
The script reads the latest
`outputs/review-eval-candidates/review-eval-candidates.json` by default and
writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`.
It refuses to run without explicit labels, so parser guesses are not silently
promoted to ground truth. Review that snippet, then paste the object into
`src/eval/corpus/confirmedReviewCorpus.ts`.
The parser's guess is a label **candidate, not ground truth** (using it directly
would be the parser grading itself). To add a real case:
@@ -38,11 +72,20 @@ would be the parser grading itself). To add a real case:
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
2. Open each produced case and confirm or correct the `expect` values against
what the artifact actually is in-game. Set `confirmed: true`.
3. Move the corrected case into a file under `src/eval/corpus/` and add it to the
corpus array.
3. Move the corrected case into
`src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports
`src/eval/corpus/index.ts`, which combines the seed corpus with confirmed
review cases.
`src/eval/corpus/confirmedReviewCorpus.test.ts` rejects common corpus mistakes:
duplicate case ids, missing OCR text, empty labels, missing source notes, or a
case that was copied in without `confirmed: true`.
Prefer cases that cover new failure modes: unseen resolutions, new sets or
characters, and OCR noise the current corpus does not exercise.
characters, equipped footer noise, and OCR noise the current corpus does not
exercise. `locked` is a capture-side visual flag rather than a text parser field;
validate it from review-export metadata and live screenshots instead of adding
it to the OCR eval labels.
## When a change moves a number
docs/scanner-ik-progress-report.md
+114 -37
View File
@@ -16,17 +16,22 @@ Inventory-Kamera-style artifact scanner:
missing-detail states block before OCR/store/review work.
- OCR uses a fast artifact profile that skips low-value fields and derives
slot, set, and main-stat value through lookup constraints when safe.
- The OCR worker pool, field crop split, page model, scroll model, and readiness
waits now mirror the relevant IK design choices more closely.
- The OCR worker pool, field crop split, page model, scroll model, and direct
detail-change verification now mirror the relevant IK design choices more
closely.
- Diagnostics now preserve state evidence, timings, screenshots where useful,
entry events, focus/input events, preflight failures, and scan-loop reasons.
- A live soak runner now measures throughput and quality, compares current vs.
IK-traineddata engines, and refuses to run against stale Electron builds.
The requested final goal is not proven complete yet. The current live dev port is
still owned by an older elevated Electron instance, so the latest code cannot be
truthfully benchmarked against IK until the app is restarted with UAC approval
and `npm run scan:goal:compare` completes a qualified 100-artifact run.
The current-vs-IK-traineddata comparison proof is now captured. On 2026-07-08,
`npm run scan:goal:compare:validated` passed with
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`.
At `limit=100`, `current` won the qualified comparison with `100/100` parsed,
`0` review, `0` misses, `378 ms/artifact` active average, and `37800 ms`
projected time for 100 artifacts. `ik-traineddata` was not qualified at
`limit=100` because it parsed `97/100`, had `5` review and `3` misses. The
separate 3 artifacts/second target is still not proven.
## What Changed
@@ -51,13 +56,19 @@ against a canonical package before it is accepted.
name, slot, main-stat label, main-stat value, level, substats, set effects,
equipped/footer, lock, and rarity.
- Fast auto-scan profile skips lower-value OCR work:
set effects, slot crop, main-stat value crop, equipped footer, crop images,
full-frame payloads, and inventory preview payloads.
set effects, main-stat value crop, crop images, full-frame payloads, and
inventory preview payloads. The equipped footer remains in real artifact-read
captures when its marker is visible, because ownership now matters for phase 1
validation. Preflight and poll captures still skip OCR/crops/lock-state work.
The slot crop remains in the fast path because it materially improved
real-read quality.
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
constraints make that safe.
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
- Exact visual duplicates are skipped before OCR.
- Exact visual duplicate skipping is disabled in the hottest path; duplicate
handling now primarily uses parsed artifact signatures so OCR is not skipped
solely from a crop fingerprint collision.
Why this matters:
@@ -76,8 +87,16 @@ manual-debug capture that OCRs every visible thing.
- `npm run scan:goal:current`
- `npm run scan:goal:ik`
- `npm run scan:goal:compare`
- `npm run scan:goal:compare:validated`
- `npm run scan:goal:compare:validated:wait`
- `npm run scan:iterate:compare:validated`
- `npm run scan:iterate:compare:validated:wait`
- `npm run scan:live:preflight`
- `npm run scan:live:preflight:wait`
- `scan-performance-assessment.json` ranks runs by quality first and speed
second.
second, and records whether the 100-artifact result is a complete
current-vs-IK comparison through `goal100Decision` and
`goal100.comparisonComplete`.
Important rule:
@@ -113,12 +132,15 @@ outside the artifact inventory/detail state.
clicking the risky lower band.
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
duplicate reads after scroll.
- Card readiness uses detail fingerprint polling:
max 420 ms, 60 ms polls, changed cards may proceed after 200 ms.
- Detail-change verification now uses the artifact OCR capture itself instead
of a separate card-ready capture before OCR.
- Scroll readiness uses inventory fingerprint polling:
max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
- Store/review writes are queued so the next tile can be clicked before disk
writes finish. The queue is still flushed before final summary.
- Store/review writes are held out of the click/capture/OCR hot path. The
latest source batches artifact store writes before the final summary instead
of issuing one save/reload cycle per artifact.
- Auto-scan artifact captures skip Electron source enumeration in the hot path
and call the GDI capture helper directly.
- Focus is done once at scan start; hot-loop captures do not refocus every tile.
Why this matters:
@@ -175,21 +197,67 @@ Latest repo validation after the recent changes:
| `npm run build` | Passed |
| `git diff --check` | Passed |
Live evidence already collected earlier on 2026-07-07:
Live evidence already collected:
- Probe click changed artifact detail successfully.
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
- Limit 20 live soak completed on the first visible page.
- Limit 45 live soak crossed into a scrolled page.
- On 2026-07-08, `/scanner/start?entry=visible-inventory&limit=50&engine=current`
completed `50/50` parsed and stored, `0` review, `0` duplicates, `0` misses,
`2` pages, `61765 ms` elapsed, `1235 ms/artifact`, `averageCaptureMs: 186`,
and `averageOcrMs: 162`.
- A deferred single-write flush experiment also completed `50/50`, but regressed
to `63616 ms` because `writeFlushMs` was `8163`; the source now uses batch
persist instead, pending a fresh elevated live measurement.
- After direct GDI hot-path optimization, a 20-artifact run completed
`20/20` parsed, `19` stored, `0` review, `1` duplicate, `0` misses,
`7966 ms` elapsed, `398 ms/artifact`, `averageCaptureMs: 193`, and
`averageOcrMs: 167`.
- A 45-artifact direct-GDI run completed `45/45` parsed, `42` stored,
`0` review, `3` duplicates, `0` misses, `2` pages, `18625 ms` elapsed,
`414 ms/artifact`, `averageCaptureMs: 187`, and `averageOcrMs: 162`.
- A 100-artifact direct-GDI run on signature
`2026-07-08-direct-gdi-hotpath` completed `100/100` parsed, `97` stored,
`0` review, `3` duplicates, `0` misses, `4` pages, `42064 ms` elapsed,
`421 ms/artifact`, `averageCaptureMs: 179`, and `averageOcrMs: 154`.
- `npm run eval` passed after the speed work with `23/23` exact-match cases,
`100%` field accuracy, and `100%` critical fields.
- The 3 artifacts/second target is now prepared in code but not live-proven:
artifact hot-path captures omit detail preview payloads, and stats expose
capture roundtrip/overhead timing. A qualifying 20-artifact run must finish in
`<= 6667 ms` with 0 misses and no silent OCR quality regression.
- Follow-up 3/s attempts on 2026-07-08 fixed the false review trigger caused by
omitted detail previews. The best clean `limit=20` run reached `7285 ms`
(`364 ms/artifact`, about `2.75 artifacts/second`) with `20/20` parsed,
`0` review, and `0` misses. The final stable run on
`2026-07-08-direct-gdi-reviewfix` completed `20/20` with `0` review,
`0` misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail-region capture,
5 OCR workers, DataURL buffer decode, and substat `PSM.SINGLE_COLUMN` were
tested and rejected as slower than the direct-GDI baseline.
- Final current-vs-IK-traineddata comparison on 2026-07-08:
`npm run scan:goal:compare:validated` passed. Evidence file:
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`,
`createdAt: 2026-07-08T18:41:11.6120957+02:00`. `goal100Decision` was
`qualified-comparison: winner=current`; `goal100.comparisonComplete` was true.
`current` completed `100/100` parsed, `97` stored, `0` review, `0` misses,
`4` pages, `378 ms/artifact` active average. `ik-traineddata` completed
`97/100` parsed, `92` stored, `5` review, `3` misses and was rejected for
parsing fewer artifacts than requested.
- The review queue now has a bounded corpus-growth workflow:
`npm run eval:review-candidates` writes a deduplicated, Git-ignored worklist
to `outputs/review-eval-candidates/`. This separates complete modern OCR
samples from stale captures and prevents the parser's own guess from being
promoted to ground truth without human confirmation. After manual checking,
`npm run eval:prepare-confirmed` turns one candidate plus explicit expected
labels into a paste-ready confirmed corpus snippet.
Current live limitation:
- `/health` still reports an older elevated build:
`2026-07-07-ocr-pool4-hotloop-no-refocus`.
- Current source expects:
`2026-07-07-ik32-fastsubstats-active-timing`.
- The live soak runner correctly refuses to benchmark the stale runtime.
- A UAC restart attempt was canceled, so the latest code is not yet live.
- The fast path and current-vs-IK-traineddata 100-artifact comparison are proven
in the current live environment. 3 artifacts/second is not proven; remaining
speed work needs a larger OCR or capture-pipeline change, not more click
tuning.
## Inventory Kamera Comparison
@@ -199,13 +267,14 @@ Current live limitation:
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
| Item wait | About 200 ms fixed wait | Fingerprint polling, accepts changed card after 200 ms |
| Item wait | About 200 ms fixed wait | No separate wait capture; OCR capture verifies changed detail |
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
| Capture hot path | Direct window/screen capture without source-list scan per item | Direct GDI capture in auto-scan artifact loop |
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
| 100-artifact proof | Reference target | Not yet proven on latest app build |
| 100-artifact proof | Reference target | Qualified current-vs-IK comparison captured; `current` won with `100/100`, 0 review, 0 misses, 37.8s projected |
What is theoretically better than before:
@@ -220,19 +289,29 @@ What is theoretically better than before:
What is not yet proven better than IK:
- Native Tesseract speed is not integrated as the default.
- The latest code has not completed the 100-artifact live run.
- Review rate and miss rate on the user's real inventory still need the new
live report.
- 3 artifacts/second is not proven.
- Native Inventory Kamera outside this app was not re-run in the same session;
the completed comparison is against the bundled `ik-traineddata` scan engine.
## Theoretical Runtime Flow
For short iteration while tuning:
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Run `npm run scan:iterate:compare:validated:wait` from a visible artifact inventory
when starting directly after UAC, or `npm run scan:iterate:compare:validated`
if preflight already passes.
3. Inspect `scan-performance-assessment.json`, review samples, and timings if the
20-artifact comparison fails quality gates.
For the intended 100-artifact comparison:
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run `npm run scan:goal:compare`.
5. Run `npm run scan:goal:compare:validated:wait` directly after UAC, or
`npm run scan:goal:compare:validated` if preflight already passes.
6. For each engine and limit (`2, 5, 20, 45, 100`):
- focus Genshin once,
- verify lookup and layout,
@@ -274,17 +353,15 @@ Expected bottleneck sequence:
## Risks and Remaining Work
1. Restart with UAC and run the latest build live.
2. Run `npm run scan:goal:compare` from a visible artifact inventory.
3. If the 100-artifact winner is not qualified, inspect:
`scan-performance-assessment.json`, review samples, diagnostic timeline, and
field timings.
4. If `ik-traineddata` wins but Tesseract.js is still slow, evaluate native
Tesseract integration.
5. Grow the eval corpus with confirmed real review samples before tightening
1. Keep `current` as the default OCR engine for now; it won the qualified
current-vs-IK-traineddata live comparison.
2. If pursuing 3 artifacts/second, focus on capture/OCR pipeline changes rather
than click timing.
3. Grow the eval corpus with confirmed real review samples before tightening
parser thresholds further.
6. Validate a positive locked-artifact sample.
7. Keep recommendations secondary until scanner quality is proven.
4. Validate a positive locked-artifact sample.
5. Keep recommendations secondary until scanner quality remains stable across
repeated live sessions.
## Definition of Done for the IK Target
docs/scanner-rework-status.md
+110 -25
View File
@@ -14,12 +14,19 @@ Current status:
- The scanner architecture now follows the relevant Inventory Kamera model:
32 artifact targets per page, lookup-derived fields, fast artifact OCR profile,
short readiness gates, page-overlap planning, and queued OCR/store work.
direct detail-fingerprint verification from the OCR capture, page-overlap
planning, and batched store work.
- The live runner can compare `current` and `ik-traineddata` engines and rejects
runs that are fast but fail miss/review quality thresholds.
- The final 100-artifact IK target is not proven yet. The dev-control port is
currently owned by an older elevated Electron build, and the runner correctly
refuses stale timing evidence until the app is restarted with UAC approval.
- The final current-vs-IK-traineddata 100-artifact comparison is now proven for
the current live environment. On 2026-07-08,
`npm run scan:goal:compare:validated` passed with evidence at
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`.
`current` won with `100/100` parsed, `0` review, `0` misses, and
`378 ms/artifact` active average. `ik-traineddata` was rejected at 100 because
it parsed `97/100`, had `5` review and `3` misses. The next optional speed
target remains `3 artifacts/second`, which means `333 ms/artifact` or faster
on clean 20-artifact iterations.
## Done (implemented, unit-tested, build green)
@@ -55,13 +62,13 @@ Current status:
2 review samples, and 0 misses.
- **Auto-scan OCR performance pass** - auto-scan captures now use an artifact
OCR mode that skips inventory-count OCR on each tile, keeps equipped-character
OCR, raises the substat crop to catch artifact level, stores automatic review
samples without full-screen/inventory screenshots, reads only the tail of large
OCR on the real artifact-read captures, raises the substat crop to catch
artifact level, stores automatic review samples without full-screen/inventory screenshots, reads only the tail of large
JSONL files, avoids review noise when only level/equipped is missing, starts
the scan with an OCR-free preflight capture, skips exact visual duplicates
before OCR, prevents repeated startup review reprocessing, omits full-frame
and inventory-preview Base64 payloads from tile captures, and applies
crop-specific Tesseract page-segmentation/whitelist parameters.
the scan with an OCR-free preflight capture, prevents repeated startup review
reprocessing, omits full-frame and inventory-preview Base64 payloads from tile
captures, and applies crop-specific Tesseract page-segmentation/whitelist
parameters.
- **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the
dev-control health/status, smart-capture, probe-click, bounded scan, and
review-tail endpoints and writes evidence to `outputs/live-soak/`. On
@@ -105,10 +112,77 @@ Current status:
summaries, groups results by limit, identifies timing bottlenecks, and rejects
winners that miss the requested count, exceed 2% misses, or exceed 15% review.
`npm run scan:assessment:test` verifies this ranking logic without Genshin.
The assessment also reports `goal100Decision` and
`goal100.comparisonComplete`, so a single-engine 100-artifact run cannot be
misread as the final IK comparison. Use
`npm run scan:iterate:compare:validated:wait` for the 20-artifact live
iteration and `npm run scan:goal:compare:validated:wait` for the final proof
when starting directly after UAC. The validator `--summary` output includes
the assessment path and timestamp for reporting.
- **State-polled guided entry** - the guided auto-entry waits for Inventory,
artifact grid, and first detail card evidence instead of sleeping the full
fixed delay every time. OCR/review/store work still starts only after artifact
detail preflight passes.
- **Hot-loop speed pass (2026-07-08)** - the scan loop no longer performs a
separate card-ready capture before OCR; the artifact OCR capture itself
verifies detail-fingerprint change. Routine click diagnostics and scan stat
publishes are throttled. Auto-scan artifact captures no longer update the
full preview/topbar UI on every tile. Store writes can be batched so the scan
path avoids per-artifact save/reload churn. Auto-scan artifact captures now
use a direct GDI hot path and skip Electron `desktopCapturer.getSources()` in
the per-artifact loop.
- **3/s instrumentation pass (2026-07-08)** - artifact hot-path captures omit
the detail-preview payload, and scan stats now split inner capture time from
end-to-end capture roundtrip time. Use `averageCaptureRoundTripMs` and
`averageCaptureRoundTripOverheadMs` in the next `limit=20` live iteration to
decide whether the next cut belongs in native capture transport or OCR.
- **3/s live attempt (2026-07-08)** - the missing-detail-preview review trigger
was fixed and tested. The best clean 20-artifact run reached `7285 ms`
(`364 ms/artifact`, about `2.75 artifacts/second`) with 0 review and 0 misses.
The final stable run on `2026-07-08-direct-gdi-reviewfix` completed `20/20`
with 0 review, 0 misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail
region capture, 5 OCR workers, DataURL buffer decode, and substat
`PSM.SINGLE_COLUMN` were tested and rejected as slower.
- **Review-to-eval loop (2026-07-08)** - `npm run eval:review-candidates`
exports the local review queue into `outputs/review-eval-candidates/` as a
human-labeling worklist. The exporter deduplicates samples, surfaces complete
fast-field captures first, marks stale captures, and now surfaces equipped
footer OCR plus `locked=true/false` payload counts for the next ownership/lock
validation pass. Its output is deliberately
ignored by Git and must not be treated as ground truth until fields are
confirmed against the real artifact. Confirmed review labels now have a
dedicated corpus file, `src/eval/corpus/confirmedReviewCorpus.ts`, with tests
that reject duplicate ids, empty labels, and unconfirmed entries. The helper
`npm run eval:prepare-confirmed` generates a paste-ready confirmed-case
snippet only when explicit expected labels are provided.
- **Prepared ownership/learning loop (2026-07-08)** - fast auto-scan no longer
drops the artifact footer by profile alone; it omits footer OCR only when the
capture option explicitly requests that or when the footer marker is absent.
Parser tests cover noisy equipped names, split `Equipped:`/name footers, and
one-letter OCR fragments that must stay `Not detected`. Scanner learning now
persists text replacements, field aliases, constrained fixes, crop adjustment
proposals, and UI-profile adjustment proposals instead of truncating everything
back to text replacements.
- **Visible-inventory merge guard (2026-07-09)** - the normal guided Auto-Scan
start no longer falls back into `auto-entry` when the artifact detail card is
missing. It now blocks and asks the operator to open the Artifact inventory
with a visible detail card. The explicit `auto-entry`, `direct-inventory`, and
`paimon-menu` Dev-Control modes remain available for targeted experiments, but
they are not the merge-ready default path.
- **Ownership live smoke (2026-07-09)** - live artifact detail capture parsed
and stored an equipped footer as `equipped: "Citlali"` and the grey lock state
as `locked: false`. A same-session visible-inventory run with
`/scanner/start?entry=visible-inventory&limit=20&engine=current` completed
`20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and
`0` misses in `8047 ms` elapsed (`402 ms/artifact`).
- **Locked artifact live proof (2026-07-09)** - a visibly locked artifact was
selected through a read-only inventory tile click. Smart Capture reported
`locked: true` with `lockSignal.ratio: 0.14797913950456323` over threshold
`0.06`, and `/scanner/start?entry=visible-inventory&limit=1&engine=current`
persisted the same artifact with `equipped: "Citlali"` and `locked: true`.
Lock detection now decodes the lock crop PNG before measuring active lock
pixels because Electron's native bitmap channel order was ambiguous in live
captures.
## Remaining — needs the live environment or a UI pass
@@ -122,23 +196,34 @@ resolution or without UI work best tested live:
The current benchmark can use IK-traineddata through Tesseract.js; native
Tesseract integration remains the next implementation step before any engine
default changes.
3. **Validate guided entry live** from world, visible inventory, and Paimon/menu
states with limits 2, 20, and 45. Confirm the artifact-tab coordinate in the
user's current 16:9 layout and keep `visible-inventory` as fallback if the
menu path is blocked.
4. **Validate locked=true** against a known locked artifact — unlocked/grey lock
was live-checked; a gold locked icon still needs a positive sample.
3. **Validate explicit entry modes separately** from world, direct inventory,
and Paimon/menu states with low limits only. These are now Dev-Control
experiments, not the normal merge path; the normal Auto-Scan button blocks
unless the visible artifact detail card is already present.
4. **Repeat locked=true on another page/session** if lock behavior changes.
The first positive live proof passed on 2026-07-09, including store
persistence. Further repeats are useful for confidence but no longer block
the merge.
5. **Broader scan soak test** — after the bounded two-item live scan passed,
the next automation validation should increase the limit gradually and watch
for repeated pages, scroll behavior, duplicate handling, and OCR review rate.
6. **100-artifact IK comparison** — after `/health.appBuild.signature` matches
current source, run `npm run scan:goal:compare` and compare qualified
100-artifact results.
5. **3 artifacts/second iteration** - not reached yet. The next credible path is
either native Tesseract/IK-traineddata integration that materially reduces
substat OCR time, or a larger capture pipeline change that avoids full-frame
PNG/Base64 transport without hurting safety checks. The target remains
`<= 6667 ms` elapsed for 20 parsed artifacts with 0 misses and no silent OCR
review regression.
Visible-page limits up to 20 and a scroll/page-transition limit of 45 have
passed. The remaining soak work is now OCR accuracy, review-rate reduction, and
larger runs after the review corpus has grown.
6. **Broader scan soak test** — direct-GDI current-engine runs now passed at
`20/20`, `45/45`, and `100/100` with 0 misses. Continue with repeat runs if
duplicate rate needs tuning.
7. **Repeatability pass** — repeat the qualified current-vs-IK-traineddata run
in a later live session before making major OCR-engine defaults or speed
claims beyond this environment.
Visible-page limits up to 20, scroll/page-transition limit 45, and the final
100-artifact current-vs-IK-traineddata comparison have passed for the current
environment. Remaining soak work is repeatability, OCR corpus growth, equipped
footer confirmation repeats, locked artifact repeats, and optional 3 artifacts/second
speed work.
## Grow the eval corpus
electron/appWindowManager.ts
+145
View File
@@ -0,0 +1,145 @@
import { BrowserWindow, Menu, screen } from "electron";
import path from "node:path";
export interface AppWindowManagerOptions {
preloadPath: string;
rendererUrl?: string;
rendererFilePath: string;
onMainReadyToShow?: () => void | Promise<void>;
}
export interface AppWindowManager {
createMainWindow: () => void;
focusMainWindow: () => { ok: boolean };
hasMainWindow: () => boolean;
getMainWindow: () => BrowserWindow | null;
sendScannerCommand: (command: unknown) => void;
createOverlayWindow: () => void;
hideOverlayWindow: () => { ok: boolean };
}
export function createAppWindowManager({
preloadPath,
rendererUrl,
rendererFilePath,
onMainReadyToShow,
}: AppWindowManagerOptions): AppWindowManager {
let mainWindow: BrowserWindow | null = null;
let overlayWindow: BrowserWindow | null = null;
function createMainWindow() {
Menu.setApplicationMenu(null);
mainWindow = new BrowserWindow({
width: 1320,
height: 860,
minWidth: 1120,
minHeight: 720,
backgroundColor: "#090711",
title: "Genshin Artifact Assistant",
show: false,
autoHideMenuBar: true,
webPreferences: {
preload: preloadPath,
contextIsolation: true,
nodeIntegration: false,
},
});
mainWindow.setMenuBarVisibility(false);
mainWindow.on("closed", () => {
mainWindow = null;
});
mainWindow.once("ready-to-show", () => {
void onMainReadyToShow?.();
focusMainWindow();
});
mainWindow.webContents.once("did-finish-load", () => {
setTimeout(() => focusMainWindow(), 350);
});
if (rendererUrl) {
mainWindow.loadURL(rendererUrl);
} else {
mainWindow.loadFile(rendererFilePath);
}
}
function focusMainWindow() {
if (!mainWindow || mainWindow.isDestroyed()) return { ok: false };
if (mainWindow.isMinimized()) mainWindow.restore();
mainWindow.show();
// Genshin often keeps foreground focus after a scan click. Toggling
// always-on-top for one tick nudges Windows to surface the dashboard again
// without leaving it pinned above other apps.
mainWindow.setAlwaysOnTop(true, "screen-saver");
mainWindow.focus();
setTimeout(() => {
if (!mainWindow || mainWindow.isDestroyed()) return;
mainWindow.setAlwaysOnTop(false);
mainWindow.focus();
}, 250);
return { ok: true };
}
function sendScannerCommand(command: unknown) {
if (!mainWindow || mainWindow.isDestroyed()) return;
mainWindow.webContents.send("scanner:command", command);
}
function createOverlayWindow() {
if (overlayWindow) {
overlayWindow.show();
return;
}
const display = screen.getPrimaryDisplay();
overlayWindow = new BrowserWindow({
x: display.workArea.x,
y: display.workArea.y,
width: display.workArea.width,
height: display.workArea.height,
transparent: true,
frame: false,
alwaysOnTop: true,
skipTaskbar: true,
resizable: false,
focusable: false,
webPreferences: {
preload: preloadPath,
contextIsolation: true,
nodeIntegration: false,
},
});
overlayWindow.setIgnoreMouseEvents(true, { forward: true });
if (rendererUrl) {
overlayWindow.loadURL(`${rendererUrl}?overlay=1`);
} else {
overlayWindow.loadFile(rendererFilePath, {
query: { overlay: "1" },
});
}
overlayWindow.on("closed", () => {
overlayWindow = null;
});
}
function hideOverlayWindow() {
overlayWindow?.close();
return { ok: true };
}
return {
createMainWindow,
focusMainWindow,
hasMainWindow: () => Boolean(mainWindow && !mainWindow.isDestroyed()),
getMainWindow: () => mainWindow,
sendScannerCommand,
createOverlayWindow,
hideOverlayWindow,
};
}
electron/devControlServer.ts
+5 -1
View File
@@ -113,6 +113,7 @@ async function writeDevCaptureSnapshot(capture: CaptureResult) {
: null,
inventoryCount: capture.inventoryCount ?? null,
locked: capture.locked,
lockSignal: capture.lockSignal,
crops: (capture.crops ?? []).map((crop) => ({ id: crop.id, label: crop.label, rect: crop.rect })),
ocr: capture.ocr ?? [],
files,
@@ -339,7 +340,10 @@ export function createDevControlServer(deps: DevControlServerDependencies): Serv
captures,
};
}
const summaries = await Promise.all(engines.map((engine) => runEngineBenchmark(engine)));
const summaries = [];
for (const engine of engines) {
summaries.push(await runEngineBenchmark(engine));
}
writeDevJson(res, 200, {
ok: true,
summary: summaries.length === 1 ? summaries[0] : { mode: "compare", limit, ocrProfile, engines: summaries },
electron/main.ts
+103 -163
View File
@@ -1,5 +1,4 @@
import { app, BrowserWindow, Menu, desktopCapturer, dialog, globalShortcut, nativeImage, screen, type NativeImage } from "electron";
import fs from "node:fs/promises";
import { app, desktopCapturer, globalShortcut, nativeImage, screen, type NativeImage } from "electron";
import { existsSync } from "node:fs";
import type { Server } from "node:http";
import { cpus } from "node:os";
@@ -7,18 +6,19 @@ import path from "node:path";
import { fileURLToPath } from "node:url";
import { createWorker, PSM } from "tesseract.js";
import { createInputHelperService, type InputHelperService } from "./services/inputHelper.js";
import { pngBufferToBitmap } from "./services/pngBitmap.js";
import { createRepositoryContext, type RepositoryContext } from "./bootstrap/repositoryContext.js";
import { registerIpcHandlers } from "./bootstrap/ipcBootstrap.js";
import { createDevControlServer } from "./devControlServer.js";
import { createAppWindowManager, type AppWindowManager } from "./appWindowManager.js";
import { createGoodFileService, type GoodFileService } from "./services/goodFileService.js";
import type { AppSnapshot } from "../src/types/domain.js";
import type {
CaptureOptions,
CaptureResult,
GoodDatabase,
GoodImportFileResult,
OcrResult,
AppRuntimeInfo,
SaveResultWithPath,
ScannerCommand,
ScannerLearningRulePayload,
ScannerStatusPayload,
@@ -39,7 +39,7 @@ import {
profileDetailRect,
} from "../src/lib/layoutProfile.js";
import { binarizeForOcr } from "../src/lib/ocrPreprocess.js";
import { detectLockState, lockIconCropRect } from "../src/lib/lockDetection.js";
import { DEFAULT_LOCK_THRESHOLD, isLocked, lockIconCropRect, lockSignalRatio } from "../src/lib/lockDetection.js";
// Chromium's renderer sandbox can refuse to fully initialize (or silently
// crash the GPU/renderer process) when the hosting process runs with a full
@@ -54,12 +54,11 @@ app.commandLine.appendSwitch("disable-gpu-sandbox");
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const isDev = Boolean(process.env.VITE_DEV_SERVER_URL);
const APP_RUNTIME_STARTED_AT = new Date().toISOString();
const APP_RUNTIME_SIGNATURE = "2026-07-07-ik32-fastsubstats-active-timing";
const APP_RUNTIME_SIGNATURE = "2026-07-08-direct-gdi-reviewfix";
let mainWindow: BrowserWindow | null = null;
let overlayWindow: BrowserWindow | null = null;
let registeredHotkeys: Record<string, boolean> = {};
let devControlServer: Server | null = null;
const captureSourceNameCache = new Map<string, string>();
let scannerDevStatus: ScannerStatusPayload = {
running: false,
reviewStatus: "",
@@ -83,6 +82,8 @@ let artifactStoreRepository: ArtifactStoreRepositoryPort | null = null;
let reviewSamplesRepository: ReviewSamplesRepositoryPort | null = null;
let scannerLearningRepository: ScannerLearningRepositoryPort | null = null;
let inputHelperService: InputHelperService | null = null;
let appWindowManager: AppWindowManager | null = null;
let goodFileService: GoodFileService | null = null;
function getInputHelperService() {
if (!inputHelperService) {
@@ -97,6 +98,7 @@ function resolveInputHelperExePath(): string | null {
const candidates = [
process.env.INPUT_HELPER_EXE,
path.join(process.resourcesPath, "input-helper", "InputHelper.exe"),
path.join(process.cwd(), "native", "input-helper", "bin", "publish", "InputHelper.exe"),
path.join(app.getAppPath(), "native", "input-helper", "bin", "publish", "InputHelper.exe"),
].filter((candidate): candidate is string => Boolean(candidate));
@@ -163,7 +165,7 @@ function getScannerLearningRepository() {
async function writeScannerLearningRules(rules: ScannerLearningRulePayload) {
const safeRules = rules && typeof rules === "object" ? rules : {};
try {
return await getScannerLearningRepository().save(safeRules as { textReplacements?: Record<string, string> });
return await getScannerLearningRepository().save(safeRules);
} catch {
return { ok: true, path: scannerLearningPath(), rules: { textReplacements: {} }, total: 0 };
}
@@ -184,8 +186,11 @@ function getSnapshotRepository() {
return context.snapshotRepository;
}
function exportPath(fileName: string) {
return path.join(app.getPath("userData"), "exports", fileName);
function getGoodFileService() {
if (!goodFileService) {
throw new Error("GOOD file service is not initialized.");
}
return goodFileService;
}
async function loadSnapshotFromDisk() {
@@ -264,6 +269,7 @@ async function getGenshinWindowBounds() {
// second display exists and isn't the one Genshin occupies, move the
// dashboard there so it can never cover the grid we're about to click.
async function moveMainWindowOffGenshin() {
const mainWindow = getAppWindowManager().getMainWindow();
if (!mainWindow || mainWindow.isDestroyed()) return;
const genshinBounds = await getGenshinWindowBounds();
const displays = screen.getAllDisplays();
@@ -304,8 +310,7 @@ async function showOverlayWindow() {
}
async function hideOverlayWindow() {
overlayWindow?.close();
return { ok: true };
return getAppWindowManager().hideOverlayWindow();
}
async function listCaptureSources() {
@@ -324,12 +329,15 @@ async function listCaptureSources() {
fetchWindowIcons: true,
});
return sources.map((source) => ({
id: source.id,
name: source.name,
isGenshinCandidate: isLikelyGenshinSourceName(source.name),
thumbnailDataUrl: source.thumbnail.resize({ width: 420 }).toDataURL(),
}));
return sources.map((source) => {
captureSourceNameCache.set(source.id, source.name);
return {
id: source.id,
name: source.name,
isGenshinCandidate: isLikelyGenshinSourceName(source.name),
thumbnailDataUrl: source.thumbnail.resize({ width: 420 }).toDataURL(),
};
});
}
async function toScreenPoint(x: number, y: number) {
@@ -394,69 +402,45 @@ async function capturePrimaryScreenViaGdi() {
async function captureSourceFromGdi(sourceId: string, sourceName: string, options: CaptureOptions = {}) {
const gdi = await capturePrimaryScreenViaGdi();
const sourceImage = nativeImage.createFromDataURL(gdi.dataUrl);
const sourceImage = nativeImageFromGdiCapture(gdi);
return await buildCaptureResult(sourceImage, sourceId, sourceName, gdi.captureTarget, options);
}
function createMainWindow() {
Menu.setApplicationMenu(null);
function nativeImageFromGdiCapture(gdi: Awaited<ReturnType<InputHelperService["capturePrimaryScreenViaGdi"]>>) {
return nativeImage.createFromDataURL(gdi.dataUrl);
}
mainWindow = new BrowserWindow({
width: 1320,
height: 860,
minWidth: 1120,
minHeight: 720,
backgroundColor: "#090711",
title: "Genshin Artifact Assistant",
show: false,
autoHideMenuBar: true,
webPreferences: {
preload: path.join(__dirname, "preload.cjs"),
contextIsolation: true,
nodeIntegration: false,
},
});
function shouldUseDirectGdiHotPath(options: CaptureOptions = {}) {
return Boolean(
options.ocrMode === "artifact" ||
options.skipOcrUnlessArtifactDetail ||
options.skipOcr ||
options.omitCrops,
);
}
mainWindow.setMenuBarVisibility(false);
mainWindow.on("closed", () => {
mainWindow = null;
});
mainWindow.once("ready-to-show", () => {
void moveMainWindowOffGenshin();
focusMainWindow();
});
mainWindow.webContents.once("did-finish-load", () => {
setTimeout(() => focusMainWindow(), 350);
});
if (isDev) {
mainWindow.loadURL(process.env.VITE_DEV_SERVER_URL!);
} else {
mainWindow.loadFile(path.join(__dirname, "../../dist/index.html"));
function getAppWindowManager() {
if (!appWindowManager) {
appWindowManager = createAppWindowManager({
preloadPath: path.join(__dirname, "preload.cjs"),
rendererUrl: process.env.VITE_DEV_SERVER_URL,
rendererFilePath: path.join(__dirname, "../../dist/index.html"),
onMainReadyToShow: moveMainWindowOffGenshin,
});
}
return appWindowManager;
}
function createMainWindow() {
getAppWindowManager().createMainWindow();
}
function focusMainWindow() {
if (!mainWindow || mainWindow.isDestroyed()) return { ok: false };
if (mainWindow.isMinimized()) mainWindow.restore();
mainWindow.show();
// Genshin often keeps foreground focus after a scan click. Toggling
// always-on-top for one tick nudges Windows to surface the dashboard again
// without leaving it pinned above other apps.
mainWindow.setAlwaysOnTop(true, "screen-saver");
mainWindow.focus();
setTimeout(() => {
if (!mainWindow || mainWindow.isDestroyed()) return;
mainWindow.setAlwaysOnTop(false);
mainWindow.focus();
}, 250);
return { ok: true };
return getAppWindowManager().focusMainWindow();
}
function sendScannerCommand(command: ScannerCommand | "probe-click") {
if (!mainWindow || mainWindow.isDestroyed()) return;
mainWindow.webContents.send("scanner:command", command);
getAppWindowManager().sendScannerCommand(command);
}
function registerScannerHotkeys() {
@@ -474,7 +458,7 @@ function startDevControlServer() {
devControlServer = createDevControlServer({
registeredHotkeys,
appBuild: appRuntimeInfo(),
hasMainWindow: () => Boolean(mainWindow && !mainWindow.isDestroyed()),
hasMainWindow: () => getAppWindowManager().hasMainWindow(),
sendScannerCommand,
clickScreen: clickScreenCommand,
scannerStatus: () => ({ ...scannerDevStatus, appBuild: appRuntimeInfo(), ocrWarmup: getOcrWarmupStatus() }),
@@ -490,43 +474,7 @@ function startDevControlServer() {
}
function createOverlayWindow() {
if (overlayWindow) {
overlayWindow.show();
return;
}
const display = screen.getPrimaryDisplay();
overlayWindow = new BrowserWindow({
x: display.workArea.x,
y: display.workArea.y,
width: display.workArea.width,
height: display.workArea.height,
transparent: true,
frame: false,
alwaysOnTop: true,
skipTaskbar: true,
resizable: false,
focusable: false,
webPreferences: {
preload: path.join(__dirname, "preload.cjs"),
contextIsolation: true,
nodeIntegration: false,
},
});
overlayWindow.setIgnoreMouseEvents(true, { forward: true });
if (isDev) {
overlayWindow.loadURL(`${process.env.VITE_DEV_SERVER_URL!}?overlay=1`);
} else {
overlayWindow.loadFile(path.join(__dirname, "../../dist/index.html"), {
query: { overlay: "1" },
});
}
overlayWindow.on("closed", () => {
overlayWindow = null;
});
getAppWindowManager().createOverlayWindow();
}
// Inventory Kamera keeps a pool of native Tesseract engines and scans artifact
@@ -818,8 +766,8 @@ async function runOcrOnCropsWithTimeout(crops: OcrCropPayload[], engine: OcrWork
function cleanOcrText(cropId: string, text: string) {
const normalized = text
.replace(/[“”]/g, '"')
.replace(/[’]/g, "'")
.replace(/[\u201c\u201d]/g, '"')
.replace(/[\u2019]/g, "'")
.replace(/\r/g, "")
.split("\n")
.map((line) => line.replace(/\s+/g, " ").trim())
@@ -1126,6 +1074,7 @@ async function getAllSources() {
async function findCaptureSourceById(sourceId: string) {
const sources = await getAllSources();
for (const source of sources) captureSourceNameCache.set(source.id, source.name);
return sources.find((source) => source.id === sourceId) ?? null;
}
@@ -1172,7 +1121,7 @@ function imageCropFingerprint(sourceImage: NativeImage, rect: Electron.Rectangle
function preprocessedCropPngBuffer(sourceImage: NativeImage, rect: Electron.Rectangle, imageSize: { width: number; height: number }, cropId = "") {
const safeRect = clampCaptureRect(rect, imageSize);
const scale = cropId === "artifact-level" ? 3 : 2;
const upscaled = sourceImage.crop(safeRect).resize({ width: Math.max(1, safeRect.width * scale), quality: "best" });
const upscaled = sourceImage.crop(safeRect).resize({ width: Math.max(1, Math.round(safeRect.width * scale)), quality: "best" });
const size = upscaled.getSize();
if (!size.width || !size.height) return upscaled.toPNG();
const binarized = binarizeForOcr(
@@ -1201,10 +1150,9 @@ function createCrops(
.filter((template) => {
if (fastArtifactProfile && (
template.id === "artifact-set-effects" ||
template.id === "artifact-slot" ||
template.id === "artifact-main-stat-value"
)) return false;
if (template.id === "artifact-footer" && (options.omitEquippedOcr || fastArtifactProfile)) return false;
if (template.id === "artifact-footer" && options.omitEquippedOcr) return false;
if (!isArtifactScanMode || template.id !== "artifact-footer" || !bitmap) return true;
return hasEquippedFooterMarker(bitmap, imageSize, template.rect);
});
@@ -1314,16 +1262,35 @@ async function buildCaptureResult(
const crops = omitCrops
? []
: createCrops(sourceImage, size, detailRect, inventoryRect, options, bitmap, { sanctified, skipOcr: shouldSkipOcr });
const locked = options.omitLockState
const lockSignal = options.omitLockState
? undefined
: (() => {
const lockRect = clampCaptureRect(lockIconCropRect(detailRect, size), size);
const lockImage = sourceImage.crop(lockRect);
const lockSize = lockImage.getSize();
return lockSize.width > 0 && lockSize.height > 0
? detectLockState({ data: lockImage.getBitmap(), width: lockSize.width, height: lockSize.height })
const ratio = lockSize.width > 0 && lockSize.height > 0
? (() => {
try {
return lockSignalRatio(pngBufferToBitmap(lockImage.toPNG()));
} catch {
return lockSignalRatio({ data: lockImage.getBitmap(), width: lockSize.width, height: lockSize.height });
}
})()
: undefined;
return typeof ratio === "number"
? {
ratio,
threshold: DEFAULT_LOCK_THRESHOLD,
rect: {
x: lockRect.x,
y: lockRect.y,
width: lockRect.width,
height: lockRect.height,
},
}
: undefined;
})();
const locked = lockSignal ? isLocked(lockSignal.ratio, lockSignal.threshold) : undefined;
const omitFullFrame = Boolean(options.omitFullFrame || options.ocrMode === "artifact");
const omitDetailPreview = Boolean(options.omitDetailPreview);
const omitInventoryPreview = Boolean(options.omitInventoryPreview || options.ocrMode === "artifact");
@@ -1383,6 +1350,7 @@ async function buildCaptureResult(
paimonMenu,
inventoryCount: count,
locked,
lockSignal,
sanctified,
layout: {
aspect: aspectRatioLabel(size),
@@ -1407,16 +1375,27 @@ async function buildCaptureResult(
}
async function captureSource(sourceId: string, delayMs = 0, focusGenshin = false, options?: CaptureOptions): Promise<CaptureResult> {
const captureStartedAt = Date.now();
if (!Number.isFinite(delayMs) || delayMs < 0) {
delayMs = 0;
}
const withElapsed = (capture: CaptureResult): CaptureResult => ({
...capture,
elapsedMs: Math.max(0, Date.now() - captureStartedAt),
});
await waitDelay(Math.floor(delayMs));
if (focusGenshin) {
await focusGenshinForScanStart();
}
if (shouldUseDirectGdiHotPath(options ?? {})) {
const cachedName = captureSourceNameCache.get(sourceId) ?? "Genshin GDI Capture";
return withElapsed(await captureSourceFromGdi(sourceId, cachedName, options ?? {}));
}
const source = await findCaptureSourceById(sourceId);
if (!source) {
throw new Error("Capture source not found.");
@@ -1425,7 +1404,7 @@ async function captureSource(sourceId: string, delayMs = 0, focusGenshin = false
const isGenshinCandidate = isLikelyGenshinSourceName(source.name);
if (isGenshinCandidate) {
try {
return await captureSourceFromGdi(sourceId, source.name, options ?? {});
return withElapsed(await captureSourceFromGdi(sourceId, source.name, options ?? {}));
} catch {
// Fall back to desktop thumbnail capture for robustness in low-permission
// or transient capture failures. OCR will still produce a best-effort result.
@@ -1434,56 +1413,16 @@ async function captureSource(sourceId: string, delayMs = 0, focusGenshin = false
const sourceImage = source.thumbnail;
if (sourceImage.isEmpty()) {
return await captureSourceFromGdi(sourceId, source.name, options ?? {});
return withElapsed(await captureSourceFromGdi(sourceId, source.name, options ?? {}));
}
return await buildCaptureResult(
return withElapsed(await buildCaptureResult(
sourceImage,
sourceId,
source.name,
sourceId.startsWith("screen:") ? "desktop-source" : "genshin-client",
options ?? {},
);
}
async function exportGood(payload: GoodDatabase): Promise<SaveResultWithPath> {
const fileNameSafe = `good-export-${new Date().toISOString().replace(/[\\/:]/g, "-").replace(/\..+?$/, "").replace(/\s+/g, "-")}.json`;
const filePath = exportPath(fileNameSafe);
try {
await fs.mkdir(path.dirname(filePath), { recursive: true });
await fs.writeFile(filePath, JSON.stringify(payload, null, 2), "utf8");
return { ok: true, path: filePath };
} catch {
return { ok: false, path: filePath };
}
}
async function importGoodFile(): Promise<GoodImportFileResult> {
const dialogOptions = {
title: "GOOD-Datei importieren",
properties: ["openFile"],
filters: [{ name: "GOOD JSON", extensions: ["json"] }],
} satisfies Electron.OpenDialogOptions;
const dialogResult = mainWindow && !mainWindow.isDestroyed()
? await dialog.showOpenDialog(mainWindow, dialogOptions)
: await dialog.showOpenDialog(dialogOptions);
if (dialogResult.canceled || dialogResult.filePaths.length === 0) {
return { ok: false, canceled: true, path: "" };
}
const filePath = dialogResult.filePaths[0];
try {
const text = await fs.readFile(filePath, "utf8");
return { ok: true, canceled: false, path: filePath, database: JSON.parse(text) };
} catch (error) {
return {
ok: false,
canceled: false,
path: filePath,
error: error instanceof Error ? error.message : String(error),
};
}
));
}
function initializeAppLifecycle() {
@@ -1494,6 +1433,7 @@ function initializeAppLifecycle() {
reviewSamplesRepository = repositoryContext.reviewSamplesRepository;
scannerLearningRepository = repositoryContext.scannerLearningRepository;
inputHelperService = createInputHelperService({ userDataPath, exePath: resolveInputHelperExePath() });
goodFileService = createGoodFileService(path.join(userDataPath, "exports"));
registerIpcHandlers({
focusMainWindow: () => focusMainWindow(),
@@ -1513,8 +1453,8 @@ function initializeAppLifecycle() {
loadReviewSamples: (limit?: number) => loadReviewSamples(limit),
loadScannerLearningRules: () => loadScannerLearningRules(),
writeScannerLearningRules: (rules: ScannerLearningRulePayload) => writeScannerLearningRules(rules),
exportGood: (exportPayload: GoodDatabase) => exportGood(exportPayload),
importGoodFile: () => importGoodFile(),
exportGood: (exportPayload: GoodDatabase) => getGoodFileService().exportGood(exportPayload),
importGoodFile: () => getGoodFileService().importGoodFile(getAppWindowManager().getMainWindow()),
listSources: () => listCaptureSources(),
captureSource: (
id: string,
@@ -1535,7 +1475,7 @@ function initializeAppLifecycle() {
});
app.on("activate", () => {
if (!mainWindow || mainWindow.isDestroyed()) {
if (!getAppWindowManager().hasMainWindow()) {
createMainWindow();
}
});
electron/repositories/scannerLearningRepository.ts
+57 -8
View File
@@ -16,22 +16,71 @@ export class ScannerLearningRepository implements ScannerLearningRepositoryPort
return {
ok: true,
path: this.filePath,
rules: parsed && typeof parsed === "object" ? parsed : { textReplacements: {} },
rules: parsed && typeof parsed === "object" ? parsed : emptyRules(),
};
} catch {
return { ok: true, path: this.filePath, rules: { textReplacements: {} } };
return { ok: true, path: this.filePath, rules: emptyRules() };
}
}
async save(rules: ScannerLearningRules): Promise<ScannerLearningSaveResult> {
const current = await this.load();
const nextTextReplacements = {
...((current.rules as { textReplacements?: Record<string, string> })?.textReplacements ?? {}),
...((rules as { textReplacements?: Record<string, string> })?.textReplacements ?? {}),
};
const payload: ScannerLearningRules = { textReplacements: nextTextReplacements };
const payload: ScannerLearningRules = mergeRules(current.rules, rules);
await fs.mkdir(path.dirname(this.filePath), { recursive: true });
await fs.writeFile(this.filePath, JSON.stringify(payload, null, 2), "utf8");
return { ok: true, path: this.filePath, rules: payload, total: Object.keys(nextTextReplacements).length };
return { ok: true, path: this.filePath, rules: payload, total: countRules(payload) };
}
}
function emptyRules(): ScannerLearningRules {
return {
textReplacements: {},
fieldAliases: {},
constrainedFixes: {},
cropAdjustments: {},
uiProfileAdjustments: {},
};
}
function mergeRules(current: ScannerLearningRules, incoming: ScannerLearningRules): ScannerLearningRules {
return {
textReplacements: {
...(current.textReplacements ?? {}),
...(incoming.textReplacements ?? {}),
},
fieldAliases: mergeNested(current.fieldAliases, incoming.fieldAliases),
constrainedFixes: {
...(current.constrainedFixes ?? {}),
...(incoming.constrainedFixes ?? {}),
},
cropAdjustments: {
...(current.cropAdjustments ?? {}),
...(incoming.cropAdjustments ?? {}),
},
uiProfileAdjustments: {
...(current.uiProfileAdjustments ?? {}),
...(incoming.uiProfileAdjustments ?? {}),
},
};
}
function mergeNested(
current: Record<string, Record<string, string>> | undefined,
incoming: Record<string, Record<string, string>> | undefined,
) {
const merged: Record<string, Record<string, string>> = {};
for (const [field, values] of Object.entries(current ?? {})) merged[field] = { ...(values ?? {}) };
for (const [field, values] of Object.entries(incoming ?? {})) merged[field] = { ...(merged[field] ?? {}), ...(values ?? {}) };
return merged;
}
function countRules(rules: ScannerLearningRules) {
const fieldAliases = Object.values(rules.fieldAliases ?? {}).reduce((sum, aliases) => sum + Object.keys(aliases ?? {}).length, 0);
return (
Object.keys(rules.textReplacements ?? {}).length
+ fieldAliases
+ Object.keys(rules.constrainedFixes ?? {}).length
+ Object.keys(rules.cropAdjustments ?? {}).length
+ Object.keys(rules.uiProfileAdjustments ?? {}).length
);
}
electron/services/goodFileService.ts
+57
View File
@@ -0,0 +1,57 @@
import { dialog, type BrowserWindow } from "electron";
import fs from "node:fs/promises";
import path from "node:path";
import type { GoodDatabase, GoodImportFileResult, SaveResultWithPath } from "../../src/types/global.js";
export interface GoodFileService {
exportGood: (payload: GoodDatabase) => Promise<SaveResultWithPath>;
importGoodFile: (parentWindow?: BrowserWindow | null) => Promise<GoodImportFileResult>;
}
export function createGoodFileService(exportDirectory: string): GoodFileService {
function exportPath(fileName: string) {
return path.join(exportDirectory, fileName);
}
async function exportGood(payload: GoodDatabase): Promise<SaveResultWithPath> {
const fileNameSafe = `good-export-${new Date().toISOString().replace(/[\\/:]/g, "-").replace(/\..+?$/, "").replace(/\s+/g, "-")}.json`;
const filePath = exportPath(fileNameSafe);
try {
await fs.mkdir(path.dirname(filePath), { recursive: true });
await fs.writeFile(filePath, JSON.stringify(payload, null, 2), "utf8");
return { ok: true, path: filePath };
} catch {
return { ok: false, path: filePath };
}
}
async function importGoodFile(parentWindow?: BrowserWindow | null): Promise<GoodImportFileResult> {
const dialogOptions = {
title: "GOOD-Datei importieren",
properties: ["openFile"],
filters: [{ name: "GOOD JSON", extensions: ["json"] }],
} satisfies Electron.OpenDialogOptions;
const dialogResult = parentWindow && !parentWindow.isDestroyed()
? await dialog.showOpenDialog(parentWindow, dialogOptions)
: await dialog.showOpenDialog(dialogOptions);
if (dialogResult.canceled || dialogResult.filePaths.length === 0) {
return { ok: false, canceled: true, path: "" };
}
const filePath = dialogResult.filePaths[0];
try {
const text = await fs.readFile(filePath, "utf8");
return { ok: true, canceled: false, path: filePath, database: JSON.parse(text) };
} catch (error) {
return {
ok: false,
canceled: false,
path: filePath,
error: error instanceof Error ? error.message : String(error),
};
}
}
return { exportGood, importGoodFile };
}
electron/services/inputHelper.ts
+1 -438
View File
@@ -13,444 +13,7 @@ import type {
ScrollResult,
} from "../../src/types/global.js";
const INPUT_HELPER_SCRIPT = String.raw`
$ErrorActionPreference = "Stop"
Add-Type -AssemblyName System.Drawing
Add-Type -AssemblyName System.Windows.Forms
$signature = @"
[DllImport("user32.dll")]
public static extern bool SetProcessDPIAware();
[DllImport("shcore.dll")]
public static extern int SetProcessDpiAwareness(int value);
[DllImport("user32.dll")]
public static extern bool SetCursorPos(int X, int Y);
[DllImport("user32.dll")]
public static extern bool GetCursorPos(out POINT lpPoint);
[DllImport("user32.dll", SetLastError=true)]
public static extern bool GetClientRect(IntPtr hWnd, out RECT lpRect);
[DllImport("user32.dll", SetLastError=true)]
public static extern bool ClientToScreen(IntPtr hWnd, ref POINT lpPoint);
[DllImport("user32.dll")]
public static extern short GetAsyncKeyState(int vKey);
[DllImport("user32.dll", SetLastError=true)]
public static extern uint SendInput(uint nInputs, INPUT[] pInputs, int cbSize);
[DllImport("user32.dll")]
public static extern bool SetForegroundWindow(IntPtr hWnd);
[DllImport("user32.dll")]
public static extern bool ShowWindowAsync(IntPtr hWnd, int nCmdShow);
[DllImport("user32.dll")]
public static extern bool BringWindowToTop(IntPtr hWnd);
[DllImport("user32.dll", SetLastError=true)]
public static extern bool AttachThreadInput(uint idAttach, uint idAttachTo, bool fAttach);
[DllImport("kernel32.dll")]
public static extern uint GetCurrentThreadId();
[DllImport("user32.dll", SetLastError=true, EntryPoint="SystemParametersInfoW")]
public static extern bool SystemParametersInfoGet(uint uiAction, uint uiParam, ref uint pvParam, uint fWinIni);
[DllImport("user32.dll", SetLastError=true, EntryPoint="SystemParametersInfoW")]
public static extern bool SystemParametersInfoSet(uint uiAction, uint uiParam, IntPtr pvParam, uint fWinIni);
[DllImport("user32.dll")]
public static extern IntPtr GetForegroundWindow();
[DllImport("user32.dll")]
public static extern bool IsWindow(IntPtr hWnd);
[DllImport("user32.dll")]
public static extern uint GetWindowThreadProcessId(IntPtr hWnd, out uint lpdwProcessId);
[StructLayout(LayoutKind.Sequential)]
public struct POINT { public int X; public int Y; }
[StructLayout(LayoutKind.Sequential)]
public struct RECT { public int Left; public int Top; public int Right; public int Bottom; }
[StructLayout(LayoutKind.Sequential)]
public struct MOUSEINPUT { public int dx; public int dy; public uint mouseData; public uint dwFlags; public uint time; public UIntPtr dwExtraInfo; }
[StructLayout(LayoutKind.Sequential)]
public struct INPUT { public int type; public MOUSEINPUT mi; }
"@
Add-Type -MemberDefinition $signature -Name InputHelper -Namespace Native
# Per-monitor DPI awareness (matches GenshinArtScanner's proven fix for the
# same symptom): the older SetProcessDPIAware() only applies a single,
# system-wide scale factor. On a mixed-DPI multi-monitor setup (e.g. Genshin
# on one display, this app's window on a differently-scaled second display),
# that single scale factor is wrong for whichever monitor didn't set it,
# silently shifting every SetCursorPos/click coordinate off-target even
# though cursor readback still matches what we asked for (both go through the
# same, wrong, virtualization layer). PROCESS_PER_MONITOR_DPI_AWARE = 2.
try {
[Native.InputHelper]::SetProcessDpiAwareness(2) | Out-Null
} catch {
[Native.InputHelper]::SetProcessDPIAware() | Out-Null
}
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
# SizeOf must receive a struct instance: passing the type object throws in
# Windows PowerShell 5.1 (RuntimeType cannot be marshalled).
$inputSize = [Runtime.InteropServices.Marshal]::SizeOf((New-Object Native.InputHelper+INPUT))
$genshinHwnd = [IntPtr]::Zero
function Send-MouseInput {
param([uint32]$flags, [int]$dx = 0, [int]$dy = 0, [long]$wheelData = 0)
$mouseInput = New-Object Native.InputHelper+INPUT
$mouseInput.type = 0
$mouseInput.mi.dx = $dx
$mouseInput.mi.dy = $dy
if ($wheelData -lt 0) { $mouseInput.mi.mouseData = [uint32](4294967296 + $wheelData) } else { $mouseInput.mi.mouseData = [uint32]$wheelData }
$mouseInput.mi.dwFlags = $flags
return [Native.InputHelper]::SendInput(1, [Native.InputHelper+INPUT[]]@($mouseInput), $inputSize)
}
# Matches Inventory Kamera exactly (see docs/DECISIONS.md ADR-008): it moves
# with bare SetCursorPos, then clicks via the InputSimulator library's
# Mouse.LeftButtonClick(), which sends button-down and button-up as ONE
# SendInput call (two INPUT structs in the same array) - back-to-back with no
# artificial delay between them, unlike two separate SendInput calls with a
# Start-Sleep in between. Returns the number of injected events (2 = ok).
function Send-MouseClickBatch {
$down = New-Object Native.InputHelper+INPUT
$down.type = 0
$down.mi.dwFlags = 0x0002
$up = New-Object Native.InputHelper+INPUT
$up.type = 0
$up.mi.dwFlags = 0x0004
return [Native.InputHelper]::SendInput(2, [Native.InputHelper+INPUT[]]@($down, $up), $inputSize)
}
function Send-KeyPressBatch {
param([int]$virtualKey)
$down = New-Object Native.InputHelper+INPUT
$down.type = 1
$down.mi.dx = $virtualKey
$up = New-Object Native.InputHelper+INPUT
$up.type = 1
$up.mi.dx = $virtualKey
# Same union bytes as KEYBDINPUT: dx low word = wVk, dy = dwFlags.
$up.mi.dy = 0x0002
return [Native.InputHelper]::SendInput(2, [Native.InputHelper+INPUT[]]@($down, $up), $inputSize)
}
function Resolve-VirtualKey {
param([string]$key)
switch ($key.ToUpperInvariant()) {
"ESC" { return 27 }
"ESCAPE" { return 27 }
"ENTER" { return 13 }
"B" { return 66 }
"C" { return 67 }
"1" { return 49 }
default { throw "unsupported key: $key" }
}
}
function Get-CursorPoint {
$pt = New-Object Native.InputHelper+POINT
[Native.InputHelper]::GetCursorPos([ref]$pt) | Out-Null
return $pt
}
function Get-ProcessNameFromHwnd {
param([IntPtr]$hwnd)
if ($hwnd -eq [IntPtr]::Zero) { return "" }
$pidValue = [uint32]0
[Native.InputHelper]::GetWindowThreadProcessId($hwnd, [ref]$pidValue) | Out-Null
if ($pidValue -eq 0) { return "" }
try {
return (Get-Process -Id ([int]$pidValue) -ErrorAction Stop).ProcessName
} catch {
return ""
}
}
function Get-CurrentProcessElevation {
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
return $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
}
function Get-ForegroundInfo {
$hwnd = [Native.InputHelper]::GetForegroundWindow()
return @{
foregroundHwnd = $hwnd.ToInt64()
foregroundProcess = Get-ProcessNameFromHwnd -hwnd $hwnd
}
}
function Get-CursorState {
$pt = New-Object Native.InputHelper+POINT
[Native.InputHelper]::GetCursorPos([ref]$pt) | Out-Null
# Only 0x8000 (key is held down right now). The 0x0001 "pressed since last
# call" bit is unreliable and fires for ESC presses that happened long
# before the scan (ESC is used constantly to navigate Genshin menus).
$esc = ([Native.InputHelper]::GetAsyncKeyState(27) -band 0x8000) -ne 0
$enter = ([Native.InputHelper]::GetAsyncKeyState(13) -band 0x8000) -ne 0
$f9 = ([Native.InputHelper]::GetAsyncKeyState(120) -band 0x8000) -ne 0
return @{ cursorX = $pt.X; cursorY = $pt.Y; escapePressed = $esc; enterPressed = $enter; f9Pressed = $f9 }
}
function Get-GenshinClientBounds {
$hwnd = Find-GenshinWindow
if ($hwnd -eq [IntPtr]::Zero) { return $null }
$rect = New-Object Native.InputHelper+RECT
if (-not [Native.InputHelper]::GetClientRect($hwnd, [ref]$rect)) { return $null }
$topLeft = New-Object Native.InputHelper+POINT
$topLeft.X = 0
$topLeft.Y = 0
if (-not [Native.InputHelper]::ClientToScreen($hwnd, [ref]$topLeft)) { return $null }
$width = $rect.Right - $rect.Left
$height = $rect.Bottom - $rect.Top
if ($width -le 0 -or $height -le 0) { return $null }
return @{
Left = $topLeft.X
Top = $topLeft.Y
Width = $width
Height = $height
}
}
function Find-GenshinWindow {
if ($script:genshinHwnd -ne [IntPtr]::Zero -and [Native.InputHelper]::IsWindow($script:genshinHwnd)) { return $script:genshinHwnd }
$proc = Get-Process | Where-Object { $_.ProcessName -match 'GenshinImpact|YuanShen|Genshin' -and $_.MainWindowHandle -ne 0 } | Select-Object -First 1
if ($proc) { $script:genshinHwnd = $proc.MainWindowHandle } else { $script:genshinHwnd = [IntPtr]::Zero }
return $script:genshinHwnd
}
# Plain SetForegroundWindow from this background helper process is silently
# refused by Windows' foreground lock. Attach our thread's input queue to the
# target (and current foreground) window thread and clear the lock timeout, so
# the foreground change is honored - the same technique Inventory Kamera uses.
function Force-Foreground {
param([IntPtr]$hwnd)
$current = [Native.InputHelper]::GetCurrentThreadId()
$targetPid = [uint32]0
$target = [Native.InputHelper]::GetWindowThreadProcessId($hwnd, [ref]$targetPid)
$fgWindow = [Native.InputHelper]::GetForegroundWindow()
$foreground = [uint32]0
if ($fgWindow -ne [IntPtr]::Zero) {
$fgPid = [uint32]0
$foreground = [Native.InputHelper]::GetWindowThreadProcessId($fgWindow, [ref]$fgPid)
}
$attachedTarget = $false
$attachedForeground = $false
$oldTimeout = [uint32]0
$timeoutRead = $false
try {
if ($target -ne 0 -and $target -ne $current) { $attachedTarget = [Native.InputHelper]::AttachThreadInput($current, $target, $true) }
if ($foreground -ne 0 -and $foreground -ne $current -and $foreground -ne $target) { $attachedForeground = [Native.InputHelper]::AttachThreadInput($current, $foreground, $true) }
$timeoutRead = [Native.InputHelper]::SystemParametersInfoGet(0x2000, 0, [ref]$oldTimeout, 0)
[Native.InputHelper]::SystemParametersInfoSet(0x2001, 0, [IntPtr]::Zero, 0x0002) | Out-Null
# Inject a no-op input (0,0 mouse move) so this process is the last input
# source, which Windows requires before it will honor a foreground change.
Send-MouseInput -flags 0x0001 | Out-Null
[Native.InputHelper]::ShowWindowAsync($hwnd, 9) | Out-Null
[Native.InputHelper]::BringWindowToTop($hwnd) | Out-Null
return [Native.InputHelper]::SetForegroundWindow($hwnd)
} finally {
if ($timeoutRead) { [Native.InputHelper]::SystemParametersInfoSet(0x2001, 0, [IntPtr]::new([int64]$oldTimeout), 0x0002) | Out-Null }
if ($attachedForeground) { [Native.InputHelper]::AttachThreadInput($current, $foreground, $false) | Out-Null }
if ($attachedTarget) { [Native.InputHelper]::AttachThreadInput($current, $target, $false) | Out-Null }
}
}
function Focus-GenshinWindow {
$hwnd = Find-GenshinWindow
$info = @{
hwnd = $hwnd.ToInt64()
focused = $false
alreadyForeground = $false
foregroundProcess = ""
targetProcess = Get-ProcessNameFromHwnd -hwnd $hwnd
}
if ($hwnd -eq [IntPtr]::Zero) { return $info }
$info.alreadyForeground = ([Native.InputHelper]::GetForegroundWindow() -eq $hwnd)
if (-not $info.alreadyForeground) {
$info.setForegroundResult = Force-Foreground -hwnd $hwnd
Start-Sleep -Milliseconds 140
}
$foreground = [Native.InputHelper]::GetForegroundWindow()
$info.focused = ($foreground -eq $hwnd)
$info.foregroundProcess = Get-ProcessNameFromHwnd -hwnd $foreground
return $info
}
while ($true) {
$line = [Console]::In.ReadLine()
if ($null -eq $line) { break }
if ($line.Trim().Length -eq 0) { continue }
$response = @{ id = ""; ok = $true }
try {
$cmd = $line | ConvertFrom-Json
$response.id = "$($cmd.id)"
switch ("$($cmd.op)") {
"ping" {
$response.pong = $true
}
"cursor" {
$state = Get-CursorState
$response.cursorX = $state.cursorX
$response.cursorY = $state.cursorY
$response.escapePressed = $state.escapePressed
$response.enterPressed = $state.enterPressed
$response.f9Pressed = $state.f9Pressed
}
"runtime" {
$response.isElevated = Get-CurrentProcessElevation
$hwnd = Find-GenshinWindow
$foregroundInfo = Get-ForegroundInfo
$response.genshinFound = ($hwnd -ne [IntPtr]::Zero)
$response.genshinHwnd = $hwnd.ToInt64()
$response.targetProcess = Get-ProcessNameFromHwnd -hwnd $hwnd
$response.foregroundProcess = $foregroundInfo.foregroundProcess
$response.foregroundHwnd = $foregroundInfo.foregroundHwnd
$response.helperPid = $PID
}
"focus" {
$focusInfo = Focus-GenshinWindow
$response.focused = $focusInfo.focused
$response.alreadyForeground = $focusInfo.alreadyForeground
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.genshinFound = ($focusInfo.hwnd -ne 0)
$response.setForegroundResult = $focusInfo.setForegroundResult
}
"click" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
$targetX = [int]$cmd.x
$targetY = [int]$cmd.y
# Matches Inventory Kamera's verified-working sequence exactly: bare
# SetCursorPos immediately followed by a click, with NO extra move
# event and NO artificial delay between moving and clicking - IK's
# Navigation.Click(x, y) does SetCursor() then Click() back-to-back,
# zero gap. Settling delays only happen after the click, in the scan
# loop. Down+up are sent as one SendInput call (see
# Send-MouseClickBatch), matching InputSimulator.Mouse.LeftButtonClick().
[Native.InputHelper]::SetCursorPos($targetX, $targetY) | Out-Null
$point = Get-CursorPoint
$onTarget = (([Math]::Abs($targetX - $point.X) -le 2) -and ([Math]::Abs($targetY - $point.Y) -le 2))
$clickEventsSent = 0
if ($onTarget) {
$clickEventsSent = Send-MouseClickBatch
}
$state = Get-CursorState
$response.cursorX = $state.cursorX
$response.cursorY = $state.cursorY
$response.escapePressed = $state.escapePressed
$response.enterPressed = $state.enterPressed
$response.f9Pressed = $state.f9Pressed
$response.moved = $onTarget
$response.focused = $focusInfo.focused
$response.alreadyForeground = $focusInfo.alreadyForeground
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.isElevated = Get-CurrentProcessElevation
# Never report a click unless the cursor is verifiably on the target.
# Real acceptance is proven later by the detail-panel fingerprint.
$response.clicked = ($onTarget -and $clickEventsSent -ge 2)
$response.inputBlocked = ($onTarget -and $clickEventsSent -lt 2)
}
"scroll" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
if ($null -ne $cmd.x -and $null -ne $cmd.y) {
[Native.InputHelper]::SetCursorPos([int]$cmd.x, [int]$cmd.y) | Out-Null
Start-Sleep -Milliseconds 30
}
$point = Get-CursorPoint
$response.cursorX = $point.X
$response.cursorY = $point.Y
$response.focused = $focusInfo.focused
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.isElevated = Get-CurrentProcessElevation
$notches = [int]$cmd.notches
$stepDelta = 120
if ($notches -lt 0) { $stepDelta = -120 }
$count = [Math]::Abs($notches)
if ($count -gt 60) { $count = 60 }
$sentTotal = 0
for ($i = 0; $i -lt $count; $i++) {
$sentTotal += Send-MouseInput -flags 0x0800 -wheelData $stepDelta
Start-Sleep -Milliseconds 45
}
$response.notchesSent = $sentTotal
$response.inputBlocked = (($count -gt 0) -and ($sentTotal -eq 0))
}
"key" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
$vk = Resolve-VirtualKey -key "$($cmd.key)"
$sent = Send-KeyPressBatch -virtualKey $vk
$response.key = "$($cmd.key)"
$response.focused = $focusInfo.focused
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.isElevated = Get-CurrentProcessElevation
$response.eventsSent = $sent
$response.inputBlocked = ($sent -lt 2)
}
"bounds" {
$clientBounds = Get-GenshinClientBounds
if ($null -eq $clientBounds) {
$response.found = $false
} else {
$response.found = $true
$response.left = $clientBounds.Left
$response.top = $clientBounds.Top
$response.width = $clientBounds.Width
$response.height = $clientBounds.Height
}
}
"capture" {
$clientBounds = Get-GenshinClientBounds
if ($null -eq $clientBounds) {
$screenBounds = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds
$clientBounds = @{
Left = $screenBounds.Left
Top = $screenBounds.Top
Width = $screenBounds.Width
Height = $screenBounds.Height
}
$response.captureTarget = "primary-screen"
} else {
$response.captureTarget = "genshin-client"
}
$bitmap = New-Object System.Drawing.Bitmap $clientBounds.Width, $clientBounds.Height
$graphics = [System.Drawing.Graphics]::FromImage($bitmap)
$graphics.CopyFromScreen($clientBounds.Left, $clientBounds.Top, 0, 0, $bitmap.Size)
$capturePath = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), "genshin-assistant-capture-" + [Guid]::NewGuid().ToString() + ".png")
$bitmap.Save($capturePath, [System.Drawing.Imaging.ImageFormat]::Png)
$graphics.Dispose()
$bitmap.Dispose()
$response.path = $capturePath
$response.width = $clientBounds.Width
$response.height = $clientBounds.Height
$response.originX = $clientBounds.Left
$response.originY = $clientBounds.Top
}
default {
$response.ok = $false
$response.error = "unknown op"
}
}
} catch {
$response.ok = $false
$response.error = $_.Exception.Message
}
Write-Output (ConvertTo-Json $response -Compress)
}
`;
import { INPUT_HELPER_SCRIPT } from "./inputHelperPowerShellFallback.js";
class InputHelperClient {
private child: ChildProcessWithoutNullStreams | null = null;
electron/services/inputHelperPowerShellFallback.ts
+441
View File
@@ -0,0 +1,441 @@
// PowerShell fallback for environments where the compiled C# sidecar is unavailable. Keep the JSON protocol aligned with native/input-helper/Program.cs.
export const INPUT_HELPER_SCRIPT = String.raw`
$ErrorActionPreference = "Stop"
Add-Type -AssemblyName System.Drawing
Add-Type -AssemblyName System.Windows.Forms
$signature = @"
[DllImport("user32.dll")]
public static extern bool SetProcessDPIAware();
[DllImport("shcore.dll")]
public static extern int SetProcessDpiAwareness(int value);
[DllImport("user32.dll")]
public static extern bool SetCursorPos(int X, int Y);
[DllImport("user32.dll")]
public static extern bool GetCursorPos(out POINT lpPoint);
[DllImport("user32.dll", SetLastError=true)]
public static extern bool GetClientRect(IntPtr hWnd, out RECT lpRect);
[DllImport("user32.dll", SetLastError=true)]
public static extern bool ClientToScreen(IntPtr hWnd, ref POINT lpPoint);
[DllImport("user32.dll")]
public static extern short GetAsyncKeyState(int vKey);
[DllImport("user32.dll", SetLastError=true)]
public static extern uint SendInput(uint nInputs, INPUT[] pInputs, int cbSize);
[DllImport("user32.dll")]
public static extern bool SetForegroundWindow(IntPtr hWnd);
[DllImport("user32.dll")]
public static extern bool ShowWindowAsync(IntPtr hWnd, int nCmdShow);
[DllImport("user32.dll")]
public static extern bool BringWindowToTop(IntPtr hWnd);
[DllImport("user32.dll", SetLastError=true)]
public static extern bool AttachThreadInput(uint idAttach, uint idAttachTo, bool fAttach);
[DllImport("kernel32.dll")]
public static extern uint GetCurrentThreadId();
[DllImport("user32.dll", SetLastError=true, EntryPoint="SystemParametersInfoW")]
public static extern bool SystemParametersInfoGet(uint uiAction, uint uiParam, ref uint pvParam, uint fWinIni);
[DllImport("user32.dll", SetLastError=true, EntryPoint="SystemParametersInfoW")]
public static extern bool SystemParametersInfoSet(uint uiAction, uint uiParam, IntPtr pvParam, uint fWinIni);
[DllImport("user32.dll")]
public static extern IntPtr GetForegroundWindow();
[DllImport("user32.dll")]
public static extern bool IsWindow(IntPtr hWnd);
[DllImport("user32.dll")]
public static extern uint GetWindowThreadProcessId(IntPtr hWnd, out uint lpdwProcessId);
[StructLayout(LayoutKind.Sequential)]
public struct POINT { public int X; public int Y; }
[StructLayout(LayoutKind.Sequential)]
public struct RECT { public int Left; public int Top; public int Right; public int Bottom; }
[StructLayout(LayoutKind.Sequential)]
public struct MOUSEINPUT { public int dx; public int dy; public uint mouseData; public uint dwFlags; public uint time; public UIntPtr dwExtraInfo; }
[StructLayout(LayoutKind.Sequential)]
public struct INPUT { public int type; public MOUSEINPUT mi; }
"@
Add-Type -MemberDefinition $signature -Name InputHelper -Namespace Native
# Per-monitor DPI awareness (matches GenshinArtScanner's proven fix for the
# same symptom): the older SetProcessDPIAware() only applies a single,
# system-wide scale factor. On a mixed-DPI multi-monitor setup (e.g. Genshin
# on one display, this app's window on a differently-scaled second display),
# that single scale factor is wrong for whichever monitor didn't set it,
# silently shifting every SetCursorPos/click coordinate off-target even
# though cursor readback still matches what we asked for (both go through the
# same, wrong, virtualization layer). PROCESS_PER_MONITOR_DPI_AWARE = 2.
try {
[Native.InputHelper]::SetProcessDpiAwareness(2) | Out-Null
} catch {
[Native.InputHelper]::SetProcessDPIAware() | Out-Null
}
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
# SizeOf must receive a struct instance: passing the type object throws in
# Windows PowerShell 5.1 (RuntimeType cannot be marshalled).
$inputSize = [Runtime.InteropServices.Marshal]::SizeOf((New-Object Native.InputHelper+INPUT))
$genshinHwnd = [IntPtr]::Zero
function Send-MouseInput {
param([uint32]$flags, [int]$dx = 0, [int]$dy = 0, [long]$wheelData = 0)
$mouseInput = New-Object Native.InputHelper+INPUT
$mouseInput.type = 0
$mouseInput.mi.dx = $dx
$mouseInput.mi.dy = $dy
if ($wheelData -lt 0) { $mouseInput.mi.mouseData = [uint32](4294967296 + $wheelData) } else { $mouseInput.mi.mouseData = [uint32]$wheelData }
$mouseInput.mi.dwFlags = $flags
return [Native.InputHelper]::SendInput(1, [Native.InputHelper+INPUT[]]@($mouseInput), $inputSize)
}
# Matches Inventory Kamera exactly (see docs/DECISIONS.md ADR-008): it moves
# with bare SetCursorPos, then clicks via the InputSimulator library's
# Mouse.LeftButtonClick(), which sends button-down and button-up as ONE
# SendInput call (two INPUT structs in the same array) - back-to-back with no
# artificial delay between them, unlike two separate SendInput calls with a
# Start-Sleep in between. Returns the number of injected events (2 = ok).
function Send-MouseClickBatch {
$down = New-Object Native.InputHelper+INPUT
$down.type = 0
$down.mi.dwFlags = 0x0002
$up = New-Object Native.InputHelper+INPUT
$up.type = 0
$up.mi.dwFlags = 0x0004
return [Native.InputHelper]::SendInput(2, [Native.InputHelper+INPUT[]]@($down, $up), $inputSize)
}
function Send-KeyPressBatch {
param([int]$virtualKey)
$down = New-Object Native.InputHelper+INPUT
$down.type = 1
$down.mi.dx = $virtualKey
$up = New-Object Native.InputHelper+INPUT
$up.type = 1
$up.mi.dx = $virtualKey
# Same union bytes as KEYBDINPUT: dx low word = wVk, dy = dwFlags.
$up.mi.dy = 0x0002
return [Native.InputHelper]::SendInput(2, [Native.InputHelper+INPUT[]]@($down, $up), $inputSize)
}
function Resolve-VirtualKey {
param([string]$key)
switch ($key.ToUpperInvariant()) {
"ESC" { return 27 }
"ESCAPE" { return 27 }
"ENTER" { return 13 }
"B" { return 66 }
"C" { return 67 }
"1" { return 49 }
default { throw "unsupported key: $key" }
}
}
function Get-CursorPoint {
$pt = New-Object Native.InputHelper+POINT
[Native.InputHelper]::GetCursorPos([ref]$pt) | Out-Null
return $pt
}
function Get-ProcessNameFromHwnd {
param([IntPtr]$hwnd)
if ($hwnd -eq [IntPtr]::Zero) { return "" }
$pidValue = [uint32]0
[Native.InputHelper]::GetWindowThreadProcessId($hwnd, [ref]$pidValue) | Out-Null
if ($pidValue -eq 0) { return "" }
try {
return (Get-Process -Id ([int]$pidValue) -ErrorAction Stop).ProcessName
} catch {
return ""
}
}
function Get-CurrentProcessElevation {
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
return $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
}
function Get-ForegroundInfo {
$hwnd = [Native.InputHelper]::GetForegroundWindow()
return @{
foregroundHwnd = $hwnd.ToInt64()
foregroundProcess = Get-ProcessNameFromHwnd -hwnd $hwnd
}
}
function Get-CursorState {
$pt = New-Object Native.InputHelper+POINT
[Native.InputHelper]::GetCursorPos([ref]$pt) | Out-Null
# Only 0x8000 (key is held down right now). The 0x0001 "pressed since last
# call" bit is unreliable and fires for ESC presses that happened long
# before the scan (ESC is used constantly to navigate Genshin menus).
$esc = ([Native.InputHelper]::GetAsyncKeyState(27) -band 0x8000) -ne 0
$enter = ([Native.InputHelper]::GetAsyncKeyState(13) -band 0x8000) -ne 0
$f9 = ([Native.InputHelper]::GetAsyncKeyState(120) -band 0x8000) -ne 0
return @{ cursorX = $pt.X; cursorY = $pt.Y; escapePressed = $esc; enterPressed = $enter; f9Pressed = $f9 }
}
function Get-GenshinClientBounds {
$hwnd = Find-GenshinWindow
if ($hwnd -eq [IntPtr]::Zero) { return $null }
$rect = New-Object Native.InputHelper+RECT
if (-not [Native.InputHelper]::GetClientRect($hwnd, [ref]$rect)) { return $null }
$topLeft = New-Object Native.InputHelper+POINT
$topLeft.X = 0
$topLeft.Y = 0
if (-not [Native.InputHelper]::ClientToScreen($hwnd, [ref]$topLeft)) { return $null }
$width = $rect.Right - $rect.Left
$height = $rect.Bottom - $rect.Top
if ($width -le 0 -or $height -le 0) { return $null }
return @{
Left = $topLeft.X
Top = $topLeft.Y
Width = $width
Height = $height
}
}
function Find-GenshinWindow {
if ($script:genshinHwnd -ne [IntPtr]::Zero -and [Native.InputHelper]::IsWindow($script:genshinHwnd)) { return $script:genshinHwnd }
$proc = Get-Process | Where-Object { $_.ProcessName -match 'GenshinImpact|YuanShen|Genshin' -and $_.MainWindowHandle -ne 0 } | Select-Object -First 1
if ($proc) { $script:genshinHwnd = $proc.MainWindowHandle } else { $script:genshinHwnd = [IntPtr]::Zero }
return $script:genshinHwnd
}
# Plain SetForegroundWindow from this background helper process is silently
# refused by Windows' foreground lock. Attach our thread's input queue to the
# target (and current foreground) window thread and clear the lock timeout, so
# the foreground change is honored - the same technique Inventory Kamera uses.
function Force-Foreground {
param([IntPtr]$hwnd)
$current = [Native.InputHelper]::GetCurrentThreadId()
$targetPid = [uint32]0
$target = [Native.InputHelper]::GetWindowThreadProcessId($hwnd, [ref]$targetPid)
$fgWindow = [Native.InputHelper]::GetForegroundWindow()
$foreground = [uint32]0
if ($fgWindow -ne [IntPtr]::Zero) {
$fgPid = [uint32]0
$foreground = [Native.InputHelper]::GetWindowThreadProcessId($fgWindow, [ref]$fgPid)
}
$attachedTarget = $false
$attachedForeground = $false
$oldTimeout = [uint32]0
$timeoutRead = $false
try {
if ($target -ne 0 -and $target -ne $current) { $attachedTarget = [Native.InputHelper]::AttachThreadInput($current, $target, $true) }
if ($foreground -ne 0 -and $foreground -ne $current -and $foreground -ne $target) { $attachedForeground = [Native.InputHelper]::AttachThreadInput($current, $foreground, $true) }
$timeoutRead = [Native.InputHelper]::SystemParametersInfoGet(0x2000, 0, [ref]$oldTimeout, 0)
[Native.InputHelper]::SystemParametersInfoSet(0x2001, 0, [IntPtr]::Zero, 0x0002) | Out-Null
# Inject a no-op input (0,0 mouse move) so this process is the last input
# source, which Windows requires before it will honor a foreground change.
Send-MouseInput -flags 0x0001 | Out-Null
[Native.InputHelper]::ShowWindowAsync($hwnd, 9) | Out-Null
[Native.InputHelper]::BringWindowToTop($hwnd) | Out-Null
return [Native.InputHelper]::SetForegroundWindow($hwnd)
} finally {
if ($timeoutRead) { [Native.InputHelper]::SystemParametersInfoSet(0x2001, 0, [IntPtr]::new([int64]$oldTimeout), 0x0002) | Out-Null }
if ($attachedForeground) { [Native.InputHelper]::AttachThreadInput($current, $foreground, $false) | Out-Null }
if ($attachedTarget) { [Native.InputHelper]::AttachThreadInput($current, $target, $false) | Out-Null }
}
}
function Focus-GenshinWindow {
$hwnd = Find-GenshinWindow
$info = @{
hwnd = $hwnd.ToInt64()
focused = $false
alreadyForeground = $false
foregroundProcess = ""
targetProcess = Get-ProcessNameFromHwnd -hwnd $hwnd
}
if ($hwnd -eq [IntPtr]::Zero) { return $info }
$info.alreadyForeground = ([Native.InputHelper]::GetForegroundWindow() -eq $hwnd)
if (-not $info.alreadyForeground) {
$info.setForegroundResult = Force-Foreground -hwnd $hwnd
Start-Sleep -Milliseconds 140
}
$foreground = [Native.InputHelper]::GetForegroundWindow()
$info.focused = ($foreground -eq $hwnd)
$info.foregroundProcess = Get-ProcessNameFromHwnd -hwnd $foreground
return $info
}
while ($true) {
$line = [Console]::In.ReadLine()
if ($null -eq $line) { break }
if ($line.Trim().Length -eq 0) { continue }
$response = @{ id = ""; ok = $true }
try {
$cmd = $line | ConvertFrom-Json
$response.id = "$($cmd.id)"
switch ("$($cmd.op)") {
"ping" {
$response.pong = $true
}
"cursor" {
$state = Get-CursorState
$response.cursorX = $state.cursorX
$response.cursorY = $state.cursorY
$response.escapePressed = $state.escapePressed
$response.enterPressed = $state.enterPressed
$response.f9Pressed = $state.f9Pressed
}
"runtime" {
$response.isElevated = Get-CurrentProcessElevation
$hwnd = Find-GenshinWindow
$foregroundInfo = Get-ForegroundInfo
$response.genshinFound = ($hwnd -ne [IntPtr]::Zero)
$response.genshinHwnd = $hwnd.ToInt64()
$response.targetProcess = Get-ProcessNameFromHwnd -hwnd $hwnd
$response.foregroundProcess = $foregroundInfo.foregroundProcess
$response.foregroundHwnd = $foregroundInfo.foregroundHwnd
$response.helperPid = $PID
}
"focus" {
$focusInfo = Focus-GenshinWindow
$response.focused = $focusInfo.focused
$response.alreadyForeground = $focusInfo.alreadyForeground
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.genshinFound = ($focusInfo.hwnd -ne 0)
$response.setForegroundResult = $focusInfo.setForegroundResult
}
"click" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
$targetX = [int]$cmd.x
$targetY = [int]$cmd.y
# Matches Inventory Kamera's verified-working sequence exactly: bare
# SetCursorPos immediately followed by a click, with NO extra move
# event and NO artificial delay between moving and clicking - IK's
# Navigation.Click(x, y) does SetCursor() then Click() back-to-back,
# zero gap. Settling delays only happen after the click, in the scan
# loop. Down+up are sent as one SendInput call (see
# Send-MouseClickBatch), matching InputSimulator.Mouse.LeftButtonClick().
[Native.InputHelper]::SetCursorPos($targetX, $targetY) | Out-Null
$point = Get-CursorPoint
$onTarget = (([Math]::Abs($targetX - $point.X) -le 2) -and ([Math]::Abs($targetY - $point.Y) -le 2))
$clickEventsSent = 0
if ($onTarget) {
$clickEventsSent = Send-MouseClickBatch
}
$state = Get-CursorState
$response.cursorX = $state.cursorX
$response.cursorY = $state.cursorY
$response.escapePressed = $state.escapePressed
$response.enterPressed = $state.enterPressed
$response.f9Pressed = $state.f9Pressed
$response.moved = $onTarget
$response.focused = $focusInfo.focused
$response.alreadyForeground = $focusInfo.alreadyForeground
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.isElevated = Get-CurrentProcessElevation
# Never report a click unless the cursor is verifiably on the target.
# Real acceptance is proven later by the detail-panel fingerprint.
$response.clicked = ($onTarget -and $clickEventsSent -ge 2)
$response.inputBlocked = ($onTarget -and $clickEventsSent -lt 2)
}
"scroll" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
if ($null -ne $cmd.x -and $null -ne $cmd.y) {
[Native.InputHelper]::SetCursorPos([int]$cmd.x, [int]$cmd.y) | Out-Null
Start-Sleep -Milliseconds 30
}
$point = Get-CursorPoint
$response.cursorX = $point.X
$response.cursorY = $point.Y
$response.focused = $focusInfo.focused
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.isElevated = Get-CurrentProcessElevation
$notches = [int]$cmd.notches
$stepDelta = 120
if ($notches -lt 0) { $stepDelta = -120 }
$count = [Math]::Abs($notches)
if ($count -gt 60) { $count = 60 }
$sentTotal = 0
for ($i = 0; $i -lt $count; $i++) {
$sentTotal += Send-MouseInput -flags 0x0800 -wheelData $stepDelta
Start-Sleep -Milliseconds 45
}
$response.notchesSent = $sentTotal
$response.inputBlocked = (($count -gt 0) -and ($sentTotal -eq 0))
}
"key" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
$vk = Resolve-VirtualKey -key "$($cmd.key)"
$sent = Send-KeyPressBatch -virtualKey $vk
$response.key = "$($cmd.key)"
$response.focused = $focusInfo.focused
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.isElevated = Get-CurrentProcessElevation
$response.eventsSent = $sent
$response.inputBlocked = ($sent -lt 2)
}
"bounds" {
$clientBounds = Get-GenshinClientBounds
if ($null -eq $clientBounds) {
$response.found = $false
} else {
$response.found = $true
$response.left = $clientBounds.Left
$response.top = $clientBounds.Top
$response.width = $clientBounds.Width
$response.height = $clientBounds.Height
}
}
"capture" {
$clientBounds = Get-GenshinClientBounds
if ($null -eq $clientBounds) {
$screenBounds = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds
$clientBounds = @{
Left = $screenBounds.Left
Top = $screenBounds.Top
Width = $screenBounds.Width
Height = $screenBounds.Height
}
$response.captureTarget = "primary-screen"
} else {
$response.captureTarget = "genshin-client"
}
$bitmap = New-Object System.Drawing.Bitmap $clientBounds.Width, $clientBounds.Height
$graphics = [System.Drawing.Graphics]::FromImage($bitmap)
$graphics.CopyFromScreen($clientBounds.Left, $clientBounds.Top, 0, 0, $bitmap.Size)
$capturePath = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), "genshin-assistant-capture-" + [Guid]::NewGuid().ToString() + ".png")
$bitmap.Save($capturePath, [System.Drawing.Imaging.ImageFormat]::Png)
$graphics.Dispose()
$bitmap.Dispose()
$response.path = $capturePath
$response.width = $clientBounds.Width
$response.height = $clientBounds.Height
$response.originX = $clientBounds.Left
$response.originY = $clientBounds.Top
}
default {
$response.ok = $false
$response.error = "unknown op"
}
}
} catch {
$response.ok = $false
$response.error = $_.Exception.Message
}
Write-Output (ConvertTo-Json $response -Compress)
}
`;
electron/services/pngBitmap.ts
+92
View File
@@ -0,0 +1,92 @@
import { inflateSync } from "node:zlib";
import type { Bitmap } from "../../src/lib/ocrPreprocess.js";
interface PngChunk {
type: string;
data: Buffer;
}
function readChunks(buffer: Buffer): PngChunk[] {
const signature = buffer.subarray(0, 8);
if (!signature.equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) {
throw new Error("Invalid PNG signature.");
}
const chunks: PngChunk[] = [];
let offset = 8;
while (offset + 12 <= buffer.length) {
const length = buffer.readUInt32BE(offset);
const type = buffer.toString("ascii", offset + 4, offset + 8);
const dataStart = offset + 8;
const dataEnd = dataStart + length;
if (dataEnd + 4 > buffer.length) throw new Error("Invalid PNG chunk length.");
chunks.push({ type, data: buffer.subarray(dataStart, dataEnd) });
offset = dataEnd + 4;
if (type === "IEND") break;
}
return chunks;
}
function paethPredictor(left: number, up: number, upperLeft: number) {
const estimate = left + up - upperLeft;
const leftDistance = Math.abs(estimate - left);
const upDistance = Math.abs(estimate - up);
const upperLeftDistance = Math.abs(estimate - upperLeft);
if (leftDistance <= upDistance && leftDistance <= upperLeftDistance) return left;
if (upDistance <= upperLeftDistance) return up;
return upperLeft;
}
function unfilterScanlines(raw: Buffer, width: number, height: number, bytesPerPixel: number) {
const stride = width * bytesPerPixel;
const output = Buffer.alloc(stride * height);
let rawOffset = 0;
for (let row = 0; row < height; row++) {
const filter = raw[rawOffset++];
const rowOffset = row * stride;
const previousRowOffset = rowOffset - stride;
for (let col = 0; col < stride; col++) {
const value = raw[rawOffset++];
const left = col >= bytesPerPixel ? output[rowOffset + col - bytesPerPixel] : 0;
const up = row > 0 ? output[previousRowOffset + col] : 0;
const upperLeft = row > 0 && col >= bytesPerPixel ? output[previousRowOffset + col - bytesPerPixel] : 0;
let restored = value;
if (filter === 1) restored = value + left;
else if (filter === 2) restored = value + up;
else if (filter === 3) restored = value + Math.floor((left + up) / 2);
else if (filter === 4) restored = value + paethPredictor(left, up, upperLeft);
else if (filter !== 0) throw new Error(`Unsupported PNG filter: ${filter}`);
output[rowOffset + col] = restored & 0xff;
}
}
return output;
}
export function pngBufferToBitmap(buffer: Buffer): Bitmap {
const chunks = readChunks(buffer);
const ihdr = chunks.find((chunk) => chunk.type === "IHDR")?.data;
if (!ihdr) throw new Error("PNG missing IHDR.");
const width = ihdr.readUInt32BE(0);
const height = ihdr.readUInt32BE(4);
const bitDepth = ihdr[8];
const colorType = ihdr[9];
const compression = ihdr[10];
const filter = ihdr[11];
const interlace = ihdr[12];
if (bitDepth !== 8 || compression !== 0 || filter !== 0 || interlace !== 0) {
throw new Error("Unsupported PNG format.");
}
const sourceBytesPerPixel = colorType === 6 ? 4 : colorType === 2 ? 3 : 0;
if (!sourceBytesPerPixel) throw new Error(`Unsupported PNG color type: ${colorType}`);
const idat = Buffer.concat(chunks.filter((chunk) => chunk.type === "IDAT").map((chunk) => chunk.data));
const unfiltered = unfilterScanlines(inflateSync(idat), width, height, sourceBytesPerPixel);
if (colorType === 6) return { data: unfiltered, width, height };
const rgba = Buffer.alloc(width * height * 4);
for (let pixel = 0; pixel < width * height; pixel++) {
rgba[pixel * 4] = unfiltered[pixel * 3];
rgba[pixel * 4 + 1] = unfiltered[pixel * 3 + 1];
rgba[pixel * 4 + 2] = unfiltered[pixel * 3 + 2];
rgba[pixel * 4 + 3] = 255;
}
return { data: rgba, width, height };
}
native/input-helper/Program.cs
+4 -4
View File
@@ -104,7 +104,7 @@ internal static class Program
case "click":
{
var info = FocusGenshinWindow();
var info = FocusGenshinWindow(includeProcessNames: false);
if (info.Focused && !info.AlreadyForeground) Thread.Sleep(120);
var targetX = GetInt(root, "x");
@@ -283,14 +283,14 @@ internal static class Program
return new CursorState { X = pt.X, Y = pt.Y, Escape = esc, Enter = enter, F9 = f9 };
}
private static FocusInfo FocusGenshinWindow()
private static FocusInfo FocusGenshinWindow(bool includeProcessNames = true)
{
var hwnd = FindGenshinWindow();
var info = new FocusInfo
{
Hwnd = hwnd,
ForegroundProcess = "",
TargetProcess = ProcessNameFromHwnd(hwnd),
TargetProcess = includeProcessNames ? ProcessNameFromHwnd(hwnd) : "",
};
if (hwnd == IntPtr.Zero) return info;
@@ -303,7 +303,7 @@ internal static class Program
var foreground = Native.GetForegroundWindow();
info.Focused = foreground == hwnd;
info.ForegroundProcess = ProcessNameFromHwnd(foreground);
info.ForegroundProcess = includeProcessNames ? ProcessNameFromHwnd(foreground) : "";
return info;
}
package.json
+10
View File
@@ -16,12 +16,22 @@
"lint": "tsc --noEmit",
"test": "vitest run",
"eval": "vitest run src/eval/ocrEval.test.ts",
"eval:review-candidates": "node scripts/export-review-eval-candidates.cjs",
"eval:prepare-confirmed": "node scripts/prepare-confirmed-review-case.cjs",
"scan:soak": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1",
"scan:goal": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun",
"scan:goal:current": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun -ScanEngine current",
"scan:goal:ik": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun -ScanEngine ik-traineddata",
"scan:goal:compare": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun -ScanEngine compare",
"scan:goal:compare:validated": "npm run scan:live:preflight && npm run scan:goal:compare && npm run scan:assessment:validate -- --latest --summary",
"scan:goal:compare:validated:wait": "npm run scan:live:preflight:wait && npm run scan:goal:compare && npm run scan:assessment:validate -- --latest --summary",
"scan:iterate:compare": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -Limits 20 -ScanEngine compare -BenchmarkOcr",
"scan:iterate:compare:validated": "npm run scan:live:preflight && npm run scan:iterate:compare && npm run scan:assessment:validate -- --latest --summary --limit=20",
"scan:iterate:compare:validated:wait": "npm run scan:live:preflight:wait && npm run scan:iterate:compare && npm run scan:assessment:validate -- --latest --summary --limit=20",
"scan:live:preflight": "node scripts/live-preflight.cjs",
"scan:live:preflight:wait": "node scripts/live-preflight.cjs --wait=120",
"scan:assessment:test": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -SelfTestAssessment",
"scan:assessment:validate": "node scripts/validate-scan-assessment.cjs",
"helper:build": "dotnet publish native/input-helper/InputHelper.csproj -c Release -o native/input-helper/bin/publish",
"data:genshin": "node scripts/generate-genshin-data.cjs"
},
scripts/dev-admin-start.ps1
+6 -1
View File
@@ -1,5 +1,6 @@
param(
[string]$ProjectRoot
[string]$ProjectRoot,
[string]$OcrWorkers = ""
)
# Mit -NoExit gestartet: dieses Fenster bleibt immer offen (siehe dev-admin.cmd),
@@ -20,6 +21,10 @@ try {
Write-Host "Projekt: $project"
Write-Host "Admin-Log: $logPath"
if ($OcrWorkers) {
$env:GAA_OCR_WORKERS = $OcrWorkers
Write-Host "GAA_OCR_WORKERS: $env:GAA_OCR_WORKERS"
}
# A UAC-elevated process gets its environment rebuilt fresh from the
# registry; it does NOT inherit PATH edits that only exist in the calling
scripts/dev-admin.ps1
+5 -1
View File
@@ -1,5 +1,6 @@
param(
[string]$ProjectRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path
[string]$ProjectRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path,
[string]$OcrWorkers = $env:GAA_OCR_WORKERS
)
$ErrorActionPreference = "Stop"
@@ -19,6 +20,9 @@ try {
"-File $(Quote-ProcessArgument $script)",
"-ProjectRoot $(Quote-ProcessArgument $project)"
) -join " "
if ($OcrWorkers) {
$arguments += " -OcrWorkers $(Quote-ProcessArgument $OcrWorkers)"
}
Start-Process -FilePath $powershellExe -ArgumentList $arguments -WorkingDirectory $project -Verb RunAs -WindowStyle Normal -ErrorAction Stop
scripts/export-review-eval-candidates.cjs
+288
View File
@@ -0,0 +1,288 @@
const fs = require("node:fs");
const path = require("node:path");
const readline = require("node:readline");
const os = require("node:os");
const DEFAULT_LIMIT = 80;
function argValue(name, fallback = "") {
const prefix = `--${name}=`;
const match = process.argv.find((entry) => entry.startsWith(prefix));
return match ? match.slice(prefix.length) : fallback;
}
function defaultReviewSamplesPath() {
const appData = process.env.APPDATA || path.join(os.homedir(), "AppData", "Roaming");
return path.join(appData, "genshin-artifact-assistant", "review-samples.jsonl");
}
function simplifyId(value) {
return String(value || "")
.replace(/[^0-9A-Za-z]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 80) || "unknown";
}
function parsedSummary(parsed) {
if (!parsed || typeof parsed !== "object") return {};
return {
name: parsed.name,
slot: parsed.slot,
level: parsed.level,
mainStat: parsed.mainStat,
mainValue: parsed.mainValue,
setName: parsed.setName,
equipped: parsed.equipped,
substats: Array.isArray(parsed.substats) ? parsed.substats : [],
confidence: parsed.confidence,
notes: Array.isArray(parsed.notes) ? parsed.notes : [],
};
}
const REQUIRED_FAST_OCR_FIELDS = [
"artifact-name",
"artifact-slot",
"artifact-main-stat-label",
"artifact-level",
"artifact-substats",
];
function ocrMap(record) {
const entries = record?.sample?.capture?.ocr;
if (!Array.isArray(entries) || entries.length === 0) return null;
const result = {};
for (const entry of entries) {
if (typeof entry?.id === "string" && typeof entry?.text === "string") {
result[entry.id] = entry.text;
}
}
return Object.keys(result).length ? result : null;
}
function candidateFromRecord(record, index) {
const ocr = ocrMap(record);
if (!ocr) return null;
const parsed = parsedSummary(record?.sample?.parsed);
const reason = record?.sample?.reason || "missing-reason";
const savedAt = record?.savedAt || "unknown";
const capture = record?.sample?.capture || {};
const ocrFieldIds = Object.keys(ocr).sort();
const missingFastFields = REQUIRED_FAST_OCR_FIELDS.filter((field) => !ocr[field]);
const hasEquippedFooterOcr = Boolean(ocr["artifact-footer"]);
const locked = typeof capture.locked === "boolean" ? capture.locked : null;
return {
id: `review-${simplifyId(savedAt)}-${index}`,
savedAt,
reason,
confirmed: false,
resolution: capture.width && capture.height ? `${capture.width}x${capture.height}` : "",
ocrFieldIds,
missingFastFields,
hasEquippedFooterOcr,
locked,
likelyStaleCapture: missingFastFields.includes("artifact-slot"),
parsed,
ocr,
reviewPrompt: {
action: "Confirm or correct parsed fields before moving this case into src/eval/corpus/confirmedReviewCorpus.ts.",
expectedFields: {
name: parsed.name || "",
slot: parsed.slot || "",
level: parsed.level ?? "",
mainStat: parsed.mainStat || "",
mainValue: parsed.mainValue || "",
setName: parsed.setName || "",
equipped: parsed.equipped || "",
substats: parsed.substats || [],
},
validationChecks: {
equipped: hasEquippedFooterOcr ? "Confirm character name or mark Not detected." : "No footer OCR in this sample.",
locked: locked === null ? "No lock-state payload in this sample." : `Confirm locked=${locked}.`,
},
},
};
}
function candidateKey(candidate) {
return [
candidate.reason,
candidate.parsed.name,
candidate.parsed.slot,
candidate.parsed.mainStat,
candidate.parsed.setName,
JSON.stringify(candidate.parsed.substats || []),
JSON.stringify(candidate.ocr),
].join("\u001f");
}
function candidatePriority(candidate) {
const reason = candidate.reason || "";
const parsed = candidate.parsed || {};
const missingCount = candidate.missingFastFields?.length || 0;
const hasUnknownCritical = [parsed.name, parsed.slot, parsed.mainStat, parsed.setName].some((value) => String(value || "").startsWith("Unknown"));
if (candidate.likelyStaleCapture) return 4;
if (missingCount === 0 && /low-field|low-total|capture-rejected|initial-selection/i.test(reason)) return 0;
if (missingCount === 0 && (parsed.notes || []).some((note) => /fuzzy|incomplete|not confidently|low/i.test(note))) return 1;
if (missingCount === 0) return 2;
if (missingCount <= 1 && !hasUnknownCritical) return 3;
return 5;
}
async function readCandidates(inputPath, limit) {
const candidates = [];
const seen = new Set();
let total = 0;
let invalid = 0;
const rl = readline.createInterface({ input: fs.createReadStream(inputPath, { encoding: "utf8" }) });
for await (const line of rl) {
if (!line.trim()) continue;
total++;
let record;
try {
record = JSON.parse(line);
} catch {
invalid++;
continue;
}
const candidate = candidateFromRecord(record, total);
if (!candidate) continue;
const key = candidateKey(candidate);
if (seen.has(key)) continue;
seen.add(key);
candidates.push(candidate);
}
candidates.sort((left, right) => {
const priority = candidatePriority(left) - candidatePriority(right);
if (priority) return priority;
const missingDiff = (left.missingFastFields?.length || 0) - (right.missingFastFields?.length || 0);
if (missingDiff) return missingDiff;
return String(right.savedAt).localeCompare(String(left.savedAt));
});
return { total, invalid, candidates: candidates.slice(0, limit), uniqueCandidates: candidates.length };
}
function increment(map, key) {
const normalizedKey = key || "unknown";
map[normalizedKey] = (map[normalizedKey] || 0) + 1;
}
function buildExportStats(candidates) {
const reasonCounts = {};
const missingFastFieldCounts = {};
let completeFastFields = 0;
let likelyStaleCaptures = 0;
let equippedFooterCandidates = 0;
let lockedTrueCandidates = 0;
let lockedFalseCandidates = 0;
for (const candidate of candidates) {
increment(reasonCounts, candidate.reason);
if (candidate.likelyStaleCapture) likelyStaleCaptures++;
if (candidate.hasEquippedFooterOcr) equippedFooterCandidates++;
if (candidate.locked === true) lockedTrueCandidates++;
if (candidate.locked === false) lockedFalseCandidates++;
const missingFields = candidate.missingFastFields || [];
if (missingFields.length === 0) completeFastFields++;
for (const field of missingFields) increment(missingFastFieldCounts, field);
}
return {
completeFastFields,
likelyStaleCaptures,
equippedFooterCandidates,
lockedTrueCandidates,
lockedFalseCandidates,
reasonCounts,
missingFastFieldCounts,
};
}
function markdownFor(summary, candidates) {
const lines = [
"# Review Eval Candidates",
"",
`Source: ${summary.inputPath}`,
`Generated: ${summary.generatedAt}`,
`Records read: ${summary.recordsRead}`,
`Unique candidates: ${summary.uniqueCandidates}`,
`Exported candidates: ${summary.exportedCandidates}`,
`Complete fast-field candidates: ${summary.exportStats.completeFastFields}`,
`Likely stale captures: ${summary.exportStats.likelyStaleCaptures}`,
`Equipped footer candidates: ${summary.exportStats.equippedFooterCandidates}`,
`Locked=true candidates: ${summary.exportStats.lockedTrueCandidates}`,
`Locked=false candidates: ${summary.exportStats.lockedFalseCandidates}`,
"",
"These cases are not ground truth yet. Confirm or correct the expected fields before committing any case into `src/eval/corpus/`.",
"",
"## Export stats",
"",
"Reason counts:",
"",
...Object.entries(summary.exportStats.reasonCounts).map(([reason, count]) => `- ${reason}: ${count}`),
"",
"Missing fast-field counts:",
"",
...Object.entries(summary.exportStats.missingFastFieldCounts).map(([field, count]) => `- ${field}: ${count}`),
"",
];
for (const candidate of candidates) {
lines.push(`## ${candidate.id}`);
lines.push("");
lines.push(`- savedAt: ${candidate.savedAt}`);
lines.push(`- reason: ${candidate.reason}`);
lines.push(`- missingFastFields: ${candidate.missingFastFields.join(", ") || "none"}`);
lines.push(`- likelyStaleCapture: ${candidate.likelyStaleCapture ? "yes" : "no"}`);
lines.push(`- hasEquippedFooterOcr: ${candidate.hasEquippedFooterOcr ? "yes" : "no"}`);
lines.push(`- locked: ${candidate.locked === null ? "unknown" : candidate.locked}`);
lines.push(`- parsed: ${candidate.parsed.name || "?"} | ${candidate.parsed.slot || "?"} | ${candidate.parsed.mainStat || "?"} | ${candidate.parsed.setName || "?"} | equipped=${candidate.parsed.equipped || "?"}`);
lines.push(`- substats: ${(candidate.parsed.substats || []).join(", ") || "?"}`);
lines.push("");
lines.push("OCR:");
for (const [id, text] of Object.entries(candidate.ocr)) {
lines.push(`- ${id}: ${JSON.stringify(text)}`);
}
lines.push("");
}
return `${lines.join("\n")}\n`;
}
async function main() {
const inputPath = path.resolve(argValue("input", defaultReviewSamplesPath()));
const outputDir = path.resolve(argValue("out", path.join(process.cwd(), "outputs", "review-eval-candidates")));
const limit = Math.max(1, Math.min(500, Number(argValue("limit", String(DEFAULT_LIMIT))) || DEFAULT_LIMIT));
if (!fs.existsSync(inputPath)) {
throw new Error(`Review sample file not found: ${inputPath}`);
}
fs.mkdirSync(outputDir, { recursive: true });
const result = await readCandidates(inputPath, limit);
const summary = {
inputPath,
outputDir,
generatedAt: new Date().toISOString(),
recordsRead: result.total,
invalidRecords: result.invalid,
uniqueCandidates: result.uniqueCandidates,
exportedCandidates: result.candidates.length,
exportStats: buildExportStats(result.candidates),
};
const payload = { summary, candidates: result.candidates };
const jsonPath = path.join(outputDir, "review-eval-candidates.json");
const mdPath = path.join(outputDir, "review-eval-candidates.md");
fs.writeFileSync(jsonPath, `${JSON.stringify(payload, null, 2)}\n`, "utf8");
fs.writeFileSync(mdPath, markdownFor(summary, result.candidates), "utf8");
console.log(JSON.stringify({ ok: true, ...summary, jsonPath, mdPath }, null, 2));
}
if (require.main === module) {
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
}
module.exports = {
buildExportStats,
candidateFromRecord,
candidatePriority,
defaultReviewSamplesPath,
markdownFor,
readCandidates,
};
scripts/live-preflight.cjs
+148
View File
@@ -0,0 +1,148 @@
const fs = require("node:fs");
const path = require("node:path");
function argValue(name, fallback = "") {
const prefix = `--${name}=`;
const match = process.argv.find((entry) => entry.startsWith(prefix));
return match ? match.slice(prefix.length) : fallback;
}
function hasFlag(name) {
return process.argv.includes(`--${name}`);
}
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function parseWaitSeconds(value) {
if (value === "") return 0;
const parsed = Number(value);
if (!Number.isInteger(parsed) || parsed < 0) {
throw new Error("--wait must be a non-negative integer number of seconds.");
}
return parsed;
}
function expectedSignature() {
const mainPath = path.join(process.cwd(), "electron", "main.ts");
const source = fs.readFileSync(mainPath, "utf8");
const match = source.match(/APP_RUNTIME_SIGNATURE\s*=\s*"([^"]+)"/);
return match?.[1] || "";
}
async function fetchJson(baseUrl, endpoint) {
let response;
try {
response = await fetch(`${baseUrl}${endpoint}`);
} catch (error) {
throw new Error(`Could not reach ${baseUrl}${endpoint}. Start the elevated app with npm run dev:admin and confirm UAC before running live scans. (${error instanceof Error ? error.message : String(error)})`);
}
const text = await response.text();
let payload;
try {
payload = text ? JSON.parse(text) : null;
} catch {
throw new Error(`${endpoint} returned non-JSON response (${response.status}).`);
}
if (!response.ok) throw new Error(`${endpoint} returned ${response.status}: ${JSON.stringify(payload)}`);
return payload;
}
function validatePreflight({ health, status, expected, requireElevated = true, requireGenshin = true }) {
const errors = [];
const appBuild = health?.appBuild;
const runtime = status?.status?.runtimeInfo;
if (!appBuild?.signature) {
errors.push("/health is missing appBuild.signature.");
} else if (expected && appBuild.signature !== expected) {
errors.push(`Runtime signature '${appBuild.signature}' does not match source '${expected}'.`);
}
if (!status?.status) errors.push("/scanner/status is missing status payload.");
if (!runtime) {
errors.push("/scanner/status is missing runtimeInfo. Open the scanner view and restart the elevated app if needed.");
} else {
if (requireElevated && runtime.isElevated !== true) errors.push("Runtime is not elevated.");
if (requireGenshin && runtime.genshinFound !== true) errors.push("Genshin process/window was not found.");
}
return {
ok: errors.length === 0,
errors,
signature: appBuild?.signature || "",
expectedSignature: expected || "",
isElevated: runtime?.isElevated,
genshinFound: runtime?.genshinFound,
targetProcess: runtime?.targetProcess || "",
foregroundProcess: runtime?.foregroundProcess || "",
};
}
function formatSummary(result) {
const lines = [
`live preflight: ${result.ok ? "PASS" : "FAIL"}`,
`signature: ${result.signature || "missing"}`,
`expected: ${result.expectedSignature || "unknown"}`,
`elevated: ${result.isElevated === true ? "yes" : result.isElevated === false ? "no" : "unknown"}`,
`genshin: ${result.genshinFound === true ? "yes" : result.genshinFound === false ? "no" : "unknown"}`,
`target: ${result.targetProcess || "unknown"}`,
`foreground: ${result.foregroundProcess || "unknown"}`,
];
if (!result.ok) {
lines.push("errors:");
for (const error of result.errors) lines.push(`- ${error}`);
}
return lines.join("\n");
}
async function runPreflight({ baseUrl, expected, requireElevated, requireGenshin }) {
const health = await fetchJson(baseUrl, "/health");
const status = await fetchJson(baseUrl, "/scanner/status");
return validatePreflight({ health, status, expected, requireElevated, requireGenshin });
}
async function waitForPreflight(options, waitSeconds) {
const deadline = Date.now() + waitSeconds * 1000;
let lastError = null;
let lastResult = null;
while (true) {
try {
const result = await runPreflight(options);
lastResult = result;
if (result.ok || Date.now() >= deadline) return result;
} catch (error) {
lastError = error;
if (Date.now() >= deadline) throw lastError;
}
await sleep(1000);
}
}
async function main() {
const baseUrl = argValue("base-url", "http://127.0.0.1:17317").replace(/\/$/, "");
const expected = argValue("expected-signature", expectedSignature());
const requireElevated = !hasFlag("allow-standard");
const requireGenshin = !hasFlag("allow-missing-genshin");
const waitSeconds = parseWaitSeconds(argValue("wait", ""));
const options = { baseUrl, expected, requireElevated, requireGenshin };
const result = waitSeconds > 0 ? await waitForPreflight(options, waitSeconds) : await runPreflight(options);
console.log(hasFlag("json") ? JSON.stringify(result, null, 2) : formatSummary(result));
if (!result.ok) process.exit(1);
}
if (require.main === module) {
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
}
module.exports = {
formatSummary,
parseWaitSeconds,
runPreflight,
validatePreflight,
};
scripts/live-soak.ps1
+108 -3
View File
@@ -77,6 +77,7 @@ function Get-ScannerStatus {
function Test-ProbeSucceeded([object]$ProbePayload) {
if ($ProbePayload.ok) { return $true }
if ($ProbePayload.changed) { return $true }
if ($ProbePayload.click -and $ProbePayload.click.clicked -and $ProbePayload.click.moved -and -not $ProbePayload.click.inputBlocked) { return $true }
return $false
}
@@ -282,6 +283,9 @@ function New-PerformanceAssessment([object[]]$Summaries) {
$limitReports += [pscustomobject]@{
limit = [int]$group.Name
engineCount = $entries.Count
enginesCompared = @($entries | ForEach-Object { $_.engine })
comparisonComplete = (@($entries | Where-Object { $_.engine -eq "current" }).Count -gt 0 -and @($entries | Where-Object { $_.engine -eq "ik-traineddata" }).Count -gt 0)
winnerEngine = $winner.engine
winnerQualified = $winner.qualified
winnerMissRate = $winner.missRate
@@ -293,19 +297,37 @@ function New-PerformanceAssessment([object[]]$Summaries) {
}
$goal100 = @($limitReports | Where-Object { $_.limit -eq 100 } | Select-Object -First 1)
$goal100Decision = "not-run: missing 100-artifact assessment"
if ($goal100.Count -gt 0) {
if (-not $goal100[0].comparisonComplete) {
$goal100Decision = "not-comparable: current and ik-traineddata were not both run"
} elseif (-not $goal100[0].winnerQualified) {
$goal100Decision = "not-qualified: 100-artifact winner failed quality gates"
} else {
$goal100Decision = "qualified-comparison: winner=$($goal100[0].winnerEngine)"
}
}
return [pscustomobject]@{
createdAt = (Get-Date).ToString("o")
goalLimit = 100
goalEngines = @("current", "ik-traineddata")
goal100Decision = $goal100Decision
goal100 = if ($goal100.Count -gt 0) { $goal100[0] } else { $null }
limits = $limitReports
}
}
function Write-PerformanceAssessment([object]$Assessment) {
if ($Assessment.goal100Decision) {
Write-Host "assessment goal100: $($Assessment.goal100Decision)"
}
foreach ($limit in @($Assessment.limits)) {
Write-Host ("assessment limit={0}: winner={1} qualified={2} missRate={3:P1} reviewRate={4:P1} activeAvg={5}ms projected100={6}ms" -f `
Write-Host ("assessment limit={0}: winner={1} qualified={2} completeCompare={3} engines={4} missRate={5:P1} reviewRate={6:P1} activeAvg={7}ms projected100={8}ms" -f `
$limit.limit,
$limit.winnerEngine,
$limit.winnerQualified,
$limit.comparisonComplete,
($limit.enginesCompared -join ","),
$limit.winnerMissRate,
$limit.winnerReviewRate,
$limit.winnerActiveAverageMsPerParsed,
@@ -359,6 +381,36 @@ function Invoke-AssessmentSelfTest {
averageCardReadyMs = 205
averageScrollReadyMs = 80
},
[pscustomobject]@{
engine = "current"
limit = 20
status = "done"
parsed = 20
review = 1
misses = 0
activeAverageMsPerParsed = 390
averageMsPerParsed = 405
activeProjectedMsFor100 = 39000
averageOcrMs = 150
averageCaptureMs = 130
averageCardReadyMs = 80
averageScrollReadyMs = 0
},
[pscustomobject]@{
engine = "ik-traineddata"
limit = 20
status = "done"
parsed = 20
review = 2
misses = 0
activeAverageMsPerParsed = 460
averageMsPerParsed = 475
activeProjectedMsFor100 = 46000
averageOcrMs = 190
averageCaptureMs = 130
averageCardReadyMs = 80
averageScrollReadyMs = 0
},
[pscustomobject]@{
engine = "broken-fast"
limit = 45
@@ -393,6 +445,7 @@ function Invoke-AssessmentSelfTest {
$assessment = New-PerformanceAssessment -Summaries $synthetic
$goal100 = $assessment.goal100
$limit20 = @($assessment.limits | Where-Object { $_.limit -eq 20 } | Select-Object -First 1)[0]
$limit45 = @($assessment.limits | Where-Object { $_.limit -eq 45 } | Select-Object -First 1)[0]
if ($goal100.winnerEngine -ne "ik-traineddata") {
@@ -401,6 +454,21 @@ function Invoke-AssessmentSelfTest {
if (-not $goal100.winnerQualified) {
throw "Assessment self-test failed: expected limit=100 winner to be qualified."
}
if (-not $goal100.comparisonComplete) {
throw "Assessment self-test failed: expected limit=100 to be a complete current vs ik-traineddata comparison."
}
if ($assessment.goal100Decision -ne "qualified-comparison: winner=ik-traineddata") {
throw "Assessment self-test failed: unexpected goal100Decision '$($assessment.goal100Decision)'."
}
if ($limit20.winnerEngine -ne "current") {
throw "Assessment self-test failed: expected current to win limit=20, got '$($limit20.winnerEngine)'."
}
if (-not $limit20.comparisonComplete) {
throw "Assessment self-test failed: expected limit=20 to be a complete current vs ik-traineddata comparison."
}
if (-not $limit20.winnerQualified) {
throw "Assessment self-test failed: expected limit=20 winner to be qualified."
}
if ($limit45.winnerEngine -ne "current") {
throw "Assessment self-test failed: expected current to win limit=45, got '$($limit45.winnerEngine)'."
}
@@ -411,6 +479,27 @@ function Invoke-AssessmentSelfTest {
throw "Assessment self-test failed: expected broken-fast run to be rejected for miss rate."
}
$singleEngineAssessment = New-PerformanceAssessment -Summaries @(
[pscustomobject]@{
engine = "current"
limit = 100
status = "done"
parsed = 100
review = 0
misses = 0
activeAverageMsPerParsed = 500
averageMsPerParsed = 520
activeProjectedMsFor100 = 50000
averageOcrMs = 180
averageCaptureMs = 120
averageCardReadyMs = 100
averageScrollReadyMs = 50
}
)
if ($singleEngineAssessment.goal100Decision -ne "not-comparable: current and ik-traineddata were not both run") {
throw "Assessment self-test failed: expected single-engine 100 run to be not-comparable, got '$($singleEngineAssessment.goal100Decision)'."
}
Write-PerformanceAssessment $assessment
Write-Host "Assessment self-test passed." -ForegroundColor Green
return $assessment
@@ -527,6 +616,7 @@ function Wait-ForScannerIdle([int]$Limit, [string]$Engine) {
$startedAt = Get-Date
$pollIndex = 0
$lastStatus = $null
$observedMatchingRun = $false
while ($true) {
Start-Sleep -Seconds $PollIntervalSeconds
@@ -536,8 +626,21 @@ function Wait-ForScannerIdle([int]$Limit, [string]$Engine) {
Save-Json "scan-$Engine-limit-$Limit-poll-$pollIndex" $statusPayload | Out-Null
$running = [bool]$statusPayload.status.running
$summaryTarget = if ($statusPayload.status.summary) { [int]$statusPayload.status.summary.targetCount } else { 0 }
$scanStart = @($statusPayload.status.diagnosticEvents | Where-Object { $_.phase -eq "scan-start" } | Select-Object -Last 1)
$scanStartDetails = if ($scanStart.Count -gt 0) { $scanStart[0].details } else { $null }
$scanStartLimit = if ($scanStartDetails -and $scanStartDetails.scanLimit) { [int]$scanStartDetails.scanLimit } else { 0 }
$scanStartEngine = if ($scanStartDetails -and $scanStartDetails.ocrEngine) { [string]$scanStartDetails.ocrEngine } else { "" }
$matchesScanStart = $scanStartLimit -eq $Limit -and ($scanStartEngine -eq "" -or $scanStartEngine -eq $Engine)
$matchesFinalSummary = $summaryTarget -eq $Limit
if ($running -and ($matchesScanStart -or $matchesFinalSummary)) {
$observedMatchingRun = $true
}
if (-not $running) {
return $statusPayload
if ($observedMatchingRun -or $matchesScanStart -or $matchesFinalSummary) {
return $statusPayload
}
Write-Host "Waiting for scanner run limit=$Limit engine=$Engine to appear; ignoring unrelated idle status." -ForegroundColor DarkGray
}
$elapsed = ((Get-Date) - $startedAt).TotalSeconds
@@ -619,6 +722,8 @@ try {
}
} elseif (-not $probe.ok -and $probe.changed) {
Write-Host "Probe index=$index changed the detail panel even though helper cursor/click readback was not clean; continuing." -ForegroundColor Yellow
} elseif (-not $probe.ok -and $probe.click -and $probe.click.clicked -and $probe.click.moved -and -not $probe.click.inputBlocked) {
Write-Host "Probe index=$index delivered input but detail did not change; continuing because the target may already be selected." -ForegroundColor Yellow
}
}
@@ -626,7 +731,7 @@ try {
foreach ($limit in $Limits) {
if ($limit -lt 1) { continue }
Write-Host "Starting bounded scanner run limit=$limit engine=$engine"
$start = Invoke-DevJson "/scanner/start?limit=$limit&engine=$engine"
$start = Invoke-DevJson "/scanner/start?entry=visible-inventory&limit=$limit&engine=$engine"
Save-Json "scan-$engine-limit-$limit-start" $start | Out-Null
$finalStatus = Wait-ForScannerIdle -Limit $limit -Engine $engine
scripts/prepare-confirmed-review-case.cjs
+109
View File
@@ -0,0 +1,109 @@
const fs = require("node:fs");
const path = require("node:path");
const EVAL_FIELDS = new Set(["name", "slot", "level", "mainStat", "mainValue", "setName", "equipped", "substats"]);
function argValue(name, fallback = "") {
const prefix = `--${name}=`;
const match = process.argv.find((entry) => entry.startsWith(prefix));
return match ? match.slice(prefix.length) : fallback;
}
function parseExpectedFields() {
const inlineJson = argValue("expect-json");
const expectFile = argValue("expect-file");
if (!inlineJson && !expectFile) {
throw new Error("Missing expected labels. Pass --expect-json=... or --expect-file=...");
}
const raw = (inlineJson || fs.readFileSync(path.resolve(expectFile), "utf8")).replace(/^\uFEFF/, "");
const parsed = JSON.parse(raw);
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
throw new Error("Expected labels must be a JSON object.");
}
const expect = {};
for (const [field, value] of Object.entries(parsed)) {
if (!EVAL_FIELDS.has(field)) throw new Error(`Unknown expected field: ${field}`);
if (field === "level") {
if (!Number.isInteger(value) || value < 0) throw new Error("Expected level must be a non-negative integer.");
expect[field] = value;
continue;
}
if (field === "substats") {
if (!Array.isArray(value) || !value.every((entry) => typeof entry === "string" && entry.trim())) {
throw new Error("Expected substats must be a non-empty string array.");
}
expect[field] = value;
continue;
}
if (typeof value !== "string" || !value.trim()) throw new Error(`Expected ${field} must be a non-empty string.`);
expect[field] = value;
}
if (Object.keys(expect).length === 0) throw new Error("Expected labels must include at least one field.");
return expect;
}
function loadCandidate(inputPath, candidateId) {
const payload = JSON.parse(fs.readFileSync(path.resolve(inputPath), "utf8"));
const candidates = Array.isArray(payload?.candidates) ? payload.candidates : [];
const candidate = candidates.find((entry) => entry.id === candidateId);
if (!candidate) throw new Error(`Candidate not found: ${candidateId}`);
if (!candidate.ocr || typeof candidate.ocr !== "object" || Object.keys(candidate.ocr).length === 0) {
throw new Error(`Candidate has no OCR payload: ${candidateId}`);
}
return candidate;
}
function stableId(value) {
return String(value || "")
.replace(/[^0-9A-Za-z]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 80) || "review-case";
}
function objectLiteral(value, indent = 2) {
return JSON.stringify(value, null, indent).replace(/"([A-Za-z_$][0-9A-Za-z_$]*)":/g, "$1:");
}
function snippetFor(candidate, expect) {
const id = stableId(argValue("id", `confirmed-${candidate.id}`));
const entry = {
id,
confirmed: true,
ocr: candidate.ocr,
expect,
meta: {
source: "review-sample",
resolution: candidate.resolution || undefined,
note: `${candidate.reason || "review-sample"} | savedAt=${candidate.savedAt || "unknown"} | sourceCandidate=${candidate.id}`,
},
};
return `${objectLiteral(entry, 2)},\n`;
}
function defaultInputPath() {
return path.join(process.cwd(), "outputs", "review-eval-candidates", "review-eval-candidates.json");
}
function main() {
const inputPath = argValue("input", defaultInputPath());
const candidateId = argValue("candidate");
if (!candidateId) throw new Error("Missing candidate id. Pass --candidate=<id>.");
const outputPath = path.resolve(argValue("out", path.join(process.cwd(), "outputs", "review-eval-candidates", `${stableId(candidateId)}.confirmed.ts`)));
const candidate = loadCandidate(inputPath, candidateId);
const expect = parseExpectedFields();
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
const snippet = snippetFor(candidate, expect);
fs.writeFileSync(outputPath, snippet, "utf8");
console.log(JSON.stringify({ ok: true, candidateId, outputPath, labeledFields: Object.keys(expect) }, null, 2));
}
if (require.main === module) {
main();
}
module.exports = {
loadCandidate,
parseExpectedFields,
snippetFor,
stableId,
};
scripts/validate-scan-assessment.cjs
+189
View File
@@ -0,0 +1,189 @@
const fs = require("node:fs");
const path = require("node:path");
function argValue(name, fallback = "") {
const prefix = `--${name}=`;
const match = process.argv.find((entry) => entry.startsWith(prefix));
return match ? match.slice(prefix.length) : fallback;
}
function hasFlag(name) {
return process.argv.includes(`--${name}`);
}
function defaultAssessmentRoot() {
return path.join(process.cwd(), "outputs", "live-soak");
}
function findLatestAssessment(rootDir = defaultAssessmentRoot()) {
const resolvedRoot = path.resolve(rootDir);
if (!fs.existsSync(resolvedRoot)) throw new Error(`Assessment root not found: ${resolvedRoot}`);
const candidates = fs.readdirSync(resolvedRoot, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => {
const assessmentPath = path.join(resolvedRoot, entry.name, "scan-performance-assessment.json");
if (!fs.existsSync(assessmentPath)) return null;
const stat = fs.statSync(assessmentPath);
return { assessmentPath, runName: entry.name, mtimeMs: stat.mtimeMs };
})
.filter(Boolean)
.sort((left, right) => {
const mtimeDiff = right.mtimeMs - left.mtimeMs;
if (Math.abs(mtimeDiff) > 1) return mtimeDiff;
return right.runName.localeCompare(left.runName);
});
if (candidates.length === 0) throw new Error(`No scan-performance-assessment.json found under: ${resolvedRoot}`);
return candidates[0].assessmentPath;
}
function loadAssessment(inputPath) {
if (!inputPath) throw new Error("Missing assessment file. Pass --input=<scan-performance-assessment.json>.");
return JSON.parse(fs.readFileSync(path.resolve(inputPath), "utf8").replace(/^\uFEFF/, ""));
}
function findLimitAssessment(assessment, limit) {
if (limit === 100 && assessment?.goal100) return assessment.goal100;
const limits = Array.isArray(assessment?.limits) ? assessment.limits : [];
return limits.find((entry) => Number(entry?.limit) === limit) || null;
}
function validateAssessment(assessment, options = {}) {
const errors = [];
const expectedWinner = options.expectedWinner || "any";
const expectedLimit = options.limit === undefined ? 100 : Number(options.limit);
if (!assessment || typeof assessment !== "object") {
return { ok: false, errors: ["Assessment must be a JSON object."] };
}
if (!Number.isInteger(expectedLimit) || expectedLimit < 1) {
errors.push(`--limit must be a positive integer, got ${options.limit}.`);
}
const limitAssessment = findLimitAssessment(assessment, expectedLimit);
if (!limitAssessment || typeof limitAssessment !== "object") {
errors.push(`Missing limit=${expectedLimit} assessment.`);
}
if (expectedLimit === 100 && assessment.goal100Decision !== `qualified-comparison: winner=${limitAssessment?.winnerEngine}`) {
errors.push(`goal100Decision is not a qualified comparison: ${assessment.goal100Decision || "<missing>"}`);
}
if (limitAssessment?.limit !== expectedLimit) {
errors.push(`limit assessment must be ${expectedLimit}, got ${limitAssessment?.limit ?? "<missing>"}.`);
}
if (limitAssessment?.comparisonComplete !== true) errors.push(`limit=${expectedLimit}.comparisonComplete must be true.`);
if (limitAssessment?.winnerQualified !== true) errors.push(`limit=${expectedLimit}.winnerQualified must be true.`);
if (expectedWinner !== "any" && limitAssessment?.winnerEngine !== expectedWinner) {
errors.push(`Expected winner '${expectedWinner}', got '${limitAssessment?.winnerEngine ?? "<missing>"}'.`);
}
const engines = Array.isArray(limitAssessment?.engines) ? limitAssessment.engines : [];
const engineNames = new Set(engines.map((entry) => entry?.engine));
for (const required of ["current", "ik-traineddata"]) {
if (!engineNames.has(required)) errors.push(`limit=${expectedLimit} is missing engine result: ${required}.`);
}
const winner = engines.find((entry) => entry?.engine === limitAssessment?.winnerEngine);
if (!winner) {
errors.push(`Winner engine is missing from limit=${expectedLimit}.engines: ${limitAssessment?.winnerEngine ?? "<missing>"}.`);
} else {
const winnerMissRate = Number(winner.missRate);
const winnerReviewRate = Number(winner.reviewRate);
const summaryWinnerMissRate = Number(limitAssessment?.winnerMissRate);
const summaryWinnerReviewRate = Number(limitAssessment?.winnerReviewRate);
const winnerActiveAverageMsPerParsed = Number(limitAssessment?.winnerActiveAverageMsPerParsed);
const winnerActiveProjectedMsFor100 = Number(limitAssessment?.winnerActiveProjectedMsFor100);
if (winner.qualified !== true) errors.push("Winner engine result must be qualified.");
if (!Number.isFinite(winnerMissRate)) {
errors.push(`Winner missRate must be a finite number, got ${winner.missRate ?? "<missing>"}.`);
} else if (winnerMissRate > 0.02) {
errors.push(`Winner missRate exceeds 2%: ${winner.missRate}.`);
}
if (!Number.isFinite(winnerReviewRate)) {
errors.push(`Winner reviewRate must be a finite number, got ${winner.reviewRate ?? "<missing>"}.`);
} else if (winnerReviewRate > 0.15) {
errors.push(`Winner reviewRate exceeds 15%: ${winner.reviewRate}.`);
}
if (!Number.isFinite(summaryWinnerMissRate)) {
errors.push(`Winner summary missRate must be a finite number, got ${limitAssessment?.winnerMissRate ?? "<missing>"}.`);
} else if (summaryWinnerMissRate > 0.02) {
errors.push(`Winner summary missRate exceeds 2%: ${limitAssessment.winnerMissRate}.`);
}
if (!Number.isFinite(summaryWinnerReviewRate)) {
errors.push(`Winner summary reviewRate must be a finite number, got ${limitAssessment?.winnerReviewRate ?? "<missing>"}.`);
} else if (summaryWinnerReviewRate > 0.15) {
errors.push(`Winner summary reviewRate exceeds 15%: ${limitAssessment.winnerReviewRate}.`);
}
if (!Number.isFinite(winnerActiveAverageMsPerParsed) || winnerActiveAverageMsPerParsed <= 0) {
errors.push(`Winner active average timing must be a positive finite number, got ${limitAssessment?.winnerActiveAverageMsPerParsed ?? "<missing>"}.`);
}
if (!Number.isFinite(winnerActiveProjectedMsFor100) || winnerActiveProjectedMsFor100 <= 0) {
errors.push(`Winner projected100 timing must be a positive finite number, got ${limitAssessment?.winnerActiveProjectedMsFor100 ?? "<missing>"}.`);
}
}
return {
ok: errors.length === 0,
errors,
createdAt: assessment.createdAt || "",
limit: expectedLimit,
winnerEngine: limitAssessment?.winnerEngine,
winnerActiveAverageMsPerParsed: limitAssessment?.winnerActiveAverageMsPerParsed,
winnerActiveProjectedMsFor100: limitAssessment?.winnerActiveProjectedMsFor100,
winnerMissRate: limitAssessment?.winnerMissRate,
winnerReviewRate: limitAssessment?.winnerReviewRate,
};
}
function formatSummary(result) {
const status = result.ok ? "PASS" : "FAIL";
const lines = [
`scan assessment: ${status}`,
`input: ${result.inputPath || "unknown"}`,
`createdAt: ${result.createdAt || "unknown"}`,
`limit: ${result.limit ?? "unknown"}`,
`winner: ${result.winnerEngine || "unknown"}`,
`activeAvg: ${result.winnerActiveAverageMsPerParsed ?? "unknown"}ms/artifact`,
`projected100: ${result.winnerActiveProjectedMsFor100 ?? "unknown"}ms`,
`missRate: ${result.winnerMissRate ?? "unknown"}`,
`reviewRate: ${result.winnerReviewRate ?? "unknown"}`,
];
if (!result.ok) {
lines.push("errors:");
for (const error of result.errors) lines.push(`- ${error}`);
}
return lines.join("\n");
}
function main() {
const inputPath = hasFlag("latest")
? findLatestAssessment(argValue("root", defaultAssessmentRoot()))
: argValue("input", process.argv[2] || "");
const expectedWinner = argValue("expect-winner", "any");
if (!["any", "current", "ik-traineddata"].includes(expectedWinner)) {
throw new Error("--expect-winner must be one of: any, current, ik-traineddata.");
}
const limit = Number(argValue("limit", "100"));
const assessment = loadAssessment(inputPath);
const result = validateAssessment(assessment, { expectedWinner, limit });
const payload = { inputPath: path.resolve(inputPath), ...result };
console.log(hasFlag("summary") ? formatSummary(payload) : JSON.stringify(payload, null, 2));
if (!result.ok) process.exit(1);
}
if (require.main === module) {
try {
main();
} catch (error) {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
}
}
module.exports = {
findLatestAssessment,
findLimitAssessment,
formatSummary,
loadAssessment,
validateAssessment,
};
src/data/genshinGameData.json
+5
View File
@@ -5272,6 +5272,11 @@
},
"pieceAliases": {
"A Note in Springs Leich": "A Note in Spring's Leich",
"Determination oT": "Viridescent Venerer's Determination",
"Determmation oT": "Viridescent Venerer's Determination",
"From Grand Dreams. Tn aking": "Moment That Ceased Upon Waking From Grand Dreams",
"From Grand Dreams Tn aking": "Moment That Ceased Upon Waking From Grand Dreams",
"Postintty That Ceased Upon": "Moment That Ceased Upon Waking From Grand Dreams",
"Viridescent Vencrers Vessel": "Viridescent Venerer's Vessel",
"Holy Crown of the Believer ": "Holy Crown of the Believer"
},
src/eval/corpus/confirmedReviewCorpus.test.ts
+18
View File
@@ -0,0 +1,18 @@
import { describe, expect, it } from "vitest";
import { confirmedReviewCorpus, ocrEvalCorpus, seedCorpus, validateConfirmedReviewCorpus } from ".";
describe("confirmed review corpus", () => {
it("contains only human-confirmed review labels", () => {
expect(validateConfirmedReviewCorpus()).toEqual([]);
});
it("does not duplicate eval case ids", () => {
const ids = ocrEvalCorpus.map((entry) => entry.id);
expect(new Set(ids).size).toBe(ids.length);
});
it("is included in the full OCR eval corpus", () => {
expect(ocrEvalCorpus).toHaveLength(seedCorpus.length + confirmedReviewCorpus.length);
});
});
src/eval/corpus/confirmedReviewCorpus.ts
+15
View File
@@ -0,0 +1,15 @@
import type { OcrEvalCase } from "../ocrEvalHarness";
export interface ConfirmedReviewEvalCase extends OcrEvalCase {
confirmed: true;
meta: NonNullable<OcrEvalCase["meta"]> & {
source: "review-sample";
note: string;
};
}
// Human-confirmed review samples belong here after their `expect` values were
// checked against the real artifact. Do not paste unconfirmed exporter output
// directly from outputs/review-eval-candidates/.
export const confirmedReviewCorpus: ConfirmedReviewEvalCase[] = [];
src/eval/corpus/index.ts
+24
View File
@@ -0,0 +1,24 @@
import { confirmedReviewCorpus, type ConfirmedReviewEvalCase } from "./confirmedReviewCorpus";
import { seedCorpus } from "./seedCorpus";
import type { OcrEvalCase } from "../ocrEvalHarness";
export { confirmedReviewCorpus, seedCorpus };
export type { ConfirmedReviewEvalCase };
export const ocrEvalCorpus: OcrEvalCase[] = [...seedCorpus, ...confirmedReviewCorpus];
export function validateConfirmedReviewCorpus(corpus: readonly ConfirmedReviewEvalCase[] = confirmedReviewCorpus) {
const errors: string[] = [];
const seen = new Set<string>();
for (const entry of corpus) {
if (seen.has(entry.id)) errors.push(`${entry.id}: duplicate id`);
seen.add(entry.id);
if (entry.confirmed !== true) errors.push(`${entry.id}: confirmed must be true`);
if (entry.meta.source !== "review-sample") errors.push(`${entry.id}: meta.source must be review-sample`);
if (!entry.meta.note?.trim()) errors.push(`${entry.id}: meta.note must describe the review source/reason`);
if (Object.keys(entry.expect).length === 0) errors.push(`${entry.id}: expect must label at least one field`);
if (Object.keys(entry.ocr).length === 0) errors.push(`${entry.id}: ocr must contain at least one field`);
}
return errors;
}
src/eval/livePreflightScript.test.ts
+56
View File
@@ -0,0 +1,56 @@
import { describe, expect, it } from "vitest";
const { formatSummary, parseWaitSeconds, validatePreflight } = require("../../scripts/live-preflight.cjs") as {
formatSummary: (result: unknown) => string;
parseWaitSeconds: (value: string) => number;
validatePreflight: (input: unknown) => { ok: boolean; errors: string[]; signature: string; isElevated?: boolean; genshinFound?: boolean };
};
function health(signature = "sig-current") {
return { ok: true, appBuild: { signature } };
}
function status(runtime = { isElevated: true, genshinFound: true, targetProcess: "GenshinImpact.exe", foregroundProcess: "GenshinImpact.exe" }) {
return { ok: true, status: { runtimeInfo: runtime } };
}
describe("live preflight script", () => {
it("accepts a matching elevated Genshin runtime", () => {
const result = validatePreflight({ health: health(), status: status(), expected: "sig-current" });
expect(result.ok).toBe(true);
expect(result.errors).toEqual([]);
expect(result.signature).toBe("sig-current");
});
it("rejects stale runtime signatures", () => {
const result = validatePreflight({ health: health("old"), status: status(), expected: "sig-current" });
expect(result.ok).toBe(false);
expect(result.errors.join("\n")).toContain("does not match");
});
it("rejects non-elevated or missing Genshin runtime by default", () => {
const result = validatePreflight({
health: health(),
status: status({ isElevated: false, genshinFound: false, targetProcess: "", foregroundProcess: "explorer.exe" }),
expected: "sig-current",
});
expect(result.ok).toBe(false);
expect(result.errors).toContain("Runtime is not elevated.");
expect(result.errors).toContain("Genshin process/window was not found.");
});
it("formats a concise human summary", () => {
const summary = formatSummary(validatePreflight({ health: health(), status: status(), expected: "sig-current" }));
expect(summary).toContain("live preflight: PASS");
expect(summary).toContain("elevated: yes");
expect(summary).toContain("genshin: yes");
});
it("parses optional wait seconds strictly", () => {
expect(parseWaitSeconds("")).toBe(0);
expect(parseWaitSeconds("120")).toBe(120);
expect(() => parseWaitSeconds("-1")).toThrow("--wait");
expect(() => parseWaitSeconds("1.5")).toThrow("--wait");
expect(() => parseWaitSeconds("soon")).toThrow("--wait");
});
});
src/eval/ocrEval.test.ts
+9 -9
View File
@@ -1,24 +1,24 @@
import { describe, expect, it } from "vitest";
import { formatReport, runOcrEval } from "./ocrEvalHarness";
import { seedCorpus } from "./corpus/seedCorpus";
import { ocrEvalCorpus } from "./corpus";
// Regression gate: the seed corpus is verified ground truth, so the parser must
// read every labeled field correctly. A drop here means an OCR/parser change
// regressed a previously-correct read - look at the printed failures. If a
// change intentionally alters a correct output, update the corpus label in the
// same commit (the label is the source of truth, not the code).
// Regression gate: every case in the combined corpus is verified ground truth,
// so the parser must read every labeled field correctly. A drop here means an
// OCR/parser change regressed a previously-correct read - look at the printed
// failures. If a change intentionally alters a correct output, update the corpus
// label in the same commit (the label is the source of truth, not the code).
describe("OCR eval harness", () => {
const report = runOcrEval(seedCorpus);
const report = runOcrEval(ocrEvalCorpus);
it("prints the accuracy report", () => {
// Surfaced in test output for humans; not an assertion.
// eslint-disable-next-line no-console
console.log("\n" + formatReport(report) + "\n");
expect(report.totalCases).toBe(seedCorpus.length);
expect(report.totalCases).toBe(ocrEvalCorpus.length);
});
it("reads every labeled field on the seed corpus correctly", () => {
it("reads every labeled field on the eval corpus correctly", () => {
const failureSummary = report.failures
.map((failure) => {
const wrong = failure.fields
src/eval/pngBitmap.test.ts
+36
View File
@@ -0,0 +1,36 @@
import { describe, expect, it } from "vitest";
import { deflateSync } from "node:zlib";
import { pngBufferToBitmap } from "../../electron/services/pngBitmap";
import { lockSignalRatio } from "../lib/lockDetection";
function chunk(type: string, data: Buffer) {
const result = Buffer.alloc(12 + data.length);
result.writeUInt32BE(data.length, 0);
result.write(type, 4, 4, "ascii");
data.copy(result, 8);
return result;
}
function rgbPng1x1(r: number, g: number, b: number) {
const ihdr = Buffer.alloc(13);
ihdr.writeUInt32BE(1, 0);
ihdr.writeUInt32BE(1, 4);
ihdr[8] = 8;
ihdr[9] = 2;
const raw = Buffer.from([0, r, g, b]);
return Buffer.concat([
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
chunk("IHDR", ihdr),
chunk("IDAT", deflateSync(raw)),
chunk("IEND", Buffer.alloc(0)),
]);
}
describe("pngBufferToBitmap", () => {
it("decodes RGB PNG pixels for lock detection", () => {
const bitmap = pngBufferToBitmap(rgbPng1x1(235, 92, 90));
expect(bitmap.width).toBe(1);
expect(bitmap.height).toBe(1);
expect(lockSignalRatio(bitmap)).toBe(1);
});
});
src/eval/prepareConfirmedReviewCaseScript.test.ts
+91
View File
@@ -0,0 +1,91 @@
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
function writeCandidatePayload(dir: string) {
const inputPath = path.join(dir, "review-eval-candidates.json");
writeFileSync(
inputPath,
JSON.stringify({
summary: {},
candidates: [
{
id: "review-2026-07-08T15-02-39-765Z-152",
savedAt: "2026-07-08T15:02:39.765Z",
reason: "automatic:parser-notes:p1:r2c3",
resolution: "1920x1080",
ocr: {
"artifact-name": "Pristine Plume of the Blessed",
"artifact-slot": "Plume of Death",
"artifact-main-stat-label": "ATK",
"artifact-level": "+20",
"artifact-substats": "- Energy Recharge+10.4%",
},
},
],
}),
"utf8",
);
return inputPath;
}
describe("prepare confirmed review case script", () => {
it("creates a confirmed corpus snippet from a candidate and explicit labels", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-confirmed-review-"));
try {
const inputPath = writeCandidatePayload(dir);
const expectPath = path.join(dir, "expect.json");
const outputPath = path.join(dir, "snippet.ts");
writeFileSync(
expectPath,
`\uFEFF${JSON.stringify({
name: "Pristine Plume of the Blessed",
slot: "Plume of Death",
level: 20,
mainStat: "ATK",
setName: "Silken Moon's Serenade",
})}`,
"utf8",
);
execFileSync(
"node",
[
"scripts/prepare-confirmed-review-case.cjs",
`--input=${inputPath}`,
"--candidate=review-2026-07-08T15-02-39-765Z-152",
`--expect-file=${expectPath}`,
`--out=${outputPath}`,
],
{ cwd: process.cwd(), stdio: "pipe" },
);
const snippet = readFileSync(outputPath, "utf8");
expect(snippet).toContain("confirmed: true");
expect(snippet).toContain("Pristine Plume of the Blessed");
expect(snippet).toContain('"artifact-name": "Pristine Plume of the Blessed"');
expect(snippet).toContain("sourceCandidate=review-2026-07-08T15-02-39-765Z-152");
expect(snippet).not.toContain("confirmed: false");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects missing explicit labels", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-confirmed-review-"));
try {
const inputPath = writeCandidatePayload(dir);
expect(() =>
execFileSync(
"node",
["scripts/prepare-confirmed-review-case.cjs", `--input=${inputPath}`, "--candidate=review-2026-07-08T15-02-39-765Z-152"],
{ cwd: process.cwd(), stdio: "pipe" },
),
).toThrow();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
src/eval/reviewEvalCandidatesScript.test.ts
+90
View File
@@ -0,0 +1,90 @@
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
function reviewRecord(savedAt: string, reason: string, ocr: Array<{ id: string; text: string }>, locked?: boolean) {
return {
savedAt,
sample: {
reason,
capture: {
width: 1920,
height: 1080,
ocr,
locked,
},
parsed: {
name: "Pristine Plume of the Blessed",
slot: ocr.some((entry) => entry.id === "artifact-slot") ? "Plume of Death" : "Unknown Slot",
level: 20,
mainStat: "ATK",
mainValue: "311",
setName: "Silken Moon's Serenade",
equipped: ocr.some((entry) => entry.id === "artifact-footer") ? "Aino" : "Not detected",
substats: ["Energy Recharge+10.4%"],
confidence: 0.9,
notes: [],
},
},
};
}
const completeOcr = [
{ id: "artifact-name", text: "Pristine Plume of the Blessed" },
{ id: "artifact-slot", text: "Plume of Death" },
{ id: "artifact-main-stat-label", text: "ATK" },
{ id: "artifact-level", text: "+20" },
{ id: "artifact-substats", text: "- Energy Recharge+10.4%" },
{ id: "artifact-footer", text: "Equipped: Aino" },
];
describe("review eval candidate exporter", () => {
it("exports deduplicated candidates with stats and stale markers", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-review-candidates-"));
try {
const inputPath = path.join(dir, "review-samples.jsonl");
const outDir = path.join(dir, "out");
const staleOcr = completeOcr.filter((entry) => entry.id !== "artifact-slot");
writeFileSync(
inputPath,
[
JSON.stringify(reviewRecord("2026-07-08T15:00:00.000Z", "automatic:missing-crops-or-ocr:p1:r0c0", completeOcr)),
JSON.stringify(reviewRecord("2026-07-08T15:00:00.000Z", "automatic:missing-crops-or-ocr:p1:r0c0", completeOcr)),
JSON.stringify(reviewRecord("2026-07-08T15:01:00.000Z", "automatic:capture-rejected:p1:r0c1", staleOcr, true)),
"{not json",
].join("\n"),
"utf8",
);
execFileSync("node", ["scripts/export-review-eval-candidates.cjs", `--input=${inputPath}`, `--out=${outDir}`, "--limit=10"], {
cwd: process.cwd(),
stdio: "pipe",
});
const payload = JSON.parse(readFileSync(path.join(outDir, "review-eval-candidates.json"), "utf8"));
const markdown = readFileSync(path.join(outDir, "review-eval-candidates.md"), "utf8");
expect(payload.summary.recordsRead).toBe(4);
expect(payload.summary.invalidRecords).toBe(1);
expect(payload.summary.uniqueCandidates).toBe(2);
expect(payload.summary.exportedCandidates).toBe(2);
expect(payload.summary.exportStats.completeFastFields).toBe(1);
expect(payload.summary.exportStats.likelyStaleCaptures).toBe(1);
expect(payload.summary.exportStats.equippedFooterCandidates).toBe(2);
expect(payload.summary.exportStats.lockedTrueCandidates).toBe(1);
expect(payload.candidates[0].missingFastFields).toEqual([]);
expect(payload.candidates[0].parsed.equipped).toBe("Aino");
expect(payload.candidates[0].reviewPrompt.expectedFields.equipped).toBe("Aino");
expect(payload.candidates[1].likelyStaleCapture).toBe(true);
expect(payload.candidates[1].locked).toBe(true);
expect(markdown).toContain("## Export stats");
expect(markdown).toContain("- artifact-slot: 1");
expect(markdown).toContain("equipped=Aino");
expect(markdown).toContain("Locked=true candidates: 1");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
src/eval/reviewSampleCorpus.ts
+1 -1
View File
@@ -10,7 +10,7 @@ import type { OcrEvalCase } from "./ocrEvalHarness";
// itself). The intended flow is:
// 1. reviewSampleToEvalCase() extracts the OCR + the parser's current guess.
// 2. A human confirms or corrects `expect` in the produced case.
// 3. The corrected case is committed into src/eval/corpus/.
// 3. The corrected case is committed into src/eval/corpus/confirmedReviewCorpus.ts.
// The `confirmed` flag records whether step 2 happened.
export interface ReviewSampleEvalCase extends OcrEvalCase {
src/eval/scanAssessmentValidatorScript.test.ts
+355
View File
@@ -0,0 +1,355 @@
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
function validAssessment() {
return {
createdAt: "2026-07-08T12:00:00.000Z",
goal100Decision: "qualified-comparison: winner=ik-traineddata",
goal100: {
limit: 100,
comparisonComplete: true,
winnerEngine: "ik-traineddata",
winnerQualified: true,
winnerActiveAverageMsPerParsed: 820,
winnerActiveProjectedMsFor100: 82000,
winnerMissRate: 0,
winnerReviewRate: 0.04,
engines: [
{ engine: "ik-traineddata", qualified: true, missRate: 0, reviewRate: 0.04 },
{ engine: "current", qualified: true, missRate: 0, reviewRate: 0.06 },
],
},
limits: [
{
limit: 20,
comparisonComplete: true,
winnerEngine: "current",
winnerQualified: true,
winnerActiveAverageMsPerParsed: 390,
winnerActiveProjectedMsFor100: 39000,
winnerMissRate: 0,
winnerReviewRate: 0.05,
engines: [
{ engine: "current", qualified: true, missRate: 0, reviewRate: 0.05 },
{ engine: "ik-traineddata", qualified: true, missRate: 0, reviewRate: 0.1 },
],
},
],
};
}
function writeAssessment(dir: string, payload: unknown) {
const inputPath = path.join(dir, "scan-performance-assessment.json");
writeFileSync(inputPath, JSON.stringify(payload), "utf8");
return inputPath;
}
describe("scan assessment validator", () => {
it("accepts a qualified complete 100-artifact comparison", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
const output = execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`], {
cwd: process.cwd(),
encoding: "utf8",
});
expect(JSON.parse(output).ok).toBe(true);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("accepts a matching expected winner", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
const output = execFileSync(
"node",
["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--expect-winner=ik-traineddata"],
{
cwd: process.cwd(),
encoding: "utf8",
},
);
expect(JSON.parse(output).ok).toBe(true);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("prints an opt-in human summary", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
const output = execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary"], {
cwd: process.cwd(),
encoding: "utf8",
});
expect(output).toContain("scan assessment: PASS");
expect(output).toContain(`input: ${inputPath}`);
expect(output).toContain("createdAt: 2026-07-08T12:00:00.000Z");
expect(output).toContain("limit: 100");
expect(output).toContain("winner: ik-traineddata");
expect(output).toContain("activeAvg: 820ms/artifact");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("accepts a qualified complete 20-artifact comparison", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
const output = execFileSync(
"node",
["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary", "--limit=20"],
{
cwd: process.cwd(),
encoding: "utf8",
},
);
expect(output).toContain("scan assessment: PASS");
expect(output).toContain("limit: 20");
expect(output).toContain("winner: current");
expect(output).toContain("activeAvg: 390ms/artifact");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects invalid requested limits", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
let stdout = "";
try {
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary", "--limit=0"], {
cwd: process.cwd(),
encoding: "utf8",
stdio: "pipe",
});
} catch (error) {
stdout = String((error as { stdout?: string }).stdout || "");
}
expect(stdout).toContain("scan assessment: FAIL");
expect(stdout).toContain("--limit must be a positive integer");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects a requested limit that is missing from the assessment", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
let stdout = "";
try {
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary", "--limit=45"], {
cwd: process.cwd(),
encoding: "utf8",
stdio: "pipe",
});
} catch (error) {
stdout = String((error as { stdout?: string }).stdout || "");
}
expect(stdout).toContain("scan assessment: FAIL");
expect(stdout).toContain("Missing limit=45 assessment");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("keeps the validated npm scripts wired through preflight and the correct assessment limit", () => {
const packageJson = JSON.parse(readFileSync(path.join(process.cwd(), "package.json"), "utf8"));
expect(packageJson.scripts["scan:goal:compare:validated"]).toBe(
"npm run scan:live:preflight && npm run scan:goal:compare && npm run scan:assessment:validate -- --latest --summary",
);
expect(packageJson.scripts["scan:goal:compare:validated:wait"]).toBe(
"npm run scan:live:preflight:wait && npm run scan:goal:compare && npm run scan:assessment:validate -- --latest --summary",
);
expect(packageJson.scripts["scan:iterate:compare"]).toBe(
"powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -Limits 20 -ScanEngine compare -BenchmarkOcr",
);
expect(packageJson.scripts["scan:iterate:compare:validated"]).toBe(
"npm run scan:live:preflight && npm run scan:iterate:compare && npm run scan:assessment:validate -- --latest --summary --limit=20",
);
expect(packageJson.scripts["scan:iterate:compare:validated:wait"]).toBe(
"npm run scan:live:preflight:wait && npm run scan:iterate:compare && npm run scan:assessment:validate -- --latest --summary --limit=20",
);
});
it("rejects a mismatched expected winner", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
expect(() =>
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--expect-winner=current"], {
cwd: process.cwd(),
stdio: "pipe",
}),
).toThrow();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("prints errors in summary mode for rejected assessments", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const inputPath = writeAssessment(dir, validAssessment());
let stdout = "";
try {
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--expect-winner=current", "--summary"], {
cwd: process.cwd(),
encoding: "utf8",
stdio: "pipe",
});
} catch (error) {
stdout = String((error as { stdout?: string }).stdout || "");
}
expect(stdout).toContain("scan assessment: FAIL");
expect(stdout).toContain("Expected winner");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects a single-engine 100-artifact run", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const payload = validAssessment();
payload.goal100Decision = "not-comparable: current and ik-traineddata were not both run";
payload.goal100.comparisonComplete = false;
payload.goal100.engines = [{ engine: "current", qualified: true, missRate: 0, reviewRate: 0 }];
const inputPath = writeAssessment(dir, payload);
expect(() =>
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`], {
cwd: process.cwd(),
stdio: "pipe",
}),
).toThrow();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects an unqualified winner", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const payload = validAssessment();
payload.goal100Decision = "not-qualified: 100-artifact winner failed quality gates";
payload.goal100.winnerQualified = false;
payload.goal100.engines[0].qualified = false;
payload.goal100.engines[0].reviewRate = 0.3;
const inputPath = writeAssessment(dir, payload);
expect(() =>
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`], {
cwd: process.cwd(),
stdio: "pipe",
}),
).toThrow();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects a winner with missing quality rates", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const payload = validAssessment();
delete (payload.goal100.engines[0] as { missRate?: number }).missRate;
delete (payload.goal100.engines[0] as { reviewRate?: number }).reviewRate;
const inputPath = writeAssessment(dir, payload);
let stdout = "";
try {
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary"], {
cwd: process.cwd(),
encoding: "utf8",
stdio: "pipe",
});
} catch (error) {
stdout = String((error as { stdout?: string }).stdout || "");
}
expect(stdout).toContain("scan assessment: FAIL");
expect(stdout).toContain("Winner missRate must be a finite number");
expect(stdout).toContain("Winner reviewRate must be a finite number");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects a winner with missing summary quality rates", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const payload = validAssessment();
delete (payload.goal100 as { winnerMissRate?: number }).winnerMissRate;
delete (payload.goal100 as { winnerReviewRate?: number }).winnerReviewRate;
const inputPath = writeAssessment(dir, payload);
let stdout = "";
try {
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary"], {
cwd: process.cwd(),
encoding: "utf8",
stdio: "pipe",
});
} catch (error) {
stdout = String((error as { stdout?: string }).stdout || "");
}
expect(stdout).toContain("scan assessment: FAIL");
expect(stdout).toContain("Winner summary missRate must be a finite number");
expect(stdout).toContain("Winner summary reviewRate must be a finite number");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("rejects a winner with missing speed timing", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const payload = validAssessment();
delete (payload.goal100 as { winnerActiveAverageMsPerParsed?: number }).winnerActiveAverageMsPerParsed;
delete (payload.goal100 as { winnerActiveProjectedMsFor100?: number }).winnerActiveProjectedMsFor100;
const inputPath = writeAssessment(dir, payload);
let stdout = "";
try {
execFileSync("node", ["scripts/validate-scan-assessment.cjs", `--input=${inputPath}`, "--summary"], {
cwd: process.cwd(),
encoding: "utf8",
stdio: "pipe",
});
} catch (error) {
stdout = String((error as { stdout?: string }).stdout || "");
}
expect(stdout).toContain("scan assessment: FAIL");
expect(stdout).toContain("Winner active average timing must be a positive finite number");
expect(stdout).toContain("Winner projected100 timing must be a positive finite number");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it("validates the latest assessment under a live-soak root", () => {
const dir = mkdtempSync(path.join(tmpdir(), "gaa-assessment-"));
try {
const older = path.join(dir, "2026-07-08T10-00-00");
const newer = path.join(dir, "2026-07-08T11-00-00");
mkdirSync(older);
mkdirSync(newer);
writeAssessment(older, { goal100Decision: "not-run: missing 100-artifact assessment" });
writeAssessment(newer, validAssessment());
const output = execFileSync("node", ["scripts/validate-scan-assessment.cjs", "--latest", `--root=${dir}`], {
cwd: process.cwd(),
encoding: "utf8",
});
const parsed = JSON.parse(output);
expect(parsed.ok).toBe(true);
expect(parsed.inputPath).toContain("2026-07-08T11-00-00");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
src/features/app/services/appControllerService.ts
+10 -7
View File
@@ -144,13 +144,16 @@ export async function captureSelectedSourceAction(
setTopbarStatus(`Capture in ${Math.round(delayMs / 1000)}s. Put Genshin in front and leave it visible.`);
}
const capture = await captureRepo.captureSource(selectedSourceId, delayMs, focusGenshin, options);
setLatestCapture(capture);
const ocrStatus = capture.ocrSkipped
? "OCR skipped for fast scan."
: capture.ocrTimedOut
? "OCR timed out; review sample needed."
: "OCR handoff is next.";
setTopbarStatus(`Captured ${capture.name} at ${capture.width}x${capture.height}. ${ocrStatus}`);
const quietArtifactScanCapture = options?.ocrMode === "artifact" && options?.ocrProfile === "fast" && options.omitFullFrame;
if (!quietArtifactScanCapture) {
setLatestCapture(capture);
const ocrStatus = capture.ocrSkipped
? "OCR skipped for fast scan."
: capture.ocrTimedOut
? "OCR timed out; review sample needed."
: "OCR handoff is next.";
setTopbarStatus(`Captured ${capture.name} at ${capture.width}x${capture.height}. ${ocrStatus}`);
}
return capture;
} catch (error) {
setTopbarStatus(error instanceof Error ? error.message : "Capture failed.");
src/features/scan/components/DiagnosticsView.tsx
+3 -2
View File
@@ -23,7 +23,7 @@ const appDiagnosisSections = [
"Genshin-Fenster erkennen, Smart Capture ausfuehren und fokussierte Artifact-Crops erzeugen.",
"Artifact-Felder deterministisch gegen das lokale Genshin-Datenpaket parsen.",
"Auto-Scan read-only aus der sichtbaren Inventory-Seite starten, inklusive Grid, Verifikation, Dedupe und Store.",
"Review-Samples, lokale Text-Lernregeln, GOOD Import/Export und Lock-Status im Store nutzen.",
"Review-Samples, lokale Lernregeln, GOOD Import/Export, Equipped-Footer und Lock-Status im Store nutzen.",
],
},
{
@@ -33,7 +33,7 @@ const appDiagnosisSections = [
items: [
"Paimon-Menue-Einstieg ist gebaut, aber live noch nicht mit 2/20/45 Limits validiert.",
"Native/IK-Tesseract ist nur als Benchmark-Pfad vorbereitet, noch nicht Standard.",
"Positive locked=true Probe an einem sicher gesperrten Artifact fehlt.",
"Positive locked=true Probe und erneuter Equipped-Footer-Livebeweis an bekannten Artifacts fehlen.",
"Empfehlungen bleiben Nebenfunktion, bis Scanner-Vertrauen und Review-Rate stabil genug sind.",
],
},
@@ -54,6 +54,7 @@ const appDiagnosisSections = [
icon: Target,
items: [
"Review-Corpus aus echten Samples vergroessern und mit `npm run eval` messbar halten.",
"Review-Export fuer Equipped-Footer und locked=true Kandidaten nutzen.",
"OCR-Benchmark gegen identische Crops fahren und erst danach Engine-Standard wechseln.",
"Paimon-Menue-Pfad live pruefen und bei Blockade sichtbar auf visible-inventory zurueckfallen.",
"Diagnose weiter als Operator-Cockpit halten: Live-Status, Evidenz und naechster sicherer Schritt.",
src/features/scan/components/hooks/useScanSummaryFooterModel.ts
+1 -1
View File
@@ -23,7 +23,7 @@ export function useScanSummaryFooterModel({
: `${scanSummary.attempted} Positionen bearbeitet, ${scanSummary.verified} Ansichten verifiziert, ${scanSummary.parsed} Artifact${scanSummary.parsed === 1 ? "" : "s"} gelesen in ${formatDuration(scanSummary.elapsedMs)} (${scanSummary.averageMsPerParsed || 0} ms/Artifact). Deine Sammlung: ${storedTotal ?? "?"} Artifacts.`;
const devCopy = devMode
? `clicked ${scanSummary.clicked} | attempted ${scanSummary.attempted} | verified ${scanSummary.verified} | parsed ${scanSummary.parsed} | misses ${scanSummary.misses} | pages ${scanSummary.pages} | active ${formatDuration(scanSummary.activeScanMs)} | flush ${scanSummary.writeFlushMs}ms | capture ${scanSummary.averageCaptureMs}ms | ocr ${scanSummary.averageOcrMs}ms | ${scanSummary.artifactsPerMinute}/min | active ${scanSummary.activeArtifactsPerMinute}/min | 100 projected ${formatDuration(scanSummary.projectedMsFor100)}`
? `clicked ${scanSummary.clicked} | attempted ${scanSummary.attempted} | verified ${scanSummary.verified} | parsed ${scanSummary.parsed} | misses ${scanSummary.misses} | pages ${scanSummary.pages} | active ${formatDuration(scanSummary.activeScanMs)} | flush ${scanSummary.writeFlushMs}ms | capture ${scanSummary.averageCaptureMs}ms | roundtrip ${scanSummary.averageCaptureRoundTripMs}ms | ocr ${scanSummary.averageOcrMs}ms | ${scanSummary.artifactsPerMinute}/min | active ${scanSummary.activeArtifactsPerMinute}/min | 100 projected ${formatDuration(scanSummary.projectedMsFor100)}`
: null;
return {
src/features/scan/components/modals/hooks/useScanDiagnosticsModalModel.ts
+4
View File
@@ -141,6 +141,8 @@ export function useScanDiagnosticsModalModel({
{ label: "avgMs", value: controller.autoScanStats.averageMsPerParsed },
{ label: "activeAvgMs", value: controller.autoScanStats.activeAverageMsPerParsed },
{ label: "avgCaptureMs", value: controller.autoScanStats.averageCaptureMs },
{ label: "avgCaptureRoundTripMs", value: controller.autoScanStats.averageCaptureRoundTripMs },
{ label: "avgCaptureRoundTripOverheadMs", value: controller.autoScanStats.averageCaptureRoundTripOverheadMs },
{ label: "captureP50Ms", value: controller.autoScanStats.captureP50Ms },
{ label: "captureP90Ms", value: controller.autoScanStats.captureP90Ms },
{ label: "avgOcrMs", value: controller.autoScanStats.averageOcrMs },
@@ -171,6 +173,8 @@ export function useScanDiagnosticsModalModel({
controller.autoScanStats.averageMsPerParsed,
controller.autoScanStats.activeAverageMsPerParsed,
controller.autoScanStats.averageCaptureMs,
controller.autoScanStats.averageCaptureRoundTripMs,
controller.autoScanStats.averageCaptureRoundTripOverheadMs,
controller.autoScanStats.captureP50Ms,
controller.autoScanStats.captureP90Ms,
controller.autoScanStats.averageOcrMs,
src/features/scan/hooks/scanViewControllerService.ts
+9
View File
@@ -51,6 +51,14 @@ export interface ScanActionContextInput {
source: string,
needsReview: boolean,
) => Promise<boolean>;
persistParsedArtifactsBatch?: (
items: Array<{
capture: CaptureResult | null;
parsed: ParsedArtifactCandidate;
source: string;
needsReview: boolean;
}>,
) => Promise<number>;
saveReviewSample: (
capture: CaptureResult | null,
parsed: ParsedArtifactCandidate | null,
@@ -107,6 +115,7 @@ export function createScanActionContext(input: ScanActionContextInput): ScanActi
appendDiagnosticEvent: input.appendDiagnosticEvent,
parseArtifact: input.parseArtifact,
persistParsedArtifact: input.persistParsedArtifact,
persistParsedArtifactsBatch: input.persistParsedArtifactsBatch,
shouldFlagArtifactForReview: (parsed) => (parsed ? shouldFlagArtifactForReview(parsed) : false),
saveReviewSample: input.saveReviewSample,
focusDashboard: input.focusDashboard,
src/features/scan/hooks/scanViewEntryActions.ts
+254
View File
@@ -0,0 +1,254 @@
import {
artifactTabClickTarget,
keyPressBlocked,
validateAutoScanEntryPreflight,
type ScanEntryMode,
} from "../../../lib/autoScanEntry";
import { wait } from "../../../lib/scanReviewUtils";
import {
summarizeClickResult,
summarizeKeyPressResult,
type createScanDiagnosticEvent,
} from "../../../lib/scanDiagnosticsLog";
import type { AutomationRepositoryPort } from "../../../infrastructure/repositories/rendererBridgeRepositories";
import type { CaptureOptions, CaptureResult } from "../../../types/global";
export interface PrepareAutoScanEntryInput {
mode: ScanEntryMode;
automationRepo: AutomationRepositoryPort;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
appendAutomationLog: (line: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
}
export async function prepareAutoScanEntry({
mode,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
}: PrepareAutoScanEntryInput) {
if (mode === "visible-inventory") {
const capture = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (candidate) => validateAutoScanEntryPreflight(candidate).ok,
});
appendDiagnosticEvent({
phase: "entry-visible",
severity: capture ? "ok" : "error",
message: capture && validateAutoScanEntryPreflight(capture).ok
? "Visible inventory preflight capture ready."
: "Visible inventory preflight capture failed.",
capture,
});
return capture;
}
if (mode === "direct-inventory") {
return tryInventoryEntrySequence({
label: "direct",
sendEscapeFirst: false,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
}
if (mode === "auto-entry") {
const directCapture = await tryInventoryEntrySequence({
label: "auto-direct",
sendEscapeFirst: false,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
const directPreflight = validateAutoScanEntryPreflight(directCapture);
if (directPreflight.ok) return directCapture;
appendAutomationLog(`auto-entry direct path failed: ${directPreflight.reason}`);
appendDiagnosticEvent({
phase: "entry-fallback",
severity: "info",
message: `Direct inventory entry did not reach an artifact detail card. Trying IK fallback. ${directPreflight.reason}`,
capture: directCapture,
});
}
return tryInventoryEntrySequence({
label: "paimon",
sendEscapeFirst: true,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
}
async function tryInventoryEntrySequence({
label,
sendEscapeFirst,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
}: {
label: string;
sendEscapeFirst: boolean;
automationRepo: AutomationRepositoryPort;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
appendAutomationLog: (line: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
}) {
if (sendEscapeFirst) {
const escapeResult = await automationRepo.keyPress?.("ESC");
appendAutomationLog(`${label} entry key ESC: ${escapeResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(escapeResult) ? "error" : "ok",
message: `${label} entry key ESC`,
details: summarizeKeyPressResult(escapeResult),
});
if (keyPressBlocked(escapeResult)) return null;
const menuProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 750,
predicate: (capture) => Boolean(capture),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: menuProbe ? "info" : "warn",
message: `${label} capture after ESC step.`,
capture: menuProbe,
});
if (menuProbe?.paimonMenu?.present) {
const closeMenuResult = await automationRepo.keyPress?.("ESC");
appendAutomationLog(`${label} entry key ESC close menu: ${closeMenuResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(closeMenuResult) ? "error" : "ok",
message: `${label} entry key ESC close menu`,
details: summarizeKeyPressResult(closeMenuResult),
});
if (keyPressBlocked(closeMenuResult)) return menuProbe;
const worldProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 750,
predicate: (capture) => Boolean(capture && !capture.paimonMenu?.present),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: worldProbe ? "info" : "warn",
message: `${label} capture after closing Paimon menu.`,
capture: worldProbe,
});
if (worldProbe?.paimonMenu?.present) {
appendAutomationLog(`${label} entry stopped: Paimon menu still visible after second ESC`);
return worldProbe;
}
}
}
const inventoryResult = await automationRepo.keyPress?.("B");
appendAutomationLog(`${label} entry key B: ${inventoryResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(inventoryResult) ? "error" : "ok",
message: `${label} entry key B`,
details: summarizeKeyPressResult(inventoryResult),
});
if (keyPressBlocked(inventoryResult)) return null;
const tabProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 1200,
predicate: (capture) => Boolean(capture && !capture.paimonMenu?.present && capture.inventoryGrid && capture.inventoryGrid.source !== "missing"),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: tabProbe ? "info" : "warn",
message: `${label} capture after Inventory-key step.`,
capture: tabProbe,
});
if (tabProbe?.paimonMenu?.present) {
appendAutomationLog(`${label} entry stopped: Paimon menu still visible after Inventory key`);
appendDiagnosticEvent({
phase: "entry-capture",
severity: "info",
message: `${label} entry stopped before tab click because Paimon menu is still visible.`,
capture: tabProbe,
});
return tabProbe;
}
if (!tabProbe?.inventoryGrid || tabProbe.inventoryGrid.source === "missing") return tabProbe;
const target = artifactTabClickTarget(tabProbe);
appendAutomationLog(`${label} entry artifact tab -> ${target.x},${target.y}`);
const click = await automationRepo.clickScreen(target.x, target.y);
appendDiagnosticEvent({
phase: "entry-click",
severity: click.inputBlocked || click.clicked === false || click.moved === false ? "warn" : "ok",
message: `${label} artifact tab click at ${target.x},${target.y}`,
details: summarizeClickResult(click),
capture: tabProbe,
});
if (click.inputBlocked || click.clicked === false) return null;
const gridProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (capture) => Boolean(capture?.inventoryGrid && capture.inventoryGrid.source !== "missing"),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: gridProbe ? "info" : "warn",
message: `${label} capture after artifact-tab click before first tile selection.`,
capture: gridProbe,
});
const firstTarget = gridProbe?.inventoryGrid?.centers?.[0];
if (!firstTarget) return gridProbe;
appendAutomationLog(`${label} entry first artifact tile -> ${firstTarget.x},${firstTarget.y}`);
const firstTileClick = await automationRepo.clickScreen(firstTarget.x, firstTarget.y);
appendDiagnosticEvent({
phase: "entry-click",
severity: firstTileClick.inputBlocked || firstTileClick.clicked === false || firstTileClick.moved === false ? "warn" : "ok",
message: `${label} first artifact tile click at ${firstTarget.x},${firstTarget.y}`,
details: summarizeClickResult(firstTileClick),
capture: gridProbe,
});
if (firstTileClick.inputBlocked || firstTileClick.clicked === false) return null;
const finalCapture = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (capture) => validateAutoScanEntryPreflight(capture).ok,
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: finalCapture ? "info" : "warn",
message: `${label} final capture after first artifact selection.`,
capture: finalCapture,
});
return finalCapture;
}
async function waitForEntryCapture({
captureFastSelectedSource,
timeoutMs,
pollMs = 150,
predicate,
}: {
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
timeoutMs: number;
pollMs?: number;
predicate: (capture: CaptureResult | null) => boolean;
}) {
const startedAt = Date.now();
let latest: CaptureResult | null = null;
while (Date.now() - startedAt <= timeoutMs) {
latest = await captureFastSelectedSource(0, true);
if (predicate(latest)) return latest;
const remaining = timeoutMs - (Date.now() - startedAt);
if (remaining <= 0) break;
await wait(Math.min(pollMs, remaining));
}
return latest;
}
src/features/scan/hooks/scanViewReviewHelpers.ts
+78 -7
View File
@@ -110,7 +110,7 @@ function shouldRecoverIntoStore(existing: StoredArtifactRecord | undefined, inco
}
function getDefaultScannerRules(loadedRules: { rules?: ScannerLearningRules } | null | undefined): ScannerLearningRules {
return { textReplacements: { ...(loadedRules?.rules?.textReplacements ?? {}) } };
return mergeLearningRulePayloads({}, loadedRules?.rules);
}
async function loadReviewSamplesAndRecover(context: ReviewStateContext, rules: ScannerLearningRules, limit = REVIEW_SAMPLE_LIMIT_INITIAL) {
@@ -213,17 +213,47 @@ export async function mergeLearningRules(
context: ReviewStateContext,
) {
if (!nextRules || countScannerLearningRules(nextRules) === 0) return null;
const merged = {
textReplacements: {
...currentRules.textReplacements,
...(nextRules.textReplacements ?? {}),
},
};
const merged = mergeLearningRulePayloads(currentRules, nextRules);
context.setScannerLearningRules(merged);
const result = await context.learningRepo?.saveRules?.(merged).catch(() => null);
return { result, merged };
}
function mergeLearningRulePayloads(
current: Partial<ScannerLearningRules> | null | undefined,
next: Partial<ScannerLearningRules> | null | undefined,
): ScannerLearningRules {
return {
textReplacements: {
...(current?.textReplacements ?? {}),
...(next?.textReplacements ?? {}),
},
fieldAliases: mergeNestedRuleMap(current?.fieldAliases, next?.fieldAliases),
constrainedFixes: {
...(current?.constrainedFixes ?? {}),
...(next?.constrainedFixes ?? {}),
},
cropAdjustments: {
...(current?.cropAdjustments ?? {}),
...(next?.cropAdjustments ?? {}),
},
uiProfileAdjustments: {
...(current?.uiProfileAdjustments ?? {}),
...(next?.uiProfileAdjustments ?? {}),
},
};
}
function mergeNestedRuleMap(
current: Record<string, Record<string, string>> | undefined,
next: Record<string, Record<string, string>> | undefined,
) {
const merged: Record<string, Record<string, string>> = {};
for (const [field, aliases] of Object.entries(current ?? {})) merged[field] = { ...(aliases ?? {}) };
for (const [field, aliases] of Object.entries(next ?? {})) merged[field] = { ...(merged[field] ?? {}), ...(aliases ?? {}) };
return merged;
}
export async function persistParsedArtifact(
capture: CaptureResult | null,
parsed: ParsedArtifactCandidate,
@@ -256,6 +286,47 @@ export async function persistParsedArtifact(
}
}
export async function persistParsedArtifactsBatch(
items: Array<{
capture: CaptureResult | null;
parsed: ParsedArtifactCandidate;
source: string;
needsReview: boolean;
}>,
context: ReviewStateContext,
) {
const { artifactRepo, onStoredArtifactsChanged, setStoredTotal, appendAutomationLog } = context;
if (!artifactRepo?.saveMany || items.length === 0) return 0;
const records: StoredArtifactRecord[] = [];
for (const item of items) {
const rejection = captureRejectionReason(item.capture, item.parsed);
if (rejection) {
appendAutomationLog(`persist skip: ${rejection}`);
continue;
}
if (!shouldPersistParsedArtifact(item.parsed, item.needsReview)) {
appendAutomationLog(`persist skip: parsed artifact bleibt vorerst nur Review (${item.parsed.name})`);
continue;
}
records.push(toStoredArtifact(item.parsed, item.source, item.needsReview, item.capture?.locked));
}
if (records.length === 0) return 0;
try {
const result = await artifactRepo.saveMany(records);
if (result?.ok) {
setStoredTotal(result.total);
void onStoredArtifactsChanged?.();
return records.length;
}
return 0;
} catch {
return 0;
}
}
export async function saveReviewSample(
capture: CaptureResult | null,
parsed: ParsedArtifactCandidate | null,
src/features/scan/hooks/scanViewScanActions.ts
+10 -238
View File
@@ -1,11 +1,11 @@
import { automationBlockReason, requiresAdminForAutomation } from "../../../lib/automationPlanner";
import { artifactTabClickTarget, keyPressBlocked, validateAutoScanEntryPreflight, type ScanEntryMode } from "../../../lib/autoScanEntry";
import { validateAutoScanEntryPreflight, type ScanEntryMode } from "../../../lib/autoScanEntry";
import { captureRejectionReason } from "../../../lib/scannerCaptureQuality";
import { runAutoScanLoop } from "../../../lib/autoScanLoop";
import { addCaptureTiming, clampScanLimit, emptyAutoScanStats, resolveScanTargetCount, updateScanTiming, type AutoScanStats, type ScanSummary } from "../../../lib/scannerSession";
import { getAutoReviewReason, wait } from "../../../lib/scanReviewUtils";
import { summarizeClickResult, summarizeKeyPressResult, type createScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import type { AutomationRepositoryPort, RuntimeRepositoryPort } from "../../../infrastructure/repositories/rendererBridgeRepositories";
import { type createScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import type { AutomationRepositoryPort, RuntimeRepositoryPort } from "../../../infrastructure/repositories/rendererBridgeRepositoryTypes";
import type {
AutomationGuard,
BooleanResult,
@@ -17,8 +17,8 @@ import type {
ScrollResult,
} from "../../../types/global";
import type { ParsedArtifactCandidate } from "../../../lib/artifactOcrParser";
import type { MutableRefObject } from "react";
import type { Dispatch, SetStateAction } from "react";
import type { Dispatch, MutableRefObject, SetStateAction } from "react";
import { prepareAutoScanEntry } from "./scanViewEntryActions";
export interface ScanActionContext {
autoScanRunning: boolean;
@@ -43,6 +43,7 @@ export interface ScanActionContext {
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
parseArtifact: (capture: CaptureResult | null) => ParsedArtifactCandidate | null;
persistParsedArtifact: (capture: CaptureResult | null, parsed: ParsedArtifactCandidate, source: string, needsReview: boolean) => Promise<boolean>;
persistParsedArtifactsBatch?: (items: Array<{ capture: CaptureResult | null; parsed: ParsedArtifactCandidate; source: string; needsReview: boolean }>) => Promise<number>;
saveReviewSample: (capture: CaptureResult | null, parsed: ParsedArtifactCandidate | null, reason?: string) => Promise<BooleanResult | null>;
shouldFlagArtifactForReview: (parsed: ParsedArtifactCandidate | null) => boolean;
focusDashboard: () => Promise<void>;
@@ -74,6 +75,7 @@ export async function runAutoReviewScan(context: ScanActionContext): Promise<voi
captureSelectedSource,
parseArtifact,
persistParsedArtifact,
persistParsedArtifactsBatch,
saveReviewSample,
shouldFlagArtifactForReview,
focusDashboard,
@@ -128,7 +130,7 @@ export async function runAutoReviewScan(context: ScanActionContext): Promise<voi
stats.attempted++;
stats.verified++;
stats.parsed++;
addCaptureTiming(stats, capture.timings);
addCaptureTiming(stats, capture.timings, capture.elapsedMs);
const reason = getAutoReviewReason(capture, parsed);
const needsReview = shouldFlagArtifactForReview(parsed);
@@ -183,6 +185,7 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
captureFastSelectedSource,
parseArtifact,
persistParsedArtifact,
persistParsedArtifactsBatch,
saveReviewSample,
shouldFlagArtifactForReview,
scanLimit: configuredScanLimit,
@@ -393,6 +396,7 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
captureFastSelectedSource,
parseArtifact,
persistParsedArtifact,
persistParsedArtifactsBatch,
saveReviewSample,
getAutoReviewReason,
shouldFlagArtifactForReview,
@@ -424,235 +428,3 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
gridLabel: result.gridLabel,
});
}
async function prepareAutoScanEntry({
mode,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
}: {
mode: ScanEntryMode;
automationRepo: AutomationRepositoryPort;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
appendAutomationLog: (line: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
}) {
if (mode === "visible-inventory") {
const capture = await captureFastSelectedSource(0, true);
appendDiagnosticEvent({
phase: "entry-visible",
severity: capture ? "ok" : "error",
message: capture ? "Visible inventory preflight capture ready." : "Visible inventory preflight capture failed.",
capture,
});
return capture;
}
if (mode === "direct-inventory") {
return tryInventoryEntrySequence({
label: "direct",
sendEscapeFirst: false,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
}
if (mode === "auto-entry") {
const directCapture = await tryInventoryEntrySequence({
label: "auto-direct",
sendEscapeFirst: false,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
const directPreflight = validateAutoScanEntryPreflight(directCapture);
if (directPreflight.ok) return directCapture;
appendAutomationLog(`auto-entry direct path failed: ${directPreflight.reason}`);
appendDiagnosticEvent({
phase: "entry-fallback",
severity: "info",
message: `Direct inventory entry did not reach an artifact detail card. Trying IK fallback. ${directPreflight.reason}`,
capture: directCapture,
});
}
return tryInventoryEntrySequence({
label: "paimon",
sendEscapeFirst: true,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
}
async function tryInventoryEntrySequence({
label,
sendEscapeFirst,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
}: {
label: string;
sendEscapeFirst: boolean;
automationRepo: AutomationRepositoryPort;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
appendAutomationLog: (line: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
}) {
if (sendEscapeFirst) {
const escapeResult = await automationRepo.keyPress?.("ESC");
appendAutomationLog(`${label} entry key ESC: ${escapeResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(escapeResult) ? "error" : "ok",
message: `${label} entry key ESC`,
details: summarizeKeyPressResult(escapeResult),
});
if (keyPressBlocked(escapeResult)) return null;
const menuProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 750,
predicate: (capture) => Boolean(capture),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: menuProbe ? "info" : "warn",
message: `${label} capture after ESC step.`,
capture: menuProbe,
});
if (menuProbe?.paimonMenu?.present) {
const closeMenuResult = await automationRepo.keyPress?.("ESC");
appendAutomationLog(`${label} entry key ESC close menu: ${closeMenuResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(closeMenuResult) ? "error" : "ok",
message: `${label} entry key ESC close menu`,
details: summarizeKeyPressResult(closeMenuResult),
});
if (keyPressBlocked(closeMenuResult)) return menuProbe;
const worldProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 750,
predicate: (capture) => Boolean(capture && !capture.paimonMenu?.present),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: worldProbe ? "info" : "warn",
message: `${label} capture after closing Paimon menu.`,
capture: worldProbe,
});
if (worldProbe?.paimonMenu?.present) {
appendAutomationLog(`${label} entry stopped: Paimon menu still visible after second ESC`);
return worldProbe;
}
}
}
const inventoryResult = await automationRepo.keyPress?.("B");
appendAutomationLog(`${label} entry key B: ${inventoryResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(inventoryResult) ? "error" : "ok",
message: `${label} entry key B`,
details: summarizeKeyPressResult(inventoryResult),
});
if (keyPressBlocked(inventoryResult)) return null;
const tabProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 1200,
predicate: (capture) => Boolean(capture && !capture.paimonMenu?.present && capture.inventoryGrid && capture.inventoryGrid.source !== "missing"),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: tabProbe ? "info" : "warn",
message: `${label} capture after Inventory-key step.`,
capture: tabProbe,
});
if (tabProbe?.paimonMenu?.present) {
appendAutomationLog(`${label} entry stopped: Paimon menu still visible after Inventory key`);
appendDiagnosticEvent({
phase: "entry-capture",
severity: "info",
message: `${label} entry stopped before tab click because Paimon menu is still visible.`,
capture: tabProbe,
});
return tabProbe;
}
if (!tabProbe?.inventoryGrid || tabProbe.inventoryGrid.source === "missing") return tabProbe;
const target = artifactTabClickTarget(tabProbe);
appendAutomationLog(`${label} entry artifact tab -> ${target.x},${target.y}`);
const click = await automationRepo.clickScreen(target.x, target.y);
appendDiagnosticEvent({
phase: "entry-click",
severity: click.inputBlocked || click.clicked === false || click.moved === false ? "warn" : "ok",
message: `${label} artifact tab click at ${target.x},${target.y}`,
details: summarizeClickResult(click),
capture: tabProbe,
});
if (click.inputBlocked || click.clicked === false) return null;
const gridProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (capture) => Boolean(capture?.inventoryGrid && capture.inventoryGrid.source !== "missing"),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: gridProbe ? "info" : "warn",
message: `${label} capture after artifact-tab click before first tile selection.`,
capture: gridProbe,
});
const firstTarget = gridProbe?.inventoryGrid?.centers?.[0];
if (!firstTarget) return gridProbe;
appendAutomationLog(`${label} entry first artifact tile -> ${firstTarget.x},${firstTarget.y}`);
const firstTileClick = await automationRepo.clickScreen(firstTarget.x, firstTarget.y);
appendDiagnosticEvent({
phase: "entry-click",
severity: firstTileClick.inputBlocked || firstTileClick.clicked === false || firstTileClick.moved === false ? "warn" : "ok",
message: `${label} first artifact tile click at ${firstTarget.x},${firstTarget.y}`,
details: summarizeClickResult(firstTileClick),
capture: gridProbe,
});
if (firstTileClick.inputBlocked || firstTileClick.clicked === false) return null;
const finalCapture = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (capture) => validateAutoScanEntryPreflight(capture).ok,
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: finalCapture ? "info" : "warn",
message: `${label} final capture after first artifact selection.`,
capture: finalCapture,
});
return finalCapture;
}
async function waitForEntryCapture({
captureFastSelectedSource,
timeoutMs,
pollMs = 150,
predicate,
}: {
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
timeoutMs: number;
pollMs?: number;
predicate: (capture: CaptureResult | null) => boolean;
}) {
const startedAt = Date.now();
let latest: CaptureResult | null = null;
while (Date.now() - startedAt <= timeoutMs) {
latest = await captureFastSelectedSource(0, true);
if (predicate(latest)) return latest;
const remaining = timeoutMs - (Date.now() - startedAt);
if (remaining <= 0) break;
await wait(Math.min(pollMs, remaining));
}
return latest;
}
src/features/scan/hooks/useScanGoodInterop.ts
+63
View File
@@ -0,0 +1,63 @@
import { useCallback } from "react";
import { goodDatabaseToStoredArtifacts, type GoodImportDatabase, storedArtifactsToGood } from "../../../lib/goodInterop";
import type { ArtifactRepositoryPort, ScanExportPort } from "../../../infrastructure/repositories/rendererBridgeRepositoryTypes";
import type { StoredArtifactRecord } from "../../../types/storage";
interface UseScanGoodInteropInput {
artifactRepo?: ArtifactRepositoryPort;
exportRepo?: ScanExportPort;
onStoredArtifactsChanged?: () => Promise<void>;
setStoredTotal: (value: number | null) => void;
bridgeReady: boolean;
}
export function useScanGoodInterop({
artifactRepo,
exportRepo,
onStoredArtifactsChanged,
setStoredTotal,
bridgeReady,
}: UseScanGoodInteropInput) {
const canGoodInterop = bridgeReady && Boolean(artifactRepo?.loadAll) && Boolean(artifactRepo?.saveMany);
const exportGoodFromStore = useCallback(async () => {
if (!artifactRepo?.loadAll || !exportRepo?.exportGood) return { ok: false, count: 0 };
const loaded = await artifactRepo.loadAll();
const records = loaded.artifacts ?? [];
const good = storedArtifactsToGood(records);
const result = await exportRepo.exportGood(good);
return { ok: Boolean(result.ok), path: result.path, count: good.artifacts.length };
}, [artifactRepo, exportRepo]);
const importGoodArtifacts = useCallback(async (records: StoredArtifactRecord[]) => {
if (!artifactRepo?.saveMany || records.length === 0) return { ok: false, added: 0, updated: 0 };
const result = await artifactRepo.saveMany(records);
if (typeof result.total === "number") setStoredTotal(result.total);
await onStoredArtifactsChanged?.();
return { ok: Boolean(result.ok), added: result.added ?? 0, updated: result.updated ?? 0 };
}, [artifactRepo, onStoredArtifactsChanged, setStoredTotal]);
const importGoodFromFile = useCallback(async () => {
if (!exportRepo?.importGoodFile || !artifactRepo?.saveMany) {
return { ok: false, added: 0, updated: 0, count: 0, error: "GOOD import is unavailable." };
}
const fileResult = await exportRepo.importGoodFile();
if (fileResult.canceled) return { ok: false, added: 0, updated: 0, count: 0, canceled: true };
if (!fileResult.ok) {
return { ok: false, added: 0, updated: 0, count: 0, path: fileResult.path, error: fileResult.error };
}
const records = goodDatabaseToStoredArtifacts(fileResult.database as GoodImportDatabase);
if (records.length === 0) {
return { ok: false, added: 0, updated: 0, count: 0, path: fileResult.path, error: "No valid GOOD artifacts found." };
}
const saved = await importGoodArtifacts(records);
return { ...saved, count: records.length, path: fileResult.path };
}, [artifactRepo, exportRepo, importGoodArtifacts]);
return {
canGoodInterop,
exportGoodFromStore,
importGoodFromFile,
importGoodArtifacts,
};
}
src/features/scan/hooks/useScanViewActions.ts
+19 -3
View File
@@ -5,6 +5,7 @@ import {
initializeLearningState,
loadReviewQueue as loadReviewQueueFromRepo,
persistParsedArtifact as persistParsedArtifactHelper,
persistParsedArtifactsBatch as persistParsedArtifactsBatchHelper,
saveReviewSample as saveReviewSampleHelper,
} from "./scanViewReviewHelpers";
import { createReviewContext, createScanActionContext } from "./scanViewControllerService";
@@ -186,6 +187,15 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
[reviewContext],
);
const parseArtifactsAndPersistBatch = useCallback(
async function parseArtifactsAndPersistBatch(
items: Array<{ capture: CaptureResult | null; parsed: ParsedArtifactCandidate; source: string; needsReview: boolean }>,
) {
return persistParsedArtifactsBatchHelper(items, reviewContext);
},
[reviewContext],
);
const handleSaveReviewSample = useCallback(
async function handleSaveReviewSample(
capture: CaptureResult | null = latestCapture,
@@ -227,6 +237,7 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
appendDiagnosticEvent,
parseArtifact,
persistParsedArtifact: parseArtifactAndPersist,
persistParsedArtifactsBatch: parseArtifactsAndPersistBatch,
saveReviewSample: handleSaveReviewSample,
focusDashboard,
captureSelectedSource: (delayMs = 0, focusGenshin = false, options) => captureSelectedSource(delayMs, focusGenshin, {
@@ -265,6 +276,7 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
appendDiagnosticEvent,
parseArtifact,
parseArtifactAndPersist,
parseArtifactsAndPersistBatch,
handleSaveReviewSample,
focusDashboard,
captureSelectedSource,
@@ -317,13 +329,17 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
severity: visibleInventoryReady ? "ok" : "info",
message: visibleInventoryReady
? "Artifact inventory detail view already visible; starting scan directly."
: "Artifact detail view is not ready; trying direct inventory entry, then Inventory Kamera fallback.",
: "Artifact detail view is not ready; guided scan waits for a visible artifact detail card instead of navigating.",
capture: preflightCapture,
});
if (!visibleInventoryReady) {
setReviewStatus("Auto-Scan wartet: Bitte Artifact-Inventar mit sichtbarer Detailkarte oeffnen und erneut starten.");
return;
}
await runVisibleGridScanAction(scanActionContext, {
scanLimit: options.scanLimit,
scanEntryMode: visibleInventoryReady ? "visible-inventory" : "auto-entry",
processInitialSelection: visibleInventoryReady,
scanEntryMode: "visible-inventory",
processInitialSelection: true,
ocrEngine: options.ocrEngine,
});
}, [
src/features/scan/hooks/useScanViewController.ts
+14 -38
View File
@@ -10,6 +10,7 @@ import {
parseLearnedArtifact as parseLearnedArtifactHelper,
} from "./scanViewReviewHelpers";
import { useScanRuntimeInfo } from "./useScanRuntimeInfo";
import { useScanGoodInterop } from "./useScanGoodInterop";
import { useScanSnapshotPublisher } from "./useScanSnapshotPublisher";
import { useScanViewActions } from "./useScanViewActions";
import { useScanViewStateSync } from "./useScanViewStateSync";
@@ -17,8 +18,6 @@ import { emptyAutoScanStats, resolveScanTargetCount, type AutoScanStats, type Sc
import { createScanDiagnosticEvent, summarizeClickResult, type ScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import type { ScanViewProps, ScanViewControllerResult } from "../types";
import type { CaptureResult, ClickResult, ReviewSampleRecord } from "../../../types/global";
import type { StoredArtifactRecord } from "../../../types/storage";
import { goodDatabaseToStoredArtifacts, type GoodImportDatabase, storedArtifactsToGood } from "../../../lib/goodInterop";
import { createRendererRepositories } from "../../../infrastructure/repositories/rendererBridgeRepositories";
export function useScanViewController({
@@ -79,7 +78,7 @@ export function useScanViewController({
const canAutoScan = bridgeReady && Boolean(automationRepo?.clickScreen) && Boolean(automationRepo?.scrollScreen);
const reviewAnalysis = useMemo(() => analyzeReviewSamples(reviewSamples), [reviewSamples]);
const learningRuleCount = countScannerLearningRules(scannerLearningRules);
const detectedInventoryCount = latestCapture?.inventoryCount?.current ?? 0;
const detectedInventoryCount = latestCapture?.inventoryCount?.total ?? latestCapture?.inventoryCount?.current ?? 0;
const activeTargetCount = autoScanRunning
? resolveScanTargetCount(scanLimit, detectedInventoryCount)
: scanSummary?.targetCount ?? resolveScanTargetCount(scanLimit, detectedInventoryCount);
@@ -175,41 +174,18 @@ export function useScanViewController({
setReviewQueueOpen,
});
const canGoodInterop = bridgeReady && Boolean(artifactRepo?.loadAll) && Boolean(artifactRepo?.saveMany);
const exportGoodFromStore = useCallback(async () => {
if (!artifactRepo?.loadAll || !exportRepo?.exportGood) return { ok: false, count: 0 };
const loaded = await artifactRepo.loadAll();
const records = loaded.artifacts ?? [];
const good = storedArtifactsToGood(records);
const result = await exportRepo.exportGood(good);
return { ok: Boolean(result.ok), path: result.path, count: good.artifacts.length };
}, [artifactRepo, exportRepo]);
const importGoodArtifacts = useCallback(async (records: StoredArtifactRecord[]) => {
if (!artifactRepo?.saveMany || records.length === 0) return { ok: false, added: 0, updated: 0 };
const result = await artifactRepo.saveMany(records);
if (typeof result.total === "number") setStoredTotal(result.total);
await onStoredArtifactsChanged?.();
return { ok: Boolean(result.ok), added: result.added ?? 0, updated: result.updated ?? 0 };
}, [artifactRepo, onStoredArtifactsChanged]);
const importGoodFromFile = useCallback(async () => {
if (!exportRepo?.importGoodFile || !artifactRepo?.saveMany) {
return { ok: false, added: 0, updated: 0, count: 0, error: "GOOD import is unavailable." };
}
const fileResult = await exportRepo.importGoodFile();
if (fileResult.canceled) return { ok: false, added: 0, updated: 0, count: 0, canceled: true };
if (!fileResult.ok) {
return { ok: false, added: 0, updated: 0, count: 0, path: fileResult.path, error: fileResult.error };
}
const records = goodDatabaseToStoredArtifacts(fileResult.database as GoodImportDatabase);
if (records.length === 0) {
return { ok: false, added: 0, updated: 0, count: 0, path: fileResult.path, error: "No valid GOOD artifacts found." };
}
const saved = await importGoodArtifacts(records);
return { ...saved, count: records.length, path: fileResult.path };
}, [artifactRepo, exportRepo, importGoodArtifacts]);
const {
canGoodInterop,
exportGoodFromStore,
importGoodFromFile,
importGoodArtifacts,
} = useScanGoodInterop({
artifactRepo,
exportRepo,
onStoredArtifactsChanged,
setStoredTotal,
bridgeReady,
});
useScanViewStateSync({
artifactRepo,
src/features/scan/hooks/useScanViewStateSync.ts
+1 -2
View File
@@ -20,7 +20,7 @@ export function useScanViewStateSync({
setStoredTotal,
}: ScanViewStateSyncInput) {
useEffect(() => {
const detectedCount = latestCapture?.inventoryCount?.current ?? 0;
const detectedCount = latestCapture?.inventoryCount?.total ?? latestCapture?.inventoryCount?.current ?? 0;
if (!scanLimitTouched && detectedCount > 0) {
setScanLimit(clampScanLimit(detectedCount));
}
@@ -32,4 +32,3 @@ export function useScanViewStateSync({
}).catch(() => undefined);
}, [artifactRepo, setStoredTotal]);
}
src/lib/artifactOcrParser.test.ts
+85
View File
@@ -140,6 +140,51 @@ describe("parseArtifactCandidate", () => {
expect(parsed?.fields.mainValue.source).toBe("derived");
});
it("derives percent main stat values when slot rules disallow flat HP ATK or DEF", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Moonlit Offering's Final Hour",
"artifact-main-stat-label": "HP",
"artifact-level": "+20",
"artifact-substats": "+ ATK+19\n- Energy Recharge+6.5%\n+ CRIT DMG+18.7%\n- DEF+53",
}));
expect(parsed?.slot).toBe("Sands of Eon");
expect(parsed?.mainStat).toBe("HP%");
expect(parsed?.mainValue).toBe("46.6%");
expect(parsed?.fields.mainStat.source).toBe("derived");
expect(parsed?.fields.mainValue.source).toBe("derived");
});
it("recovers a known piece from a truncated Viridescent Determination OCR tail", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Determmation oT",
"artifact-main-stat-label": "Energy Recharge",
"artifact-level": "+20",
"artifact-substats": "- ATK+5.8%\n- Elemental Mastery+37\nHP+11.7%\n+ ATK+54",
}));
expect(parsed?.name).toBe("Viridescent Venerer's Determination");
expect(parsed?.slot).toBe("Sands of Eon");
expect(parsed?.setName).toBe("Viridescent Venerer");
expect(parsed?.mainStat).toBe("Energy Recharge");
expect(parsed?.mainValue).toBe("51.8%");
});
it("recovers long Disenchantment piece names from truncated OCR fragments", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Postintty That Ceased Upon",
"artifact-main-stat-label": "DEF",
"artifact-level": "+0",
"artifact-substats": "+ DEF+21\n+ Energy Recharge+4.5%\n+ CRIT Rate+3.1%\n- ATK+14",
}));
expect(parsed?.name).toBe("Moment That Ceased Upon Waking From Grand Dreams");
expect(parsed?.slot).toBe("Sands of Eon");
expect(parsed?.setName).toBe("Disenchantment in Deep Shadow");
expect(parsed?.mainStat).toBe("DEF%");
expect(parsed?.mainValue).toBe("8.7%");
});
it("derives slot and set from the piece name when fast auto-scan skips slot OCR", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Gladiator's Nostalgia",
@@ -265,6 +310,46 @@ describe("parseArtifactCandidate", () => {
expect(parsed?.equipped).toBe("Bennett");
});
it("recognizes equipped characters when OCR splits the footer label and name", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-title": "Gladiator's Nostalgia\nFlower of Life",
"artifact-main-stat": "HP\n4,780",
"artifact-substats": "+ Energy Recharge+11.0%\n+ ATK+9.9%\n+ HP+14.6%\n+ CRIT DMG+12.4%",
"artifact-set-effects": "Gladiator's Finale:\n2-Piece Set: ATK +18%",
"artifact-footer": "Equipped:\nBennett",
}));
expect(parsed?.equipped).toBe("Bennett");
expect(parsed?.fields.equipped.source).toBe("fallback");
});
it("does not persist unknown one-letter equipped fragments as characters", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-title": "Vessel of Plenty",
"artifact-main-stat": "Goblet of Eonothem\nDLT\n58.3%",
"artifact-substats": "- DEF+58\n- Elemental Mastery+47\n+ CRIT Rate+5.8%\n+ HP+299",
"artifact-footer": "Equipped: I -",
}));
expect(parsed?.equipped).toBe("Not detected");
expect(parsed?.fields.equipped.source).toBe("missing");
});
it("normalizes equipped footer trailing fragments to a known character", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Pristine Plume of the Blessed",
"artifact-slot": "Plume of Death",
"artifact-main-stat-label": "ATK",
"artifact-level": "+20",
"artifact-substats": "+ HP+16.9%\n- ATK+8.7%\n+ CRIT DMG+13.2%\n+ Elemental Mastery+21",
"artifact-set-effects": "2-Piece Set: Energy Recharge +20%.",
"artifact-footer": "Equipped: Linnea l",
}));
expect(parsed?.equipped).toBe("Linnea");
expect(parsed?.fields.equipped.source).toBe("fallback");
});
it("recognizes ATK percent main stats from OCR text on non-fixed slots", () => {
const sands = parseArtifactCandidate(captureFromOcr({
"artifact-title": "Myths of the Night Realm\nSands of Eon",
src/lib/artifactOcrParser.ts
+35 -1
View File
@@ -82,6 +82,7 @@ export function parseArtifactCandidate(capture: CaptureResult | null): ParsedArt
const parsedLevel = levelField.value ? Number.parseInt(levelField.value, 10) : null;
const level = parsedLevel ?? 0;
let mainStatField = inferMainStat(slotField.value, mainText);
mainStatField = promoteSlotPercentMainStat(slotField.value, mainStatField);
let mainValueField = findMainValue(mainText, mainStatField.value, slotField.value, parsedLevel);
if (!mainStatField.value && mainValueField.value) {
const inferredFromValue = inferMainStatFromValue(slotField.value, mainValueField.value, mainText, parsedLevel);
@@ -227,6 +228,8 @@ function parseSetName(setText: string, artifactName: ParsedField): ParsedField {
const setFromText = fuzzyFindKnown(`${directLine ?? ""}\n${setText}`, knownSets, 0.64);
if (setFromText && (!setFromPiece || setFromText.score >= 0.78)) return field(setFromText.value, Math.round(setFromText.score * 100), setFromText.score >= 0.95 ? "ocr" : "fallback");
if (setFromPiece) return field(setFromPiece, derivedConfidence(artifactName, 92), "derived");
const partialSetFromPiece = deriveSetFromPartialPieceName(artifactName.value);
if (partialSetFromPiece) return field(partialSetFromPiece, 72, "derived");
return setFromText ? field(setFromText.value, Math.round(setFromText.score * 100), "fallback") : field("", 0, "missing");
}
@@ -275,6 +278,22 @@ function firstUsefulLine(text: string, rejectIncludes: string[]) {
.find((line) => line.length > 5 && !rejectIncludes.some((reject) => simplifyForMatch(line).includes(simplifyForMatch(reject)))) ?? "";
}
function deriveSetFromPartialPieceName(text: string) {
const words = cleanupOcrLabel(text)
.split(/\s+/)
.map((word) => simplifyForMatch(word))
.filter((word) => word.length >= 5);
if (words.length < 2) return "";
const candidates = knownPieceNames.filter((piece) => {
const normalizedPiece = simplifyForMatch(piece);
const hits = words.filter((word) => normalizedPiece.includes(word)).length;
return hits >= 2;
});
const sets = [...new Set(candidates.map((piece) => pieceToSet.get(piece)).filter((set): set is string => Boolean(set)))];
return sets.length === 1 ? sets[0] : "";
}
function findMainValue(text: string, mainStat: string, slot: string, level: number | null): ParsedField {
const cleaned = text.replace(/\b20\b/g, " ").replace(/[Oo]/g, "0");
const percentValue = extractPercentValue(cleaned);
@@ -504,7 +523,7 @@ function parseEquippedCharacter(text: string): ParsedField {
const wholeTextMatch = fallbackSearch ? fuzzyFindKnown(fallbackSearch, knownCharacters, 0.88) : null;
if (wholeTextMatch) return field(wholeTextMatch.value, Math.round(wholeTextMatch.score * 100), "fallback");
return afterLabel ? field(afterLabel, 50, "fallback") : field("Not detected", 45, "missing");
return field("Not detected", 45, "missing");
}
function field(value: string, confidence: number, source: ParsedField["source"]): ParsedField {
@@ -520,6 +539,20 @@ function promotePercentVariant(stat: string, text: string) {
return stat;
}
function promoteSlotPercentMainStat(slot: string, mainStat: ParsedField): ParsedField {
if (!["ATK", "HP", "DEF"].includes(mainStat.value)) return mainStat;
const references = getSlotMainStatValueReferences(slot);
const hasFlat = references.some((candidate) => candidate.stat === mainStat.value);
const hasPercent = references.some((candidate) => candidate.stat === `${mainStat.value}%`);
if (hasFlat || !hasPercent) return mainStat;
return {
...mainStat,
value: `${mainStat.value}%`,
confidence: Math.max(mainStat.confidence, 90),
source: "derived",
};
}
function getSlotMainStatValueReferences(slot: string): MainStatValueReference[] {
const valueReferences = mainStatValueReferences[slot];
return Array.isArray(valueReferences) ? valueReferences as MainStatValueReference[] : [];
@@ -607,6 +640,7 @@ function cleanupOcrLabel(line: string) {
function cleanupCharacterNoise(text: string) {
return text
.replace(/^.*?equipped\s*:?\s*/i, "")
.replace(/^(?:by|to)\s+/i, "")
.replace(/[^A-Za-z'\-\s]/g, " ")
.replace(/\s+/g, " ")
.trim();
src/lib/autoScanController.test.ts
+2 -2
View File
@@ -6,10 +6,10 @@ describe("autoScanController", () => {
expect(classifyAutoScanCapture({ signature: "", lastDetailSignature: "", seen: new Set() })).toMatchObject({ kind: "unreadable" });
});
it("separates stuck detail views from duplicates", () => {
it("treats repeated readable signatures as duplicates", () => {
const seen = new Set(["same"]);
expect(classifyAutoScanCapture({ signature: "same", lastDetailSignature: "same", seen })).toMatchObject({ kind: "stuck" });
expect(classifyAutoScanCapture({ signature: "same", lastDetailSignature: "same", seen })).toMatchObject({ kind: "duplicate" });
expect(classifyAutoScanCapture({ signature: "same", lastDetailSignature: "other", seen })).toMatchObject({ kind: "duplicate" });
});
src/lib/autoScanController.ts
-1
View File
@@ -14,7 +14,6 @@ export function classifyAutoScanCapture({
seen: ReadonlySet<string>;
}): AutoScanCaptureDecision {
if (!signature) return { kind: "unreadable", countAsMiss: true };
if (signature === lastDetailSignature && seen.has(signature)) return { kind: "stuck", countAsMiss: true, signature };
if (seen.has(signature)) return { kind: "duplicate", countAsDuplicate: true, signature };
return { kind: "new", countAsParsed: true, signature };
}
src/lib/autoScanLoop.test.ts
+1 -1
View File
@@ -265,9 +265,9 @@ describe("autoScanLoop fingerprints", () => {
ocrProfile: "fast",
ocrEngine: "ik-traineddata",
omitFullFrame: true,
omitDetailPreview: true,
omitInventoryPreview: true,
omitCropImages: true,
omitEquippedOcr: true,
skipOcrUnlessArtifactDetail: true,
});
});
src/lib/autoScanLoop.ts
+123 -59
View File
@@ -32,6 +32,14 @@ export type AutoScanLoopDependencies = {
source: string,
needsReview: boolean,
) => Promise<boolean>;
persistParsedArtifactsBatch?: (
items: Array<{
capture: CaptureResult | null;
parsed: ParsedArtifactCandidate;
source: string;
needsReview: boolean;
}>,
) => Promise<number>;
saveReviewSample: (
capture: CaptureResult | null,
parsed: ParsedArtifactCandidate | null,
@@ -65,15 +73,17 @@ export type AutoScanLoopResult = {
};
// Card-ready gating replaces a fixed settle delay: poll the detail fingerprint
// until it has changed and stabilized (or the budget is spent). See cardReadyGate.
const CARD_READY_MAX_MS = 420;
const CARD_READY_POLL_MS = 60;
const CARD_READY_STABLE_SAMPLES = 2;
const CARD_READY_ACCEPT_CHANGED_AFTER_MS = 200;
// until it has changed, then read the artifact immediately. See cardReadyGate.
const CARD_READY_MAX_MS = 180;
const CARD_READY_POLL_MS = 25;
const CARD_READY_STABLE_SAMPLES = 1;
const CARD_READY_ACCEPT_CHANGED_AFTER_MS = 0;
const SCROLL_READY_MAX_MS = 760;
const SCROLL_READY_POLL_MS = 80;
const SCROLL_READY_STABLE_SAMPLES = 2;
const SCROLL_READY_ACCEPT_CHANGED_AFTER_MS = 100;
const STATS_PUBLISH_INTERVAL_MS = 250;
const ROUTINE_CLICK_LOG_INTERVAL = 12;
const MISS_ABORT_THRESHOLD = 3;
const UNREADABLE_ABORT_THRESHOLD = 5;
@@ -87,6 +97,7 @@ export async function runAutoScanLoop(
captureFastSelectedSource,
parseArtifact,
persistParsedArtifact,
persistParsedArtifactsBatch,
saveReviewSample,
getAutoReviewReason,
shouldFlagArtifactForReview,
@@ -102,34 +113,56 @@ export async function runAutoScanLoop(
const maxTargets = resolveScanTargetCount(options.scanLimit, options.detectedInventoryCount);
const rowsToSkip = clampSkipRows(options.skipRows);
const seen = new Set<string>();
const seenDetailFingerprints = new Set<string>();
const seenPageFingerprints = new Set<string>();
let page = 0;
let blockedReason = "";
let aborted = false;
let consecutiveMisses = 0;
let rowsQueued = 0;
let writeQueue: Promise<void> = Promise.resolve();
function updateStats(preserveActiveScanMs = false) {
updateScanTiming(stats, startedAt, Date.now(), { preserveActiveScanMs });
setAutoScanStats({ ...stats });
let flushingWrites = false;
const writeQueue: Array<{ label: string; task: () => Promise<void> }> = [];
const batchedPersistQueue: Array<{
capture: CaptureResult | null;
parsed: ParsedArtifactCandidate;
source: string;
needsReview: boolean;
}> = [];
let lastStatsPublishAt = 0;
function updateStats(preserveActiveScanMs = false, forcePublish = false) {
updateScanTiming(stats, startedAt, Date.now(), { preserveActiveScanMs: preserveActiveScanMs || flushingWrites });
const now = Date.now();
if (forcePublish || now - lastStatsPublishAt >= STATS_PUBLISH_INTERVAL_MS) {
lastStatsPublishAt = now;
setAutoScanStats({ ...stats });
}
}
function enqueueWrite(label: string, task: () => Promise<void>) {
writeQueue = writeQueue
.catch(() => undefined)
.then(async () => {
try {
await task();
} catch (error) {
appendAutomationLog(`write failed ${label}: ${error instanceof Error ? error.message : String(error)}`);
}
});
writeQueue.push({ label, task });
}
async function flushWrites() {
await writeQueue.catch(() => undefined);
updateStats(true);
flushingWrites = true;
try {
if (persistParsedArtifactsBatch && batchedPersistQueue.length > 0) {
const items = batchedPersistQueue.splice(0);
try {
stats.stored += await persistParsedArtifactsBatch(items);
} catch (error) {
appendAutomationLog(`write failed persist-batch:${items.length}: ${error instanceof Error ? error.message : String(error)}`);
}
}
for (const item of writeQueue.splice(0)) {
try {
await item.task();
} catch (error) {
appendAutomationLog(`write failed ${item.label}: ${error instanceof Error ? error.message : String(error)}`);
}
}
} finally {
flushingWrites = false;
}
updateStats(true, true);
}
async function finish(result: AutoScanLoopResult) {
@@ -137,7 +170,7 @@ export async function runAutoScanLoop(
stats.activeScanMs = Math.max(0, flushStartedAt - startedAt);
await flushWrites();
stats.writeFlushMs += Math.max(0, Date.now() - flushStartedAt);
updateStats(true);
updateStats(true, true);
return { ...result, stats: { ...stats } };
}
@@ -152,6 +185,10 @@ export async function runAutoScanLoop(
}
function persistArtifactLater(capture: CaptureResult | null, parsed: ParsedArtifactCandidate, source: string, needsReview: boolean) {
if (persistParsedArtifactsBatch) {
batchedPersistQueue.push({ capture, parsed, source, needsReview });
return;
}
enqueueWrite(`persist:${source}:${parsed.name}`, async () => {
if (await persistParsedArtifact(capture, parsed, source, needsReview)) {
stats.stored++;
@@ -192,9 +229,15 @@ export async function runAutoScanLoop(
}
async function clickTarget(target: GridTarget, label: string) {
appendAutomationLog(`${label} r${target.row} c${target.col} -> ${target.x},${target.y}`);
const clickNumber = stats.clicked + 1;
const routineLog = clickNumber <= 2 || clickNumber % ROUTINE_CLICK_LOG_INTERVAL === 0 || label !== "click";
if (routineLog) appendAutomationLog(`${label} r${target.row} c${target.col} -> ${target.x},${target.y}`);
const clickStartedAt = Date.now();
const clickResult = await api.clickScreen(target.x, target.y);
appendClickDiagnostics(clickResult, `${label} r${target.row} c${target.col}`);
stats.clickMs += Math.max(0, Date.now() - clickStartedAt);
if (routineLog || reportedClickDeliveryFailure(clickResult) || clickResult.inputBlocked) {
appendClickDiagnostics(clickResult, `${label} r${target.row} c${target.col}`);
}
stats.clicked++;
stats.attempted = stats.clicked;
updateStats();
@@ -236,7 +279,6 @@ export async function runAutoScanLoop(
let lastDetailSignature = "";
let lastDetailViewFingerprint = detailFingerprint(currentCapture);
if (lastDetailViewFingerprint) seenDetailFingerprints.add(lastDetailViewFingerprint);
const shouldSkipInitialGridTarget = Boolean(options.processInitialSelection && options.skipInitialGridTarget);
let initialProcessedOffset = 0;
@@ -247,9 +289,9 @@ export async function runAutoScanLoop(
ocrProfile: "fast",
...(options.ocrEngine ? { ocrEngine: options.ocrEngine } : {}),
omitFullFrame: true,
omitDetailPreview: true,
omitInventoryPreview: true,
omitCropImages: true,
omitEquippedOcr: true,
skipOcrUnlessArtifactDetail: true,
});
const initialSurfaceRejection = validateAutoScanEntryPreflight(initialCapture);
@@ -280,7 +322,7 @@ export async function runAutoScanLoop(
saveAutomaticReviewSample(initialCapture, parsed, `automatic:initial-selection-rejected`);
stats.verified++;
stats.misses++;
addCaptureTiming(stats, initialCapture.timings);
addCaptureTiming(stats, initialCapture.timings, initialCapture.elapsedMs);
initialProcessedOffset = shouldSkipInitialGridTarget ? 1 : 0;
lastDetailViewFingerprint = detailFingerprint(initialCapture);
updateStats();
@@ -288,7 +330,7 @@ export async function runAutoScanLoop(
} else {
stats.verified++;
stats.parsed++;
addCaptureTiming(stats, initialCapture.timings);
addCaptureTiming(stats, initialCapture.timings, initialCapture.elapsedMs);
initialProcessedOffset = shouldSkipInitialGridTarget ? 1 : 0;
const signature = sessionSignature(parsed);
seen.add(signature);
@@ -376,6 +418,30 @@ export async function runAutoScanLoop(
return { ready, capture: latestCapture };
}
async function captureArtifactAfterClick(): Promise<{ capture: CaptureResult | null; fingerprint: string; abortReason: string }> {
const capture = await captureSelectedSource(0, false, {
ocrMode: "artifact",
ocrProfile: "fast",
...(options.ocrEngine ? { ocrEngine: options.ocrEngine } : {}),
omitFullFrame: true,
omitDetailPreview: true,
omitInventoryPreview: true,
omitCropImages: true,
skipOcrUnlessArtifactDetail: true,
});
return { capture, fingerprint: detailFingerprint(capture), abortReason: "" };
}
function validateHotArtifactCapture(capture: CaptureResult | null) {
const sourceRejection = captureSourceRejectionReason(capture);
if (sourceRejection) return { ok: false, reason: sourceRejection };
if (!capture?.artifactDetail?.present) {
const confidence = capture?.artifactDetail ? ` (${capture.artifactDetail.confidence}% Detail-Marker)` : "";
return { ok: false, reason: `Keine Artifact-Detailansicht erkannt${confidence}. Artifact-Inventar mit sichtbarer Detailkarte offen lassen.` };
}
return { ok: true, reason: "" };
}
try {
while (!blockedReason && !shouldStop() && stats.parsed < maxTargets) {
page++;
@@ -437,13 +503,15 @@ export async function runAutoScanLoop(
appendAutomationLog(`warn r${target.row} c${target.col}: helper reported cursor/click miss; verifying detail change`);
}
let ready = await awaitCardReady();
if (ready.abortReason) {
blockedReason = ready.abortReason;
let read = await captureArtifactAfterClick();
if (read.abortReason) {
blockedReason = read.abortReason;
aborted = true;
break;
}
let changedDetail = ready.changed;
let capture = read.capture;
let currentDetailFingerprint = read.fingerprint;
let changedDetail = Boolean(currentDetailFingerprint) && currentDetailFingerprint !== lastDetailViewFingerprint;
if (!changedDetail) {
if (options.processInitialSelection && !initialSelectionDuplicateSkipped && !reportedClickDeliveryFailure(clickResult)) {
@@ -469,13 +537,15 @@ export async function runAutoScanLoop(
if (reportedClickDeliveryFailure(clickResult)) {
appendAutomationLog(`warn r${target.row} c${target.col}: retry helper reported cursor/click miss; verifying detail change`);
}
ready = await awaitCardReady();
if (ready.abortReason) {
blockedReason = ready.abortReason;
read = await captureArtifactAfterClick();
if (read.abortReason) {
blockedReason = read.abortReason;
aborted = true;
break;
}
changedDetail = ready.changed;
capture = read.capture;
currentDetailFingerprint = read.fingerprint;
changedDetail = Boolean(currentDetailFingerprint) && currentDetailFingerprint !== lastDetailViewFingerprint;
}
if (!changedDetail) {
@@ -504,36 +574,28 @@ export async function runAutoScanLoop(
stats.verified++;
if (ready.fingerprint) {
if (seenDetailFingerprints.has(ready.fingerprint)) {
consecutiveMisses = 0;
stats.duplicates++;
lastDetailViewFingerprint = ready.fingerprint;
updateStats();
appendAutomationLog(`duplicate visual r${target.row} c${target.col}: OCR uebersprungen`);
continue;
let captureSurfaceRejection = validateHotArtifactCapture(capture);
for (let retry = 1; !captureSurfaceRejection.ok && retry <= 2; retry++) {
appendAutomationLog(`retry capture r${target.row} c${target.col}: ${captureSurfaceRejection.reason}`);
await wait(120);
read = await captureArtifactAfterClick();
if (read.abortReason) {
blockedReason = read.abortReason;
aborted = true;
break;
}
seenDetailFingerprints.add(ready.fingerprint);
capture = read.capture;
currentDetailFingerprint = read.fingerprint;
captureSurfaceRejection = validateHotArtifactCapture(capture);
}
const capture = await captureSelectedSource(0, false, {
ocrMode: "artifact",
ocrProfile: "fast",
...(options.ocrEngine ? { ocrEngine: options.ocrEngine } : {}),
omitFullFrame: true,
omitInventoryPreview: true,
omitCropImages: true,
omitEquippedOcr: true,
skipOcrUnlessArtifactDetail: true,
});
const captureSurfaceRejection = validateAutoScanEntryPreflight(capture);
if (aborted) break;
if (!captureSurfaceRejection.ok) {
blockedReason = captureSurfaceRejection.reason;
appendAutomationLog(`blocked r${target.row} c${target.col}: ${blockedReason}`);
break;
}
if (capture?.ocrTimedOut) {
addCaptureTiming(stats, capture.timings);
addCaptureTiming(stats, capture.timings, capture.elapsedMs);
stats.misses++;
consecutiveMisses++;
lastDetailViewFingerprint = detailFingerprint(capture);
@@ -546,9 +608,11 @@ export async function runAutoScanLoop(
continue;
}
const parseStartedAt = Date.now();
const parsed = parseArtifact(capture);
stats.parseMs += Math.max(0, Date.now() - parseStartedAt);
const rejection = captureRejectionReason(capture, parsed);
addCaptureTiming(stats, capture?.timings);
addCaptureTiming(stats, capture?.timings, capture?.elapsedMs);
if (rejection) {
saveAutomaticReviewSample(capture, parsed, `automatic:capture-rejected:p${page}:r${target.row}c${target.col}`);
src/lib/lockDetection.test.ts
+21
View File
@@ -18,6 +18,18 @@ function bitmap(goldPixels: number, total: number): Bitmap {
return { data, width: total, height: 1 };
}
function solidPixels(pixels: Array<{ b: number; g: number; r: number }>): Bitmap {
const data = Buffer.alloc(pixels.length * 4);
pixels.forEach((pixel, index) => {
const offset = index * 4;
data[offset] = pixel.b;
data[offset + 1] = pixel.g;
data[offset + 2] = pixel.r;
data[offset + 3] = 255;
});
return { data, width: pixels.length, height: 1 };
}
describe("lockDetection", () => {
it("places the lock crop on the lock button in the substat panel", () => {
const size = { width: 2560, height: 1440 };
@@ -41,4 +53,13 @@ describe("lockDetection", () => {
expect(detectLockState(bitmap(20, 100))).toBe(true);
expect(detectLockState(bitmap(1, 100))).toBe(false);
});
it("counts the current red lock glyph but ignores grey unlocked button pixels", () => {
expect(lockSignalRatio(solidPixels([{ b: 90, g: 92, r: 235 }]))).toBe(1);
expect(lockSignalRatio({ data: Buffer.from([235, 92, 90, 255]), width: 1, height: 1 })).toBe(1);
expect(lockSignalRatio({ data: Buffer.from([255, 235, 92, 90]), width: 1, height: 1 })).toBe(1);
expect(lockSignalRatio(solidPixels([{ b: 235, g: 235, r: 235 }]))).toBe(0);
expect(lockSignalRatio({ data: Buffer.from([255, 235, 235, 235]), width: 1, height: 1 })).toBe(0);
expect(lockSignalRatio(solidPixels([{ b: 120, g: 122, r: 128 }]))).toBe(0);
});
});
src/lib/lockDetection.ts
+47 -9
View File
@@ -2,15 +2,15 @@ import { clampRect, type LayoutRect } from "./layoutProfile.js";
import type { Bitmap } from "./ocrPreprocess.js";
// EXPERIMENTAL, read-only lock-status detection (nice-to-have). Genshin shows a
// padlock at the top-right of the artifact detail card: a bright gold fill when
// locked, a dim outline when not. This estimates that icon region and measures
// the fraction of bright "lock-gold" pixels; above a threshold the piece is
// considered locked.
// padlock at the top-right of the artifact detail card: a highlighted red/pink
// lock in the current UI when locked, and a dim grey/white button when not.
// Older UI captures may still use gold highlights. This estimates that icon
// region and measures the fraction of active lock-colour pixels; above a
// threshold the piece is considered locked.
//
// The crop position and threshold need calibration against a reference 16:9
// screenshot before this is wired into the capture pipeline, so it ships pure and
// unit-tested but unused by main.ts. It never drives any in-game action - it only
// reads state for triage.
// The crop position and threshold were validated with unlocked=false and
// locked=true live samples on 2026-07-09. It never drives any in-game action -
// it only reads state for triage and export.
export function lockIconCropRect(detailRect: LayoutRect, imageSize: { width: number; height: number }): LayoutRect {
return clampRect(
@@ -29,14 +29,52 @@ function isLockGold(b: number, g: number, r: number): boolean {
return r >= 180 && g >= 140 && b <= 120 && r > b + 40 && g > b + 20;
}
// Current Genshin detail lock indicator: pink/red lock glyph and dark button
// when the selected artifact is locked. Unlocked buttons are mostly grey/white.
function isLockRed(b: number, g: number, r: number): boolean {
return r >= 180 && g <= 145 && b <= 145 && r > g + 35 && r > b + 35;
}
function isActiveLockPixel(c0: number, c1: number, c2: number): boolean {
const colorMatches = (left: number, middle: number, right: number) =>
isLockGold(left, middle, right) ||
isLockRed(left, middle, right) ||
isLockGold(right, middle, left) ||
isLockRed(right, middle, left);
return colorMatches(c0, c1, c2);
}
function inferAlphaChannel(data: Buffer | Uint8Array, pixels: number): number | null {
const highCounts = [0, 0, 0, 0];
for (let pixel = 0; pixel < pixels; pixel++) {
const index = pixel * 4;
for (let channel = 0; channel < 4; channel++) {
if (data[index + channel] >= 245) highCounts[channel]++;
}
}
const ranked = highCounts
.map((count, channel) => ({ count, channel }))
.sort((left, right) => right.count - left.count);
const best = ranked[0];
const second = ranked[1];
if (best.count / pixels < 0.9) return null;
if (second && second.count / pixels > 0.8) return null;
return best.channel;
}
export function lockSignalRatio(bitmap: Bitmap): number {
const { data, width, height } = bitmap;
const pixels = width * height;
if (pixels === 0) return 0;
const alphaChannel = inferAlphaChannel(data, pixels);
let gold = 0;
for (let pixel = 0; pixel < pixels; pixel++) {
const index = pixel * 4;
if (isLockGold(data[index], data[index + 1], data[index + 2])) gold++;
const colorChannels = [0, 1, 2, 3]
.filter((channel) => channel !== alphaChannel)
.map((channel) => data[index + channel])
.slice(0, 3);
if (colorChannels.length === 3 && isActiveLockPixel(colorChannels[0], colorChannels[1], colorChannels[2])) gold++;
}
return gold / pixels;
}
src/lib/scanReviewUtils.test.ts
+47
View File
@@ -0,0 +1,47 @@
import { describe, expect, it } from "vitest";
import type { CaptureResult } from "../types/global";
import type { ParsedArtifactCandidate, ParsedField } from "./artifactOcrParser";
import { getAutoReviewReason } from "./scanReviewUtils";
function field(value: string): ParsedField {
return { value, confidence: 95, source: "ocr" };
}
describe("getAutoReviewReason", () => {
it("does not require a detail preview image for clean automatic scan captures", () => {
const capture: CaptureResult = {
id: "window:test",
name: "Genshin",
width: 1920,
height: 1080,
dataUrl: "",
capturedAt: new Date(0).toISOString(),
crops: [{ id: "artifact-name", label: "Artifact name", rect: { x: 0, y: 0, width: 10, height: 10 } }],
ocr: [{ id: "artifact-name", label: "Artifact name", text: "Gladiator's Nostalgia", confidence: 95 }],
};
const parsed: ParsedArtifactCandidate = {
name: "Gladiator's Nostalgia",
slot: "Flower of Life",
level: 20,
mainStat: "HP",
mainValue: "4780",
substats: ["CRIT Rate", "CRIT DMG", "Energy Recharge", "ATK%"],
setName: "Gladiator's Finale",
equipped: "",
confidence: 95,
notes: [],
fields: {
name: field("Gladiator's Nostalgia"),
slot: field("Flower of Life"),
level: field("20"),
mainStat: field("HP"),
mainValue: field("4780"),
setName: field("Gladiator's Finale"),
equipped: field(""),
substats: field("CRIT Rate, CRIT DMG, Energy Recharge, ATK%"),
},
};
expect(getAutoReviewReason(capture, parsed)).toBe("");
});
});
src/lib/scanReviewUtils.ts
+1 -1
View File
@@ -8,7 +8,7 @@ export interface ReviewReasonInput {
}
export function getAutoReviewReason(capture: ReviewReasonInput["capture"], parsed: ReviewReasonInput["parsed"]) {
if (!capture.detailDataUrl || !capture.crops?.length || !capture.ocr?.length) return "missing-crops-or-ocr";
if (!capture.crops?.length || !capture.ocr?.length) return "missing-crops-or-ocr";
if (!shouldSaveReviewSample(parsed)) return "";
const lowFields = Object.entries(parsed.fields)
.filter(([, field]) => field.confidence < 70)
src/lib/scannerLearning.test.ts
+26 -1
View File
@@ -23,6 +23,25 @@ describe("scannerLearning", () => {
expect(learned?.ocr?.[0]?.text).toContain("Energy Recharge+6.5%");
});
it("applies field aliases and constrained fixes before parsing", () => {
const learned = applyScannerLearningRules(capture("Equipped; Bennet\nAubade of Morningstar and Moor"), {
fieldAliases: {
equipped: {
Bennet: "Bennett",
},
setName: {
Moor: "Moon",
},
},
constrainedFixes: {
"Equipped;": "Equipped:",
},
});
expect(learned?.ocr?.[0]?.text).toContain("Equipped: Bennett");
expect(learned?.ocr?.[0]?.text).toContain("Aubade of Morningstar and Moon");
});
it("marks low confidence or noted parses for review", () => {
expect(shouldSaveReviewSample({ confidence: 96, notes: [], fields: { name: { confidence: 95 } } })).toBe(false);
expect(shouldSaveReviewSample({ confidence: 96, notes: ["Artifact name was fuzzy-matched"], fields: { name: { confidence: 95 } } })).toBe(false);
@@ -100,7 +119,13 @@ describe("scannerLearning", () => {
});
it("counts learned rules", () => {
expect(countScannerLearningRules({ textReplacements: { one: "1", two: "2" } })).toBe(2);
expect(countScannerLearningRules({
textReplacements: { one: "1", two: "2" },
fieldAliases: { equipped: { Bennet: "Bennett" } },
cropAdjustments: { "artifact-footer": { dy: -2, approved: false } },
uiProfileAdjustments: { "1080p-footer": { fieldId: "artifact-footer", dy: -2, approved: false } },
constrainedFixes: { "Moor": "Moon" },
})).toBe(6);
});
it("does not flag DB review when only non-critical fields are weak", () => {
src/lib/scannerLearning.ts
+54 -3
View File
@@ -15,21 +15,35 @@ export const DEFAULT_SCANNER_LEARNING_RULES: ScannerLearningRules = {
"Elemental Masterv": "Elemental Mastery",
"Equipped;": "Equipped:",
},
fieldAliases: {},
constrainedFixes: {},
cropAdjustments: {},
uiProfileAdjustments: {},
};
export function mergeScannerLearningRules(...rules: Array<Partial<ScannerLearningRules> | null | undefined>): ScannerLearningRules {
return rules.reduce<ScannerLearningRules>(
(merged, rule) => ({
textReplacements: { ...merged.textReplacements, ...(rule?.textReplacements ?? {}) },
fieldAliases: deepMergeRecord(merged.fieldAliases, rule?.fieldAliases),
constrainedFixes: { ...merged.constrainedFixes, ...(rule?.constrainedFixes ?? {}) },
cropAdjustments: { ...merged.cropAdjustments, ...(rule?.cropAdjustments ?? {}) },
uiProfileAdjustments: { ...merged.uiProfileAdjustments, ...(rule?.uiProfileAdjustments ?? {}) },
}),
{ textReplacements: { ...DEFAULT_SCANNER_LEARNING_RULES.textReplacements } },
{
textReplacements: { ...DEFAULT_SCANNER_LEARNING_RULES.textReplacements },
fieldAliases: {},
constrainedFixes: {},
cropAdjustments: {},
uiProfileAdjustments: {},
},
);
}
export function applyScannerLearningRules(capture: CaptureResult | null, rules?: Partial<ScannerLearningRules> | null) {
if (!capture?.ocr?.length) return capture;
const merged = mergeScannerLearningRules(rules);
const replacements = Object.entries(merged.textReplacements ?? {}).filter(([from]) => from.length > 0);
const replacements = replacementEntriesFromLearningRules(merged);
if (replacements.length === 0) return capture;
return {
@@ -90,7 +104,14 @@ function reviewRelevantConfidence(fields: Record<string, { confidence: number }>
}
export function countScannerLearningRules(rules?: Partial<ScannerLearningRules> | null) {
return Object.keys(rules?.textReplacements ?? {}).length;
const fieldAliasCount = Object.values(rules?.fieldAliases ?? {}).reduce((sum, aliases) => sum + Object.keys(aliases ?? {}).length, 0);
return (
Object.keys(rules?.textReplacements ?? {}).length
+ fieldAliasCount
+ Object.keys(rules?.constrainedFixes ?? {}).length
+ Object.keys(rules?.cropAdjustments ?? {}).length
+ Object.keys(rules?.uiProfileAdjustments ?? {}).length
);
}
export function deriveScannerLearningRules(
@@ -149,6 +170,36 @@ function expectedValuesForOcrEntry(id: string, parsed: ParsedArtifactCandidate)
}
}
function replacementEntriesFromLearningRules(rules: ScannerLearningRules) {
const entries = new Map<string, string>();
for (const [from, to] of Object.entries(rules.textReplacements ?? {})) {
if (from.length > 0) entries.set(from, to);
}
for (const aliases of Object.values(rules.fieldAliases ?? {})) {
for (const [from, to] of Object.entries(aliases ?? {})) {
if (from.length > 0) entries.set(from, to);
}
}
for (const [from, to] of Object.entries(rules.constrainedFixes ?? {})) {
if (from.length > 0) entries.set(from, to);
}
return [...entries.entries()];
}
function deepMergeRecord(
left: Record<string, Record<string, string>> | undefined,
right: Record<string, Record<string, string>> | undefined,
) {
const merged: Record<string, Record<string, string>> = {};
for (const [field, aliases] of Object.entries(left ?? {})) {
merged[field] = { ...(aliases ?? {}) };
}
for (const [field, aliases] of Object.entries(right ?? {})) {
merged[field] = { ...(merged[field] ?? {}), ...(aliases ?? {}) };
}
return merged;
}
function deriveReplacementPairs(rawText: string, expected: string) {
if (!expected || expected.startsWith("Unknown")) return [];
const normalizedExpected = normalizeLearningText(expected);
src/lib/scannerSession.test.ts
+6 -2
View File
@@ -21,8 +21,8 @@ describe("scannerSession helpers", () => {
it("updates scan timing and projects the 100-artifact run", () => {
const stats = { ...emptyAutoScanStats, parsed: 4, verified: 2 };
addCaptureTiming(stats, { totalMs: 1200, ocrMs: 900 });
addCaptureTiming(stats, { totalMs: 800, ocrMs: 500 });
addCaptureTiming(stats, { totalMs: 1200, ocrMs: 900 }, 1500);
addCaptureTiming(stats, { totalMs: 800, ocrMs: 500 }, 1000);
updateScanTiming(stats, 1000, 9000);
expect(stats.elapsedMs).toBe(8000);
expect(stats.activeScanMs).toBe(8000);
@@ -33,6 +33,10 @@ describe("scannerSession helpers", () => {
expect(stats.projectedMsFor100).toBe(200000);
expect(stats.activeProjectedMsFor100).toBe(200000);
expect(stats.captureMs).toBe(2000);
expect(stats.captureRoundTripMs).toBe(2500);
expect(stats.averageCaptureRoundTripMs).toBe(1250);
expect(stats.captureRoundTripOverheadMs).toBe(500);
expect(stats.averageCaptureRoundTripOverheadMs).toBe(250);
expect(stats.ocrMs).toBe(1400);
expect(stats.averageCaptureMs).toBe(1000);
expect(stats.averageOcrMs).toBe(700);
src/lib/scannerSession.ts
+31
View File
@@ -18,9 +18,19 @@ export type AutoScanStats = {
projectedMsFor100: number;
activeProjectedMsFor100: number;
captureMs: number;
captureRoundTripMs: number;
captureRoundTripOverheadMs: number;
ocrMs: number;
averageCaptureMs: number;
averageCaptureRoundTripMs: number;
averageCaptureRoundTripOverheadMs: number;
averageOcrMs: number;
clickMs: number;
averageClickMs: number;
parseMs: number;
averageParseMs: number;
loopOverheadMs: number;
averageLoopOverheadMs: number;
captureP50Ms: number;
captureP90Ms: number;
ocrP50Ms: number;
@@ -60,9 +70,19 @@ export const emptyAutoScanStats: AutoScanStats = {
projectedMsFor100: 0,
activeProjectedMsFor100: 0,
captureMs: 0,
captureRoundTripMs: 0,
captureRoundTripOverheadMs: 0,
ocrMs: 0,
averageCaptureMs: 0,
averageCaptureRoundTripMs: 0,
averageCaptureRoundTripOverheadMs: 0,
averageOcrMs: 0,
clickMs: 0,
averageClickMs: 0,
parseMs: 0,
averageParseMs: 0,
loopOverheadMs: 0,
averageLoopOverheadMs: 0,
captureP50Ms: 0,
captureP90Ms: 0,
ocrP50Ms: 0,
@@ -101,7 +121,14 @@ export function updateScanTiming(
stats.projectedMsFor100 = stats.averageMsPerParsed > 0 ? stats.averageMsPerParsed * 100 : 0;
stats.activeProjectedMsFor100 = stats.activeAverageMsPerParsed > 0 ? stats.activeAverageMsPerParsed * 100 : 0;
stats.averageCaptureMs = stats.verified > 0 ? Math.round(stats.captureMs / stats.verified) : 0;
stats.averageCaptureRoundTripMs = stats.verified > 0 ? Math.round(stats.captureRoundTripMs / stats.verified) : 0;
stats.captureRoundTripOverheadMs = Math.max(0, stats.captureRoundTripMs - stats.captureMs);
stats.averageCaptureRoundTripOverheadMs = stats.verified > 0 ? Math.round(stats.captureRoundTripOverheadMs / stats.verified) : 0;
stats.averageOcrMs = stats.verified > 0 ? Math.round(stats.ocrMs / stats.verified) : 0;
stats.averageClickMs = stats.clicked > 0 ? Math.round(stats.clickMs / stats.clicked) : 0;
stats.averageParseMs = stats.parsed > 0 ? Math.round(stats.parseMs / stats.parsed) : 0;
stats.loopOverheadMs = Math.max(0, stats.activeScanMs - stats.captureMs - stats.clickMs - stats.parseMs - stats.cardReadyMs - stats.scrollReadyMs);
stats.averageLoopOverheadMs = stats.parsed > 0 ? Math.round(stats.loopOverheadMs / stats.parsed) : 0;
const samples = timingSamples.get(stats);
stats.captureP50Ms = percentile(samples?.captureMs, 50);
stats.captureP90Ms = percentile(samples?.captureMs, 90);
@@ -115,11 +142,15 @@ export function updateScanTiming(
export function addCaptureTiming(
stats: AutoScanStats,
timing?: { totalMs?: number; ocrMs?: number } | null,
elapsedMs?: number | null,
) {
if (!timing) return stats;
const captureMs = Math.max(0, Math.round(timing.totalMs ?? 0));
const ocrMs = Math.max(0, Math.round(timing.ocrMs ?? 0));
stats.captureMs += captureMs;
if (typeof elapsedMs === "number" && Number.isFinite(elapsedMs)) {
stats.captureRoundTripMs += Math.max(0, Math.round(elapsedMs));
}
stats.ocrMs += ocrMs;
const samples = timingSamples.get(stats) ?? { captureMs: [], ocrMs: [] };
samples.captureMs.push(captureMs);
src/styles/base.css
+2471
View File
File diff suppressed because it is too large Load Diff
src/styles/diagnostics.css
+164
View File
@@ -0,0 +1,164 @@
/* Diagnose / Dev view - all developer info, separated from the Scan workspace. */
.diagnose-view {
display: grid;
gap: 14px;
min-height: 0;
height: 100%;
overflow-y: auto;
overscroll-behavior: contain;
padding-right: 6px;
width: 100%;
}
.diagnose-header {
display: flex;
align-items: flex-start;
justify-content: space-between;
gap: 12px;
flex-wrap: wrap;
}
.diagnose-header-actions {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.diagnose-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 14px;
}
.diagnose-card {
display: grid;
gap: 10px;
align-content: start;
border: 1px solid var(--line);
border-radius: 8px;
background:
linear-gradient(180deg, rgba(255, 255, 255, 0.05), rgba(255, 255, 255, 0.015)),
rgba(18, 12, 35, 0.7);
box-shadow: var(--glass-shadow);
backdrop-filter: blur(18px);
padding: 16px;
}
.diagnose-card-heading {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
}
.diagnose-card-heading h3 {
margin: 2px 0 0;
font-size: 17px;
}
.app-diagnosis-card {
gap: 12px;
}
.diagnosis-source {
display: inline-flex;
align-items: center;
gap: 6px;
border: 1px solid rgba(255, 255, 255, 0.11);
border-radius: 999px;
padding: 7px 10px;
color: var(--text-soft);
font-size: 12px;
white-space: nowrap;
}
.app-diagnosis-grid {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 10px;
}
.app-diagnosis-section {
display: grid;
gap: 8px;
min-width: 0;
border-left: 2px solid rgba(255, 255, 255, 0.2);
border-radius: 8px;
background: rgba(255, 255, 255, 0.035);
padding: 11px 12px;
}
.app-diagnosis-section.ok {
border-left-color: var(--mint);
}
.app-diagnosis-section.warn {
border-left-color: var(--amber);
}
.app-diagnosis-section.risk {
border-left-color: #ff6b8a;
}
.app-diagnosis-section.next {
border-left-color: var(--cyan);
}
.app-diagnosis-title {
display: flex;
align-items: center;
gap: 7px;
min-width: 0;
color: var(--text);
}
.app-diagnosis-title strong {
overflow: hidden;
font-size: 13px;
text-overflow: ellipsis;
white-space: nowrap;
}
.app-diagnosis-section ul {
display: grid;
gap: 7px;
margin: 0;
padding-left: 16px;
color: var(--text-soft);
font-size: 12px;
line-height: 1.35;
}
.app-diagnosis-section li::marker {
color: rgba(255, 255, 255, 0.45);
}
@media (max-width: 1200px) {
.diagnose-grid {
grid-template-columns: 1fr;
}
.app-diagnosis-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
}
@media (max-width: 720px) {
.scan-evidence-capture {
grid-template-columns: 1fr;
}
.scan-evidence-images {
grid-template-columns: 1fr;
}
.app-diagnosis-grid {
grid-template-columns: 1fr;
}
.diagnosis-source {
width: 100%;
justify-content: center;
}
}
src/styles/global.css
+2 -2634
View File
File diff suppressed because it is too large Load Diff
src/types/global.d.ts
Vendored
+30
View File
@@ -89,7 +89,18 @@ export interface CaptureResult {
text: string;
};
locked?: boolean;
lockSignal?: {
ratio: number;
threshold: number;
rect: {
x: number;
y: number;
width: number;
height: number;
};
};
sanctified?: boolean;
elapsedMs?: number;
layout?: {
aspect: string;
isSixteenNine: boolean;
@@ -192,6 +203,25 @@ export type ScannerCommand =
export interface ScannerLearningRulePayload {
textReplacements?: Record<string, string>;
fieldAliases?: Record<string, Record<string, string>>;
constrainedFixes?: Record<string, string>;
cropAdjustments?: Record<string, {
dx?: number;
dy?: number;
dw?: number;
dh?: number;
reason?: string;
approved?: boolean;
}>;
uiProfileAdjustments?: Record<string, {
fieldId: string;
dx?: number;
dy?: number;
dw?: number;
dh?: number;
reason?: string;
approved?: boolean;
}>;
}
export interface LoadScannerLearningRulesResult {