bao/genshin-assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve IK-style artifact scanner pipeline

Browse Source
This commit is contained in:
AzuTear
2026-07-07 22:02:24 +02:00
parent 8ebbe91c39
commit f791d1464c
70 changed files with 7408 additions and 445 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3
View File
@@ -5,6 +5,9 @@ node_modules/
dist/
dist-electron/
outputs/dist/
outputs/admin-start/
outputs/live-capture/
outputs/live-soak/
# Logs
*.log
docs/ARCHITECTURE.md
+19 -1
View File
@@ -55,6 +55,9 @@ flowchart LR
| `src/App.tsx` | Main app shell, scan view, triage view, build view, overlay preview |
| `src/lib/artifactOcrParser.ts` | Converts OCR output into a parsed artifact candidate with confidence and notes |
| `src/lib/fuzzyMatch.ts` | Generic fuzzy string matching for OCR text against known game data |
| `src/lib/genshinLookup.ts` | Pure lookup and validation API for generated Genshin data |
| `src/lib/autoScanEntry.ts` | Pure entry-mode planning and auto-scan preflight validation |
| `src/lib/cardReadyGate.ts` | Detail/page fingerprint readiness gate for scan timing |
| `src/lib/scoring.ts` | Recommendation and build scoring logic |
| `src/lib/demoData.ts` | Temporary local demo snapshot |
| `src/data/genshinGameData.json` | Generated local dictionary of characters, artifact sets, slots, and stats |
@@ -131,6 +134,17 @@ sequenceDiagram
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
- Click verification: after each click the parsed detail-panel signature should change. An unchanged signature is a soft miss (it can also mean two OCR-identical neighbor pieces, common among +0 artifacts), so it is retried once with a small offset, logged with the stuck artifact name, and then skipped - never fatal on its own. The scan aborts only when the first ~6 clicks of page 1 produce nothing new (diagnosis hint: elevated Genshin blocks SendInput via UIPI, or grid coordinates are wrong) or a later page yields zero new artifacts.
- Scan stats separate clicked (click attempts), parsed (readable captures), stored (persisted), review (review samples), duplicates, and misses, so "scanned" cannot be mistaken for "successfully read".
- Artifact grid automation uses Inventory Kamera's 32-target full-page model
(`8 x 4` safe click targets). The apparent lower fifth row sits in the
bottom control band on 16:9 captures and is not clicked automatically.
- Guided auto-entry is state gated. The normal scan button first performs a
lightweight no-OCR preflight; OCR/store/review work starts only after the
artifact inventory grid and artifact detail card are visually confirmed.
- The entry path tries direct inventory (`B`) first and uses the IK-style
ESC/B fallback only when direct entry does not reach an artifact detail card.
- Card and page waits are fingerprint based. The scanner can proceed as soon as
the expected visual state changes and stabilizes, while still accepting IK-like
200 ms item and 100 ms scroll readiness points.
- Scrolling sends one wheel notch per grid row with the cursor anchored over the inventory (assumption: roughly one row per notch; overlap is absorbed by dedupe, and a page without new artifacts stops the scan).
- The scan never deletes, enhances, feeds, locks, or spends anything; it only selects tiles to read them.
@@ -156,4 +170,8 @@ Parsed artifacts from both modes are persisted into `artifact-store.json` keyed
## Performance
Current OCR is prototype-grade and may be slower than the target scanner. Two batch-scan bottlenecks were removed: input/capture no longer spawn a PowerShell process (and recompile Win32 interop) per action, and the Tesseract worker is created once and reused across captures. The eventual batch scanner should still move expensive capture/OCR/build work into workers or a native sidecar.
Current OCR is still measured against the IK target rather than assumed good.
The app keeps a Tesseract.js worker pool, can use the Inventory-Kamera
`genshin_fast_09_04_21.traineddata` path for comparison, and reports capture,
OCR, card-ready, scroll-ready, active-scan, and projected-100 timings. A default
engine change requires a same-capture benchmark and a qualified live soak result.
docs/AUTOMATION_LIVE_SCAN.md
+282 -1
View File
@@ -23,6 +23,12 @@ Validated live on 2026-07-07 with Genshin open in the artifact inventory at
This proves that the current elevated app plus helper path can deliver mouse
movement and click input to the focused Genshin client in this environment.
Latest-source timing is not proven while `/health.appBuild.signature` differs
from the `APP_RUNTIME_SIGNATURE` in `electron/main.ts`. On 2026-07-07 the port
was still owned by an older elevated runtime, so goal scans were intentionally
blocked by the stale-build gate. Restart the elevated app through
`npm run dev:admin` and confirm UAC before collecting new 100-artifact evidence.
## Elevation And UAC
Use:
@@ -98,6 +104,66 @@ For live validation, prefer a bounded scan first:
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?limit=2"
```
The visible-inventory path remains the safest first check. The normal guided
entry tries the read-only direct world path first:
`B -> artifact tab -> first artifact tile`. If that does not produce a visible
artifact detail card, it falls back to the Inventory Kamera-compatible sequence:
`ESC -> B -> artifact tab -> first artifact tile`.
The normal Auto-Scan button uses a guided start. It first takes one lightweight
preflight capture without OCR, full-frame payload, review scoring, or storing.
If an artifact detail card is already visible, it starts the visible-inventory
scan. Otherwise it runs the guided entry above. OCR/review/store work starts
only after the artifact-detail preflight passes.
Guided entry uses short state polling for the Inventory screen, artifact grid,
and first detail card instead of waiting the full fixed delay every time; if the
state never appears, the same timeout budget returns the last diagnostic capture.
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=paimon-menu&limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=auto-entry&limit=2"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/start?entry=visible-inventory&limit=2&engine=ik-traineddata"
```
Those paths send only read-only navigation. `ESC` is not a universal "go to
world" command: from the world it opens the Paimon menu, while from the
already-open Paimon menu it returns to the world. This is why the normal
`auto-entry` path first tries `B` directly and uses the IK-style `ESC -> B`
fallback only when direct entry did not reach an artifact detail card.
The scan starts only after a valid lookup package, supported 16:9 layout,
detected artifact grid, Genshin-client capture, and visual artifact-detail
markers are all present. If any preflight check fails, keep using the
visible-inventory path while tuning the entry step.
The visual preflight also classifies the Paimon menu. The Paimon profile/card
grid can look like an inventory grid if only fixed 16:9 coordinates are used, so
the scanner must reject `paimonMenu.present` before any artifact OCR, review
sample creation, store write, or grid scan starts. The guided entry may still
take lightweight skip-OCR captures while navigating, but those captures are only
state evidence.
The same guard also runs inside the scan loop. If the app is on the main game
screen, a Paimon/menu screen, a generic primary-screen capture, or any screen
without an artifact detail card, auto-scan must block instead of clicking tiles
or trying OCR.
After each click the loop polls the detail fingerprint with a short bounded
budget instead of sleeping blindly. The current budget is 420 ms with 60 ms
polls; if the card changes and stabilizes earlier, OCR starts earlier, and if it
does not change the loop retries or stops through the normal miss guards. If the
card changed but remains animated, the loop now proceeds after 200 ms, matching
Inventory Kamera's select-next-item wait more closely without removing the
detail-change guard.
The outer scan start focuses Genshin once; hot-loop fingerprint/OCR captures do
not re-run the focus helper before every tile, which avoids an OS focus ping on
each artifact while still relying on click readback, foreground checks, and the
detail-card guard for safety.
After a scroll, the loop now uses the same cheap fingerprint polling model for
the inventory pane: it proceeds as soon as the next page fingerprint changed and
stabilized instead of always sleeping the old fixed 760 ms settle delay. Changed
but still animated inventory pages may proceed after 100 ms, again matching IK's
fast-scroll wait while still blocking unchanged pages.
Then poll:
```powershell
@@ -105,10 +171,221 @@ Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
ConvertTo-Json -Depth 12
```
Before live timing, verify that the endpoint is the current app instance:
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/health" |
ConvertTo-Json -Depth 6
```
The response must include `appBuild.signature` and
`appBuild.expectedOcrWorkerPoolSize`. If `appBuild` is missing, or
`/scanner/status` still reports the old OCR warmup start time, the local port is
still owned by a stale elevated Electron process. Close the old Administrator
window/app and restart with `npm run dev:admin` before running scanner probes.
Use the status `stats` timing fields for IK comparisons: `elapsedMs`,
`activeScanMs`, `writeFlushMs`, `averageMsPerParsed`,
`activeAverageMsPerParsed`, `averageCaptureMs`, `averageOcrMs`,
`artifactsPerMinute`, and `projectedMsFor100`. `elapsedMs` is end-to-end
including queued writes; `activeScanMs` is the click/capture/OCR loop before
the final store/review flush. A run only counts as speed
evidence when `parsed`, `stored`, `review`, `duplicates`, and `misses` are read
together; raw click count alone is not scanner throughput. If `averageOcrMs`
dominates `averageMsPerParsed`, the next speed lever is an IK-style OCR worker
queue. If `averageCaptureMs` dominates, crop payload/capture work is the
bottleneck.
The `/scanner/start?limit=N` endpoint sends a renderer command payload with a
temporary scan limit. It does not change the normal UI setting. The normal
hotkeys and buttons still use the UI's configured scan limit.
Lookup and benchmark utility endpoints:
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/status"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/lookup/regenerate"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/ocr/warmup?engine=ik-traineddata"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=ik-traineddata"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&engine=compare"
Invoke-RestMethod "http://127.0.0.1:17317/scanner/benchmark-ocr?limit=5&profile=full"
```
The benchmark endpoint measures the current Tesseract.js engine and the
Inventory-Kamera-traineddata Tesseract.js path against the artifact crop set and
returns timing/field counts, min/p50/p90/max timing, OCR p50/p90 timing,
20/45/100-artifact projections, skipped-OCR count, and the active OCR worker
pool size. It also returns per-field OCR timings under
`ocrFieldAverages`, which is the first place to look before changing crop or
parser behavior. Individual captures also report whether the artifact was
detected as `sanctified`; level/substat crops are shifted in that state to match
Inventory Kamera's crop model. By default it uses the auto-scan `fast` OCR profile,
which omits the low-value set-effect crop, the slot crop that can be derived
from the matched artifact piece name, and the main-stat-value crop that can be
derived from slot, main-stat label, and level. The fast profile also uses
Inventory Kamera's tighter substat crop height; full/manual captures keep the
larger recovery crop for debugging difficult samples. Auto-scan also omits per-crop diagnostic Base64 images from hot-loop OCR
captures while keeping the detail screenshot, OCR text, crop rect metadata, and
timings. OCR crops are passed to Tesseract as PNG buffers internally, not as
Base64 DataURLs, to avoid encode/decode overhead in batch scans. When
`skipOcrUnlessArtifactDetail` blocks OCR because no artifact detail card is
visible, OCR crop preprocessing is skipped too. Auto-scan readiness and scroll
checks use native detail/inventory fingerprints and omit preview DataURLs in
poll captures. Fast preflight/poll captures also omit crop list construction, crop images, and lock-state
detection unless a caller explicitly overrides that option; add
`profile=full` to OCR every artifact detail crop for debugging. It uses the same
artifact-detail guard as auto-scan: if the current screen is not a confirmed artifact detail view, OCR is
skipped and the response shows `skippedOcrCaptures` instead of burning time on
invalid crops.
For speed, the fast auto-scan profile also skips the optional Equipped footer
OCR. Name, level, main-stat label, and substats remain in the OCR hot path;
slot, set, and main-stat value are derived when the lookup/parser can validate
them. Use a full/manual capture when equipped ownership or every debug crop matters.
Local store/review writes are serialized through an internal queue but no longer
block the next inventory click. The scan still flushes the queue before it
returns its final summary, so `stored` and `review` counts remain final-state
numbers.
The app warms the default OCR worker pool in the background after startup; check
`/scanner/status` -> `ocrWarmup.current` before timing the first artifact. Use
`/scanner/ocr/warmup?engine=ik-traineddata` before comparing Inventory
Kamera-traineddata timings so the benchmark is not dominated by worker creation.
`engine=ik-traineddata` uses Inventory Kamera's local
`genshin_fast_09_04_21.traineddata` through Tesseract.js when the file is found
in `data/tessdata`, `IK_TESSDATA_DIR`, `work/Inventory_Kamera`, `work/refs`,
or the local `_ik_ref*` folders.
`engine=compare` runs `current` and `ik-traineddata` against the same visible
artifact detail state. The auto-scan default must stay `current` until the IK
traineddata path wins on the same captures. For a controlled live comparison,
start the scanner with `engine=ik-traineddata`; this only changes the OCR
worker language for that run and leaves the default UI/hotkey path on
`current`.
The OCR pool defaults to four workers because the fast artifact crop set has
four useful OCR parameter groups; set `GAA_OCR_WORKERS=1..8` before startup to
benchmark a different worker count. Inventory Kamera's native engine pool is
still the reference design, but the current app path remains Tesseract.js until
native OCR is integrated and measured. Crops are scheduled across the whole
worker pool and each worker caches its last Tesseract parameter profile; this is
closer to Inventory Kamera's multi-engine field OCR than the earlier
parameter-group-serial scheduler.
## Diagnostic Evidence
The Diagnose page contains a compact evidence timeline for scanner work. It logs
runtime pings, focus attempts, key presses, entry captures, artifact-tab clicks,
preflight failures, grid/count metadata, detail fingerprints, and detail/inventory
screenshots. The same last events are also published through:
```powershell
Invoke-RestMethod "http://127.0.0.1:17317/scanner/status" |
ConvertTo-Json -Depth 18
```
Use this before changing scanner behavior: run the smallest failing action, read
the evidence timeline, then decide whether the failure is focus/input, entry
navigation, grid detection, capture quality, OCR, or parser validation.
If Paimon entry shows `entry key ESC` or `entry key B` with `eventsSent: 0`, the
running `InputHelper.exe` probably predates keyboard support or is blocked. Stop
the elevated app/helper, run `npm run helper:build`, then restart with
`npm run dev:admin` so the app loads the rebuilt helper.
## Soak-Test Helper
After the elevated app is running and Genshin is open on the artifact inventory,
the non-elevated terminal can drive the local dev-control endpoints and save a
full evidence bundle:
```powershell
npm run scan:soak
```
The helper writes timestamped JSON snapshots and a transcript to:
```text
outputs/live-soak/<timestamp>/
```
Default sequence:
1. `/health`
2. `/scanner/status`
3. `/capture/smart?skipOcr=1`
4. `/automation/probe-click?index=1`
5. `/automation/probe-click?index=3`
6. `/scanner/start?limit=2`
7. `/scanner/start?limit=5`
8. `/scanner/start?limit=10`
9. `/scanner/start?limit=20`
10. `/review/samples?limit=30`
For the actual Inventory-Kamera speed target, use the explicit goal run after
`/health` shows the current `appBuild`:
```powershell
npm run scan:goal
npm run scan:goal:current
npm run scan:goal:ik
npm run scan:goal:compare
```
That run first warms/benchmarks `current` vs. `ik-traineddata`, then scans
limits `2, 5, 20, 45, 100` with the selected scan engine, and writes
`scan-run-summary.json` plus `scan-run-summary.csv`. `npm run scan:goal`
uses the default `current` scan engine; use `scan:goal:ik` for a native
IK-traineddata scan pass. Use `scan:goal:compare` to run both scan engines
back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for
`averageMsPerParsed`, `activeAverageMsPerParsed`, `averageCaptureMs`,
`captureP50Ms`, `captureP90Ms`, `averageOcrMs`, `ocrP50Ms`, `ocrP90Ms`,
`averageCardReadyMs`, `averageScrollReadyMs`, `artifactsPerMinute`, and
`projectedMsFor100`.
The run also writes `scan-performance-assessment.json`, which groups results by
limit, picks the best qualified engine, and labels the dominant bottleneck as
OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
run, parse the requested count, keep miss rate under 2%, and keep review rate
at or below 15%; review and miss rates are penalized before active average speed
is used as the tie-breaker.
The assessment ranking can be verified without Genshin or the Electron app:
```powershell
npm run scan:assessment:test
```
This self-test rejects synthetic runs that are fast but have too many misses or
too many review samples, so the final IK comparison cannot be won by speed alone.
For the current implementation summary and IK comparison rationale, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
Use the readiness timings to compare against Inventory Kamera's fixed waits:
IK waits about 200 ms after selecting the next inventory item and about 100 ms
after fast scrolls. If `averageCardReadyMs` or `averageScrollReadyMs` dominates
the active average while OCR is already low, tune the fingerprint gate before
touching OCR again.
The runner reads `APP_RUNTIME_SIGNATURE` from `electron/main.ts` and refuses
to run against a stale Electron process when `/health.appBuild.signature` does
not match the current source. Use `-AllowStaleBuild` only for deliberate
debugging of an older instance.
Current dev builds also expose `/dev/shutdown` on localhost. The start cleanup
script calls it before falling back to `Stop-Process`, so a previous elevated
app can shut itself down cleanly even when the caller cannot terminate an
administrator process directly. Older builds without that endpoint still need
manual close or a confirmed `npm run dev:admin` restart.
Review samples are saved as a compact summary by default so Vite does not try to
watch large Base64 payloads under `outputs/`. Full review payloads can be saved
with `-SaveFullReviewSamples` when needed.
It stops on a failed probe, blocked scan, stopped scan, or timeout unless
`-ContinueAfterBlocked` is supplied directly:
```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\live-soak.ps1 -Limits 2,5 -ContinueAfterBlocked
```
## Anti-Cheat And Safety Boundary
Do not describe the current implementation as bypassing anti-cheat. The app
@@ -133,7 +410,9 @@ The current 16:9 layout profile is calibrated from a 1920x1080 English
artifact-inventory capture:
- detail rect approximately `x=1308`, `y=120`, `width=492`, `height=838`
- inventory grid: `8 x 5`
- inventory grid: `8 x 4` safe automated targets, matching Inventory Kamera's
32-artifact full-page model. The apparent lower fifth row is in the bottom
control band and is intentionally not clicked during auto-scan.
- first tile center: `x=179`, `y=254`, `row=0`, `col=0`
- second tile center: `x=325`, `y=254`, `row=0`, `col=1`
- inventory count crop successfully read `2059/2400` in the live session
@@ -152,3 +431,5 @@ Before marking an automation change done:
5. If Genshin is available, run `/automation/probe-click?index=1`.
6. For scan-loop changes, run `/scanner/start?limit=2` before any broader scan.
7. Record new live findings in this file and in `docs/scanner-rework-status.md`.
8. For IK-target claims, attach or cite `scan-performance-assessment.json` from
a non-stale `npm run scan:goal:compare` run.
docs/CHECKLISTS.md
+13
View File
@@ -4,6 +4,7 @@
- [ ] The expected crop or capture behavior is clear.
- [ ] Genshin is not accessed through memory reads, hooks, injection, or game files.
- [ ] Auto-scan starts only from confirmed artifact inventory plus visible detail card, or blocks with a reason.
- [ ] Capture failures are shown to the user.
- [ ] OCR uncertainty remains inspectable in Details.
- [ ] Parser output does not silently trust low-confidence text.
@@ -12,6 +13,18 @@
- [ ] `npm run build` passes.
- [ ] Manual Smart Capture is tested when possible.
## IK-Speed Or OCR-Engine Claim
- [ ] `/health.appBuild.signature` matches the current `APP_RUNTIME_SIGNATURE`.
- [ ] `npm run scan:assessment:test` passes.
- [ ] The run includes `scan-performance-assessment.json`.
- [ ] The 100-artifact run finishes cleanly.
- [ ] Parsed count is at least the requested count.
- [ ] Miss rate is at or below 2%.
- [ ] Review rate is at or below 15%.
- [ ] Speed comparison uses active scan timing plus quality, not click count alone.
- [ ] The default OCR engine is changed only after same-capture benchmark evidence.
## UI Change
- [ ] The main workflow remains visible without unnecessary scrolling.
docs/DECISIONS.md
+47
View File
@@ -16,6 +16,7 @@ This document contains Architecture Decision Records.
| ADR-008 | Replace the PowerShell input/capture helper with a C# sidecar | Accepted | 2026-07-05 |
| ADR-009 | Resolution-anchored layout profiles and OCR preprocessing over color detection | Accepted | 2026-07-05 |
| ADR-010 | Elevated dev runner and bounded live automation probes | Accepted | 2026-07-07 |
| ADR-011 | Quality-gated Inventory Kamera comparison before OCR default changes | Accepted | 2026-07-07 |
## ADR-001: Build A Local Electron App First
@@ -286,3 +287,49 @@ Document the workflow in [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
- The implementation remains inside the allowed safety boundary: no memory
reads, hooks, injection, game-file modification, deleting, feeding, enhancing,
locking/unlocking, or spending resources.
## ADR-011: Quality-Gated Inventory Kamera Comparison Before OCR Default Changes
### Status
Accepted
### Context
The product target is not merely to click through 100 artifacts quickly. It is
to scan the first 100 artifacts with accuracy at least as good as Inventory
Kamera and speed equal to or better than Inventory Kamera. A faster scan that
creates too many misses, review samples, or false positives is worse than a
slower qualified run.
The app can now compare the current OCR path with Inventory Kamera's
`genshin_fast_09_04_21.traineddata` through the same visible crop set. It also
has hot-loop timing fields for capture, OCR, card readiness, scroll readiness,
active scan time, and projected 100-artifact time.
### Decision
Use a quality-gated live soak and benchmark before changing the default OCR
engine or claiming IK parity. A qualified scan result must:
- finish cleanly,
- parse at least the requested count,
- keep miss rate at or below 2%,
- keep review rate at or below 15%,
- report active scan timing and bottlenecks,
- come from a runtime whose `/health.appBuild.signature` matches the current
source.
`scripts/live-soak.ps1` writes the evidence bundle and
`scan-performance-assessment.json`. `npm run scan:assessment:test` verifies that
the ranking logic rejects fast but low-quality synthetic runs without needing
Genshin.
### Consequences
- Speed claims cannot be based on click count or elapsed time alone.
- A new OCR engine cannot become the default just because it is theoretically
closer to IK; it must win the same-capture benchmark and a qualified live run.
- Stale elevated Electron instances are treated as invalid evidence, not as a
harmless warning.
- The goal remains open until the 100-artifact qualified comparison is captured.
docs/PROJECT.md
+14 -2
View File
@@ -3,6 +3,8 @@
This document is the source of truth for project intent, scope, runtime facts, and operational expectations.
For implementation structure, see [ARCHITECTURE.md](ARCHITECTURE.md). For engineering standards, see [CONVENTIONS.md](CONVENTIONS.md).
For the latest Inventory-Kamera comparison work, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
## Project Identity
@@ -58,6 +60,7 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
| --- | --- | --- |
| Safety | Never perform irreversible in-game actions. | Code review and manual test |
| Performance | Single artifact read should feel interactive and batch scan should not stall on false progress. | Capture latency monitored manually; auto-scan stops on blocked verification |
| IK target | First 100 artifacts should scan with accuracy at least as good as Inventory Kamera and equal or better speed. | `npm run scan:goal:compare` quality-gated report |
| Privacy | Captures and parsed data stay local by default. | No remote upload in scanner path |
| Reliability | Uncertain OCR must be visible to the user. | Confidence and details view |
| Learning loop | Scanner mistakes should become reusable local review samples. | `review-samples.jsonl` |
@@ -95,6 +98,9 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
- The parser already uses known sets, pieces, slots, stat aliases, set aliases, character aliases, and derived slot/set mapping.
- Review samples, learned replacements, parser notes, and stored artifacts already persist locally.
- The auto-scan loop is no longer a naive click spammer: it has preflight, verification, miss handling, page fingerprinting, and stop conditions.
- The scanner now has an Inventory-Kamera comparison path: 32 safe artifact
targets per page, lookup-derived fields, fast OCR crop profile, current vs.
IK-traineddata benchmark endpoint, and a quality-gated live soak runner.
- Elevated live automation is validated in the current dev environment:
`/automation/probe-click?index=1` changed the selected artifact and
`/scanner/start?limit=2` completed with 2/2 verified reads and 0 misses.
@@ -108,6 +114,9 @@ The app is not intended to replace deep min-max tools. It prioritizes time savin
- Learned fixes currently focus on text replacements; they do not yet update crop offsets, UI profile variants, or scanner targeting rules in a structured way.
- The scan page is cleaner than before, but it still exposes too much operator/debug state in the main flow.
- Recommendations and build logic exist, but the scanner is not yet reliable enough to make them the core focus.
- The latest source has not yet completed the final 100-artifact live comparison
because the current dev-control port is still owned by a stale elevated
Electron process. Live timing must wait for a UAC-approved restart.
### Current product conclusion
@@ -233,8 +242,11 @@ Status:
2. Tighten the game data generator and parser contract, then backfill regression tests from real bad samples.
3. Continue moving auto-scan behavior out of `App.tsx` and into isolated scanner modules.
4. Soak-test the elevated C# helper automation path with gradually larger scan limits and page scroll transitions.
5. Extend the learning system from text-only fixes into crop/UI profile tuning.
6. Resume recommendation work only when scan accuracy is consistently trustworthy.
5. Run `npm run scan:goal:compare` after `/health.appBuild.signature` matches
the current source and use the quality-gated 100-artifact report as the IK
target evidence.
6. Extend the learning system from text-only fixes into crop/UI profile tuning.
7. Resume recommendation work only when scan accuracy is consistently trustworthy.
## Open Questions
docs/ocr-eval.md
+3
View File
@@ -2,12 +2,15 @@
Field-level accuracy measurement for the artifact OCR parser. This is the gate
every OCR, crop, layout, or parser change runs against (see ADR-007).
It is necessary but not sufficient for the IK target: live scan speed and
review/miss rates are measured by `npm run scan:goal:compare`.
## Run it
```powershell
npm run eval # full accuracy report for the seed corpus
npm test # runs the eval gate alongside the rest of the suite
npm run scan:assessment:test # verifies quality-first scan ranking logic
```
The report prints exact-match rate, overall field accuracy, a per-field
docs/scanner-ik-progress-report.md
+301
View File
@@ -0,0 +1,301 @@
# Scanner IK Progress Report - 2026-07-07
This report summarizes the scanner/OCR work toward the current target:
scan the first 100 artifacts with accuracy at least as good as Inventory Kamera
and speed equal to or better than Inventory Kamera, without memory reads, hooks,
injection, game-file modification, or unsafe in-game actions.
## Executive Summary
The scanner has moved from a fragile OCR-first prototype toward an
Inventory-Kamera-style artifact scanner:
- Artifact scan is now the first-class path.
- Auto-scan starts only after a validated artifact inventory/detail preflight.
- Main-game, Paimon-menu, primary-screen, unsupported-layout, missing-grid, and
missing-detail states block before OCR/store/review work.
- OCR uses a fast artifact profile that skips low-value fields and derives
slot, set, and main-stat value through lookup constraints when safe.
- The OCR worker pool, field crop split, page model, scroll model, and readiness
waits now mirror the relevant IK design choices more closely.
- Diagnostics now preserve state evidence, timings, screenshots where useful,
entry events, focus/input events, preflight failures, and scan-loop reasons.
- A live soak runner now measures throughput and quality, compares current vs.
IK-traineddata engines, and refuses to run against stale Electron builds.
The requested final goal is not proven complete yet. The current live dev port is
still owned by an older elevated Electron instance, so the latest code cannot be
truthfully benchmarked against IK until the app is restarted with UAC approval
and `npm run scan:goal:compare` completes a qualified 100-artifact run.
## What Changed
### Lookup and validation
- `scripts/generate-genshin-data.cjs` was extended into a stricter lookup
package generator.
- `src/lib/genshinLookup.ts` provides pure matching and validation for sets,
pieces, slots, stats, characters, aliases, GOOD keys, source version, and
validation summaries.
- Auto-scan preflight blocks if the lookup package is invalid.
Why this matters:
IK succeeds partly because raw OCR is not trusted by itself. The app now follows
the same principle: OCR text is normalized, matched, constrained, and derived
against a canonical package before it is accepted.
### OCR and parser pipeline
- Artifact detail crops are split into field-specific regions:
name, slot, main-stat label, main-stat value, level, substats, set effects,
equipped/footer, lock, and rarity.
- Fast auto-scan profile skips lower-value OCR work:
set effects, slot crop, main-stat value crop, equipped footer, crop images,
full-frame payloads, and inventory preview payloads.
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
constraints make that safe.
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
- Exact visual duplicates are skipped before OCR.
Why this matters:
The fast path spends OCR only on fields that materially change the artifact
identity or review decision. That is closer to IK's queued crop model than a
manual-debug capture that OCRs every visible thing.
### Engine comparison and benchmark path
- `/scanner/ocr/warmup?engine=current|ik-traineddata` warms OCR workers.
- `/scanner/benchmark-ocr?engine=current|ik-traineddata|compare` benchmarks the
same visible artifact crops.
- Auto-scan accepts `ocrEngine: "current" | "ik-traineddata"`.
- `scripts/live-soak.ps1` supports:
- `npm run scan:goal`
- `npm run scan:goal:current`
- `npm run scan:goal:ik`
- `npm run scan:goal:compare`
- `scan-performance-assessment.json` ranks runs by quality first and speed
second.
Important rule:
A fast engine cannot win if it has too many misses or too much review. A
qualified winner must finish cleanly, parse the requested count, keep miss rate
at or below 2%, and keep review rate at or below 15%.
### Auto-scan entry and safety
- The normal auto button runs a guided start:
1. focus Genshin,
2. run a lightweight no-OCR preflight,
3. if artifact detail is visible, use visible-inventory mode,
4. otherwise try direct `B -> artifact tab -> first artifact tile`,
5. if needed, fall back to the IK-style ESC/B inventory sequence,
6. start OCR only after artifact grid and detail card pass preflight.
- Entry captures are state evidence only. They do not create review samples,
store artifacts, or run artifact OCR before the detail preflight passes.
- Entry waits now poll for state readiness instead of always sleeping the full
fixed delay.
- Scan loop also rechecks the same safety boundary after each click and scroll.
Why this matters:
The previous failure mode was dangerous from a product-quality point of view:
when the game was not in artifact inventory, the scanner could still take
screenshots and try to read artifacts. The current path is explicitly blocked
outside the artifact inventory/detail state.
### Scan loop and speed
- Grid model uses IK's 32-artifact visible page concept (`8 x 4`) instead of
clicking the risky lower band.
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
duplicate reads after scroll.
- Card readiness uses detail fingerprint polling:
max 420 ms, 60 ms polls, changed cards may proceed after 200 ms.
- Scroll readiness uses inventory fingerprint polling:
max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
- Store/review writes are queued so the next tile can be clicked before disk
writes finish. The queue is still flushed before final summary.
- Focus is done once at scan start; hot-loop captures do not refocus every tile.
Why this matters:
IK uses fixed waits around 200 ms after selecting inventory items and 100 ms
after fast scrolls. The app now keeps those as safety ceilings/acceptance points
while allowing earlier continuation when visual evidence is ready.
### Diagnostics and logging
- Scanner diagnostics now capture timeline events for runtime, focus, keypress,
entry captures, tab clicks, first-tile clicks, preflight, OCR/skips, grid,
counts, detail/page fingerprints, and failure reasons.
- `/scanner/status` publishes recent diagnostic evidence.
- Review sample output is compact by default so Vite does not watch large Base64
payloads during live soak runs.
- The live runner writes timestamped JSON snapshots, CSV summaries, transcript,
benchmark data, and performance assessment files under `outputs/live-soak/`.
Why this matters:
Future scanner bugs can be debugged from captured evidence instead of relying
only on a human description of what appeared on screen.
## Vorgehensweise
1. Read the local Inventory Kamera reference under `work/Inventory_Kamera`.
2. Copy the proven concepts, not the entire implementation:
32 artifact targets per page, fixed coordinate ratios, queued OCR work,
short item/scroll waits, read-only inventory navigation, and Tesseract
traineddata comparison.
3. Harden the app's own architecture around those concepts:
pure lookup API, parser derivation, renderer scan orchestration,
Electron capture/OCR boundary, sidecar input helper, and diagnostics.
4. Add tests before trusting behavior:
lookup validation, parser derivation, auto-entry planning/preflight,
card-ready gates, page planning, scan-loop blocking, OCR eval corpus.
5. Add live tooling before claiming performance:
bounded probes, stale-build gate, benchmark endpoint, soak runner, CSV/JSON
assessment, and quality-first comparison.
## Tests and Evidence
Latest repo validation after the recent changes:
| Check | Result |
| --- | --- |
| PowerShell parse for `scripts/live-soak.ps1` | Passed |
| `npm run scan:assessment:test` | Passed |
| `npm run lint` | Passed |
| Focused scanner tests | Passed |
| `npm test` | Passed, 171 tests |
| OCR eval seed corpus | 100% exact match, 100% field accuracy, 100% critical fields |
| `npm run build` | Passed |
| `git diff --check` | Passed |
Live evidence already collected earlier on 2026-07-07:
- Probe click changed artifact detail successfully.
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
- Limit 20 live soak completed on the first visible page.
- Limit 45 live soak crossed into a scrolled page.
Current live limitation:
- `/health` still reports an older elevated build:
`2026-07-07-ocr-pool4-hotloop-no-refocus`.
- Current source expects:
`2026-07-07-ik32-fastsubstats-active-timing`.
- The live soak runner correctly refuses to benchmark the stale runtime.
- A UAC restart attempt was canceled, so the latest code is not yet live.
## Inventory Kamera Comparison
| Area | Inventory Kamera | Current app status |
| --- | --- | --- |
| Safe scope | Reads inventory through screen/click automation | Same safety boundary: screen capture and read-only input only |
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
| Item wait | About 200 ms fixed wait | Fingerprint polling, accepts changed card after 200 ms |
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
| 100-artifact proof | Reference target | Not yet proven on latest app build |
What is theoretically better than before:
- The app no longer spends OCR on invalid screens.
- It no longer treats click count as scanner success.
- It can prove whether `current` or `ik-traineddata` wins on the same capture
set instead of changing engines blindly.
- It can reject fast-but-wrong results automatically.
- It can identify whether the bottleneck is OCR, capture, card readiness, or
scroll readiness.
What is not yet proven better than IK:
- Native Tesseract speed is not integrated as the default.
- The latest code has not completed the 100-artifact live run.
- Review rate and miss rate on the user's real inventory still need the new
live report.
## Theoretical Runtime Flow
For the intended 100-artifact comparison:
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run `npm run scan:goal:compare`.
6. For each engine and limit (`2, 5, 20, 45, 100`):
- focus Genshin once,
- verify lookup and layout,
- verify artifact grid and detail card,
- click one safe grid target,
- poll detail fingerprint,
- skip duplicate visuals,
- OCR only the fast artifact crop set,
- parse through lookup constraints,
- queue store/review writes,
- scroll with inventory fingerprint polling,
- stop on repeated pages, invalid surfaces, blocked input, OCR timeout, or
repeated misses.
7. Write CSV, JSON snapshots, transcript, benchmark report, and performance
assessment.
8. Declare a winner only if the 100-artifact run is qualified by quality.
Expected bottleneck sequence:
- If OCR dominates, compare `current` vs `ik-traineddata`, crop count, and
worker pool size.
- If capture dominates, reduce payload construction and preview/crop image work.
- If card-ready dominates, tune the detail fingerprint gate.
- If scroll-ready dominates, tune page fingerprint polling and scroll notches.
## What Is Better Than Before
- Auto-scan is artifact-detail gated; no more blind OCR from main gameplay or
menu screens.
- Paimon/menu detection blocks before scan-loop OCR or writes.
- The normal button is one coherent guided flow instead of a separate "get to
inventory first, then scan" workflow.
- The app has a real lookup layer instead of raw OCR plus scattered hardcoded
assumptions.
- The scanner can compare OCR engines without changing the default blindly.
- Performance reports now include quality decisions, not just elapsed time.
- Diagnostics are concrete enough for later self-troubleshooting.
- Stale elevated runtime is detected before live soak, avoiding false evidence.
## Risks and Remaining Work
1. Restart with UAC and run the latest build live.
2. Run `npm run scan:goal:compare` from a visible artifact inventory.
3. If the 100-artifact winner is not qualified, inspect:
`scan-performance-assessment.json`, review samples, diagnostic timeline, and
field timings.
4. If `ik-traineddata` wins but Tesseract.js is still slow, evaluate native
Tesseract integration.
5. Grow the eval corpus with confirmed real review samples before tightening
parser thresholds further.
6. Validate a positive locked-artifact sample.
7. Keep recommendations secondary until scanner quality is proven.
## Definition of Done for the IK Target
The goal is complete only when current evidence proves all of these:
- The app is running the latest runtime signature.
- The 100-artifact scan finishes cleanly.
- Parsed count is at least 100.
- Miss rate is at or below 2%.
- Review rate is at or below 15%.
- The run is equal to or faster than the recorded IK reference or the selected
IK-traineddata/native baseline on the same machine and inventory setup.
- The evidence bundle is saved under `outputs/live-soak/`.
- Any chosen default OCR engine is backed by the same-capture benchmark.
docs/scanner-rework-status.md
+89 -2
View File
@@ -5,6 +5,22 @@ Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in
live automation runbook, see
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
## Current IK-Speed Target Status
See [scanner-ik-progress-report.md](scanner-ik-progress-report.md) for the full
report.
Current status:
- The scanner architecture now follows the relevant Inventory Kamera model:
32 artifact targets per page, lookup-derived fields, fast artifact OCR profile,
short readiness gates, page-overlap planning, and queued OCR/store work.
- The live runner can compare `current` and `ik-traineddata` engines and rejects
runs that are fast but fail miss/review quality thresholds.
- The final 100-artifact IK target is not proven yet. The dev-control port is
currently owned by an older elevated Electron build, and the runner correctly
refuses stale timing evidence until the app is restarted with UAC approval.
## Done (implemented, unit-tested, build green)
- **OCR eval harness** — `src/eval/`, `npm run eval`, gate in `npm test`. See
@@ -37,6 +53,62 @@ live automation runbook, see
- **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live
with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records,
2 review samples, and 0 misses.
- **Auto-scan OCR performance pass** - auto-scan captures now use an artifact
OCR mode that skips inventory-count OCR on each tile, keeps equipped-character
OCR, raises the substat crop to catch artifact level, stores automatic review
samples without full-screen/inventory screenshots, reads only the tail of large
JSONL files, avoids review noise when only level/equipped is missing, starts
the scan with an OCR-free preflight capture, skips exact visual duplicates
before OCR, prevents repeated startup review reprocessing, omits full-frame
and inventory-preview Base64 payloads from tile captures, and applies
crop-specific Tesseract page-segmentation/whitelist parameters.
- **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the
dev-control health/status, smart-capture, probe-click, bounded scan, and
review-tail endpoints and writes evidence to `outputs/live-soak/`. On
2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10,
and 20 against the elevated running app. The limit 20 run finished `done` with
20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1
miss, and 1 page.
- **Scroll/page-transition live soak** - after the helper and loop fixes,
`scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture`
completed `done` on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35
stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the
scanner can cross from the first visible page into a scrolled page in the live
1920x1080 setup.
- **Lookup package layer** - `scripts/generate-genshin-data.cjs` now emits
normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source
version metadata, generated time, and validation summary. `src/lib/genshinLookup.ts`
provides pure matching and validation APIs, and the scanner status/dev-control
path exposes lookup validity. Auto-scan preflight blocks when the lookup package
is invalid.
- **Inventory-Kamera-style field split** - artifact detail crops now separate
name, slot, main-stat label, main-stat value, level, substats, set effects, and
footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives
slot/set/main-stat through lookup constraints before falling back to review.
- **Paimon-menu auto-entry scaffold** - auto-scan supports
`scanEntryMode: "paimon-menu"` and `/scanner/start?entry=paimon-menu&limit=N`.
The entry sends only read-only navigation (`ESC`, `B`, artifact-tab click),
then requires a valid lookup, supported layout, and detected artifact grid
before the scan loop starts. The existing visible-inventory start remains the
fallback/debug path.
- **OCR benchmark endpoint scaffold** - `/scanner/benchmark-ocr?limit=N` captures
identical artifact crops with the current engine and returns timing/field counts.
`/scanner/benchmark-ocr?engine=compare` can also compare the local
Inventory-Kamera-traineddata Tesseract.js path when
`genshin_fast_09_04_21.traineddata` is present in `data/tessdata`, `work/`, or
`IK_TESSDATA_DIR`. The OCR worker pool defaults to four workers and can be
tuned with `GAA_OCR_WORKERS=1..8`. Native Tesseract is still not the default
and should only replace `tesseract.js` after the benchmark proves it faster
and more accurate on the same crops.
- **Quality-gated live comparison** - `scripts/live-soak.ps1` now supports
goal runs for `current`, `ik-traineddata`, and `compare`, writes CSV/JSON
summaries, groups results by limit, identifies timing bottlenecks, and rejects
winners that miss the requested count, exceed 2% misses, or exceed 15% review.
`npm run scan:assessment:test` verifies this ranking logic without Genshin.
- **State-polled guided entry** - the guided auto-entry waits for Inventory,
artifact grid, and first detail card evidence instead of sleeping the full
fixed delay every time. OCR/review/store work still starts only after artifact
detail preflight passes.
## Remaining — needs the live environment or a UI pass
@@ -46,12 +118,27 @@ resolution or without UI work best tested live:
1. **Validate/tune OCR preprocessing** on more real captures — confirm invert +
threshold + upscale factor help (not hurt) actual Tesseract reads. The
text-level eval harness cannot measure image preprocessing.
2. **Validate locked=true** against a known locked artifact — unlocked/grey lock
2. **Wire and benchmark native IK-traineddata OCR** against the same crop set.
The current benchmark can use IK-traineddata through Tesseract.js; native
Tesseract integration remains the next implementation step before any engine
default changes.
3. **Validate guided entry live** from world, visible inventory, and Paimon/menu
states with limits 2, 20, and 45. Confirm the artifact-tab coordinate in the
user's current 16:9 layout and keep `visible-inventory` as fallback if the
menu path is blocked.
4. **Validate locked=true** against a known locked artifact — unlocked/grey lock
was live-checked; a gold locked icon still needs a positive sample.
3. **Broader scan soak test** — after the bounded two-item live scan passed,
5. **Broader scan soak test** — after the bounded two-item live scan passed,
the next automation validation should increase the limit gradually and watch
for repeated pages, scroll behavior, duplicate handling, and OCR review rate.
6. **100-artifact IK comparison** — after `/health.appBuild.signature` matches
current source, run `npm run scan:goal:compare` and compare qualified
100-artifact results.
Visible-page limits up to 20 and a scroll/page-transition limit of 45 have
passed. The remaining soak work is now OCR accuracy, review-rate reduction, and
larger runs after the review corpus has grown.
## Grow the eval corpus
electron/bootstrap/ipcBootstrap.ts
+3
View File
@@ -14,6 +14,7 @@ import type {
ClickResult,
AutomationGuard,
ScrollResult,
KeyPressResult,
SaveResultWithPath,
SaveSnapshotResult,
GoodDatabase,
@@ -60,6 +61,7 @@ interface CaptureHandlersDependencies {
captureSource: (sourceId: string, delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult>;
clickScreen: (x: number, y: number) => Promise<ClickResult>;
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => Promise<ScrollResult>;
keyPress: (key: string) => Promise<KeyPressResult>;
getAutomationGuard: () => Promise<AutomationGuard>;
}
@@ -96,6 +98,7 @@ export function registerIpcHandlers(dependencies: IpcBootstrapDependencies) {
captureSource: dependencies.captureSource,
clickScreen: dependencies.clickScreen,
scrollScreen: dependencies.scrollScreen,
keyPress: dependencies.keyPress,
getAutomationGuard: dependencies.getAutomationGuard,
});
}
electron/devControlServer.ts
+177 -3
View File
@@ -1,6 +1,8 @@
import fs from "node:fs/promises";
import http, { type Server } from "node:http";
import path from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import type {
CaptureOptions,
CaptureResult,
@@ -9,14 +11,20 @@ import type {
ReviewSampleListResult,
ScannerCommand,
ScannerStatusPayload,
AppRuntimeInfo,
} from "../src/types/global.js";
import { validateLookupPackage } from "../src/lib/genshinLookup.js";
const execFileAsync = promisify(execFile);
interface DevControlServerDependencies {
registeredHotkeys: Record<string, boolean>;
appBuild: AppRuntimeInfo;
hasMainWindow: () => boolean;
sendScannerCommand: (command: ScannerCommand | "probe-click") => void;
clickScreen: (x: number, y: number) => Promise<ClickResult>;
scannerStatus: () => ScannerStatusPayload;
warmOcr: (engine: "current" | "ik-traineddata") => Promise<unknown>;
loadReviewSamples: (limit?: number) => Promise<ReviewSampleListResult>;
listCaptureSources: () => Promise<CaptureSourceInfo[]>;
captureSource: (
@@ -25,6 +33,7 @@ interface DevControlServerDependencies {
focusGenshin?: boolean,
options?: CaptureOptions,
) => Promise<CaptureResult>;
requestShutdown?: (reason: string) => void;
}
function writeDevJson(res: http.ServerResponse, statusCode: number, payload: unknown) {
@@ -136,13 +145,30 @@ export function createDevControlServer(deps: DevControlServerDependencies): Serv
const url = new URL(req.url ?? "/", "http://127.0.0.1");
if (url.pathname === "/health") {
writeDevJson(res, 200, { ok: true, hotkeys: deps.registeredHotkeys, hasWindow: deps.hasMainWindow() });
writeDevJson(res, 200, { ok: true, hotkeys: deps.registeredHotkeys, hasWindow: deps.hasMainWindow(), appBuild: deps.appBuild });
return;
}
if (url.pathname === "/dev/shutdown") {
if (!deps.requestShutdown) {
writeDevJson(res, 501, { ok: false, error: "shutdown not supported" });
return;
}
const reason = url.searchParams.get("reason") || "dev-control shutdown requested";
writeDevJson(res, 200, { ok: true, appBuild: deps.appBuild, reason });
setTimeout(() => deps.requestShutdown?.(reason), 50);
return;
}
if (url.pathname === "/scanner/start") {
const limit = Number(url.searchParams.get("limit") ?? Number.NaN);
const command: ScannerCommand = Number.isFinite(limit) && limit > 0
? { type: "start-auto", scanLimit: limit }
const entry = url.searchParams.get("entry");
const engine = url.searchParams.get("engine");
const scanEntryMode = entry === "paimon-menu" || entry === "visible-inventory" || entry === "direct-inventory" || entry === "auto-entry"
? entry
: undefined;
const ocrEngine = engine === "ik-traineddata" ? "ik-traineddata" : engine === "current" ? "current" : undefined;
const hasLimit = Number.isFinite(limit) && limit > 0;
const command: ScannerCommand = hasLimit || scanEntryMode || ocrEngine
? { type: "start-auto", scanLimit: hasLimit ? limit : undefined, scanEntryMode, ocrEngine }
: "start-auto";
deps.sendScannerCommand(command);
writeDevJson(res, 200, { ok: true, command });
@@ -174,6 +200,154 @@ export function createDevControlServer(deps: DevControlServerDependencies): Serv
writeDevJson(res, 200, { ok: true, status: deps.scannerStatus() });
return;
}
if (url.pathname === "/scanner/ocr/warmup") {
const engineParam = url.searchParams.get("engine");
const engine = engineParam === "ik-traineddata" ? "ik-traineddata" : "current";
deps.warmOcr(engine)
.then((status) => writeDevJson(res, 200, { ok: true, status }))
.catch((error: unknown) => writeDevJson(res, 500, { ok: false, error: error instanceof Error ? error.message : String(error) }));
return;
}
if (url.pathname === "/scanner/lookup/status") {
const status = validateLookupPackage();
writeDevJson(res, status.valid ? 200 : 409, { ok: status.valid, status });
return;
}
if (url.pathname === "/scanner/lookup/regenerate") {
execFileAsync("node", ["scripts/generate-genshin-data.cjs"], { cwd: process.cwd(), windowsHide: true, timeout: 120000 })
.then(({ stdout, stderr }) => {
const status = validateLookupPackage();
writeDevJson(res, status.valid ? 200 : 409, { ok: status.valid, status, stdout, stderr });
})
.catch((error: unknown) => writeDevJson(res, 500, { ok: false, error: error instanceof Error ? error.message : String(error) }));
return;
}
if (url.pathname === "/scanner/benchmark-ocr") {
const limit = Math.max(1, Math.min(100, Number(url.searchParams.get("limit") ?? 1) || 1));
const sourceId = url.searchParams.get("sourceId");
const engineParam = url.searchParams.get("engine");
const profileParam = url.searchParams.get("profile");
const ocrProfile: "full" | "fast" = profileParam === "full" ? "full" : "fast";
const engines: Array<"current" | "ik-traineddata"> = engineParam === "compare"
? ["current", "ik-traineddata"]
: engineParam === "ik-traineddata"
? ["ik-traineddata"]
: ["current"];
deps.listCaptureSources()
.then(async (sources) => {
const source = findGenshinSource(sources, sourceId);
if (!source) {
writeDevJson(res, 404, { ok: false, error: "No Genshin capture source found.", sources: sourceListForError(sources) });
return;
}
const benchmarkSource = source;
async function runEngineBenchmark(engine: "current" | "ik-traineddata") {
const startedAt = Date.now();
const captures: Array<{
index: number;
elapsedMs: number;
ocrFields: number;
timedOut: boolean;
ocrSkipped: boolean;
artifactDetailConfidence: number;
sanctified: boolean;
prepareMs: number;
ocrMs: number;
totalMs: number;
ocrProfile?: "full" | "fast";
ocrWorkerPoolSize?: number;
ocrFieldMs?: Record<string, number>;
}> = [];
for (let index = 0; index < limit; index += 1) {
const captureStartedAt = Date.now();
const capture = await deps.captureSource(benchmarkSource.id, index === 0 ? 150 : 0, true, {
ocrMode: "artifact",
ocrProfile,
ocrEngine: engine,
omitFullFrame: true,
omitInventoryPreview: true,
skipOcrUnlessArtifactDetail: true,
});
captures.push({
index,
elapsedMs: Date.now() - captureStartedAt,
ocrFields: capture.ocr?.length ?? 0,
timedOut: Boolean(capture.ocrTimedOut),
ocrSkipped: Boolean(capture.ocrSkipped),
artifactDetailConfidence: capture.artifactDetail?.confidence ?? 0,
sanctified: Boolean(capture.sanctified),
prepareMs: capture.timings?.prepareMs ?? 0,
ocrMs: capture.timings?.ocrMs ?? 0,
totalMs: capture.timings?.totalMs ?? 0,
ocrProfile: capture.timings?.ocrProfile,
ocrWorkerPoolSize: capture.timings?.ocrWorkerPoolSize,
ocrFieldMs: capture.timings?.ocrFieldMs,
});
}
const elapsedMs = Date.now() - startedAt;
const timings = captures.map((capture) => capture.elapsedMs).sort((left, right) => left - right);
const ocrTimings = captures.map((capture) => capture.ocrMs).filter((value) => value > 0).sort((left, right) => left - right);
const averageMs = Math.round(elapsedMs / limit);
const averageOcrMs = ocrTimings.length > 0
? Math.round(ocrTimings.reduce((total, value) => total + value, 0) / ocrTimings.length)
: 0;
const percentile = (ratio: number) => timings[Math.min(timings.length - 1, Math.max(0, Math.ceil(timings.length * ratio) - 1))] ?? 0;
const ocrPercentile = (ratio: number) => ocrTimings[Math.min(ocrTimings.length - 1, Math.max(0, Math.ceil(ocrTimings.length * ratio) - 1))] ?? 0;
const ocrFieldTotals = captures.reduce<Record<string, { totalMs: number; count: number; maxMs: number }>>((fields, capture) => {
for (const [field, elapsed] of Object.entries(capture.ocrFieldMs ?? {})) {
const current = fields[field] ?? { totalMs: 0, count: 0, maxMs: 0 };
current.totalMs += elapsed;
current.count += 1;
current.maxMs = Math.max(current.maxMs, elapsed);
fields[field] = current;
}
return fields;
}, {});
const ocrFieldAverages = Object.fromEntries(
Object.entries(ocrFieldTotals).map(([field, timing]) => [
field,
{
averageMs: Math.round(timing.totalMs / Math.max(1, timing.count)),
maxMs: timing.maxMs,
count: timing.count,
},
]),
);
return {
engine,
nativeTesseract: "not-enabled",
workerPoolSize: captures.find((capture) => capture.ocrWorkerPoolSize)?.ocrWorkerPoolSize ?? null,
ocrProfile,
limit,
elapsedMs,
averageMs,
averageOcrMs,
minMs: timings[0] ?? 0,
p50Ms: percentile(0.5),
p90Ms: percentile(0.9),
maxMs: timings[timings.length - 1] ?? 0,
ocrP50Ms: ocrPercentile(0.5),
ocrP90Ms: ocrPercentile(0.9),
ocrFieldAverages,
projectedMs: {
artifacts20: averageMs * 20,
artifacts45: averageMs * 45,
artifacts100: averageMs * 100,
},
skippedOcrCaptures: captures.filter((capture) => capture.ocrSkipped).length,
captures,
};
}
const summaries = await Promise.all(engines.map((engine) => runEngineBenchmark(engine)));
writeDevJson(res, 200, {
ok: true,
summary: summaries.length === 1 ? summaries[0] : { mode: "compare", limit, ocrProfile, engines: summaries },
});
})
.catch((error: unknown) => writeDevJson(res, 500, { ok: false, error: error instanceof Error ? error.message : String(error) }));
return;
}
if (url.pathname === "/review/samples") {
deps.loadReviewSamples(Number(url.searchParams.get("limit") ?? 20))
.then((payload: unknown) => writeDevJson(res, 200, payload))
electron/ipc/captureHandlers.ts
+4 -1
View File
@@ -1,11 +1,12 @@
import { ipcMain } from "electron";
import type { CaptureOptions, CaptureResult, CaptureSourceInfo, ClickResult, ScrollResult, AutomationGuard } from "../../src/types/global.js";
import type { CaptureOptions, CaptureResult, CaptureSourceInfo, ClickResult, ScrollResult, AutomationGuard, KeyPressResult } from "../../src/types/global.js";
interface CaptureCommandDependencies {
listSources: () => Promise<CaptureSourceInfo[]>;
captureSource: (sourceId: string, delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult>;
clickScreen: (x: number, y: number) => Promise<ClickResult>;
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => Promise<ScrollResult>;
keyPress: (key: string) => Promise<KeyPressResult>;
getAutomationGuard: () => Promise<AutomationGuard>;
}
@@ -14,6 +15,7 @@ export function registerCaptureHandlers({
captureSource,
clickScreen,
scrollScreen,
keyPress,
getAutomationGuard,
}: CaptureCommandDependencies) {
ipcMain.handle("capture:listSources", async () => listSources());
@@ -22,5 +24,6 @@ export function registerCaptureHandlers({
});
ipcMain.handle("automation:clickScreen", async (_event, x: number, y: number) => clickScreen(x, y));
ipcMain.handle("automation:scrollScreen", async (_event, notches: number, anchorX?: number, anchorY?: number) => scrollScreen(notches, anchorX, anchorY));
ipcMain.handle("automation:keyPress", async (_event, key: string) => keyPress(key));
ipcMain.handle("automation:getGuard", async () => getAutomationGuard());
}
electron/main.ts
+593 -61
View File
@@ -2,9 +2,10 @@ import { app, BrowserWindow, Menu, desktopCapturer, dialog, globalShortcut, nati
import fs from "node:fs/promises";
import { existsSync } from "node:fs";
import type { Server } from "node:http";
import { cpus } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { createWorker } from "tesseract.js";
import { createWorker, PSM } from "tesseract.js";
import { createInputHelperService, type InputHelperService } from "./services/inputHelper.js";
import { createRepositoryContext, type RepositoryContext } from "./bootstrap/repositoryContext.js";
import { registerIpcHandlers } from "./bootstrap/ipcBootstrap.js";
@@ -15,6 +16,8 @@ import type {
CaptureResult,
GoodDatabase,
GoodImportFileResult,
OcrResult,
AppRuntimeInfo,
SaveResultWithPath,
ScannerCommand,
ScannerLearningRulePayload,
@@ -50,6 +53,8 @@ app.commandLine.appendSwitch("disable-gpu-sandbox");
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const isDev = Boolean(process.env.VITE_DEV_SERVER_URL);
const APP_RUNTIME_STARTED_AT = new Date().toISOString();
const APP_RUNTIME_SIGNATURE = "2026-07-07-ik32-fastsubstats-active-timing";
let mainWindow: BrowserWindow | null = null;
let overlayWindow: BrowserWindow | null = null;
@@ -211,6 +216,7 @@ async function readRuntimeInfo() {
ok: true,
isElevated: result.isElevated,
platform: result.platform,
appBuild: appRuntimeInfo(),
hotkeys: registeredHotkeys,
genshinFound: result.genshinFound,
genshinHwnd: result.genshinHwnd ?? undefined,
@@ -220,7 +226,7 @@ async function readRuntimeInfo() {
helperPid: result.helperPid,
};
} catch {
return { ok: false, isElevated: false, platform: process.platform, hotkeys: registeredHotkeys };
return { ok: false, isElevated: false, platform: process.platform, appBuild: appRuntimeInfo(), hotkeys: registeredHotkeys };
}
}
@@ -326,12 +332,40 @@ async function listCaptureSources() {
}));
}
async function toScreenPoint(x: number, y: number) {
const point = { x: Math.round(x), y: Math.round(y) };
const bounds = await getGenshinWindowBounds();
if (!bounds) return point;
const looksClientRelative =
point.x >= 0 &&
point.y >= 0 &&
point.x <= bounds.width &&
point.y <= bounds.height &&
(bounds.x !== 0 || bounds.y !== 0);
if (!looksClientRelative) return point;
return {
x: bounds.x + point.x,
y: bounds.y + point.y,
};
}
async function clickScreenCommand(x: number, y: number) {
return getInputHelperService().clickScreen(Math.round(x), Math.round(y));
const point = await toScreenPoint(x, y);
return getInputHelperService().clickScreen(point.x, point.y);
}
async function scrollScreenCommand(notches: number, anchorX?: number, anchorY?: number) {
return getInputHelperService().scrollScreen(notches, anchorX, anchorY);
if (typeof anchorX === "number" && typeof anchorY === "number") {
const point = await toScreenPoint(anchorX, anchorY);
return getInputHelperService().scrollScreen(notches, point.x, point.y);
}
return getInputHelperService().scrollScreen(notches);
}
async function keyPressCommand(key: string) {
return getInputHelperService().keyPress(key);
}
async function getAutomationGuardCommand() {
@@ -439,13 +473,19 @@ function startDevControlServer() {
if (!isDev || devControlServer) return;
devControlServer = createDevControlServer({
registeredHotkeys,
appBuild: appRuntimeInfo(),
hasMainWindow: () => Boolean(mainWindow && !mainWindow.isDestroyed()),
sendScannerCommand,
clickScreen: clickScreenCommand,
scannerStatus: () => scannerDevStatus,
scannerStatus: () => ({ ...scannerDevStatus, appBuild: appRuntimeInfo(), ocrWarmup: getOcrWarmupStatus() }),
warmOcr: (engine) => warmOcrWorkerPool(engine),
loadReviewSamples,
listCaptureSources,
captureSource,
requestShutdown: (reason) => {
console.log(`[dev-control] shutdown requested: ${reason}`);
app.quit();
},
});
}
@@ -489,60 +529,284 @@ function createOverlayWindow() {
});
}
function dataUrlToBuffer(dataUrl: string) {
const base64 = dataUrl.replace(/^data:image\/png;base64,/, "");
return Buffer.from(base64, "base64");
// Inventory Kamera keeps a pool of native Tesseract engines and scans artifact
// fields concurrently. Our fast artifact profile has four useful OCR parameter
// groups, so the default pool is four workers unless the machine is smaller or
// GAA_OCR_WORKERS explicitly overrides it.
const OCR_WORKER_POOL_SIZE = resolveOcrWorkerPoolSize();
const IK_TRAINEDDATA_LANG = "genshin_fast_09_04_21";
type OcrWorker = Awaited<ReturnType<typeof createWorker>>;
type OcrWorkerEngine = "current" | "ik-traineddata";
type OcrCropPayload = { id: string; label: string; image: Buffer };
type OcrWarmupStatus = {
engine: OcrWorkerEngine;
status: "cold" | "warming" | "ready" | "error";
workerPoolSize: number;
startedAt?: string;
readyAt?: string;
elapsedMs?: number;
error?: string;
};
type OcrWorkerSlot = {
worker: OcrWorker;
parametersKey: string;
};
type OcrWorkerPoolState = {
poolPromise: Promise<OcrWorkerSlot[]> | null;
runQueue: Promise<unknown>;
};
const ocrWorkerPools: Record<OcrWorkerEngine, OcrWorkerPoolState> = {
current: { poolPromise: null, runQueue: Promise.resolve() },
"ik-traineddata": { poolPromise: null, runQueue: Promise.resolve() },
};
const ocrWarmupStatuses: Record<OcrWorkerEngine, OcrWarmupStatus> = {
current: { engine: "current", status: "cold", workerPoolSize: OCR_WORKER_POOL_SIZE },
"ik-traineddata": { engine: "ik-traineddata", status: "cold", workerPoolSize: OCR_WORKER_POOL_SIZE },
};
function resolveOcrWorkerPoolSize() {
const requested = Number(process.env.GAA_OCR_WORKERS ?? Number.NaN);
if (Number.isFinite(requested) && requested > 0) {
return Math.max(1, Math.min(8, Math.floor(requested)));
}
const logicalCores = cpus().length || 4;
return Math.max(2, Math.min(4, logicalCores - 1));
}
// One shared OCR worker. Creating a Tesseract worker per capture added ~1s
// to every artifact during batch scans.
let ocrWorkerPromise: ReturnType<typeof createWorker> | null = null;
function getOcrWorker() {
if (!ocrWorkerPromise) {
ocrWorkerPromise = createWorker("eng");
}
return ocrWorkerPromise;
function appRuntimeInfo(): AppRuntimeInfo {
return {
signature: APP_RUNTIME_SIGNATURE,
pid: process.pid,
startedAt: APP_RUNTIME_STARTED_AT,
cwd: process.cwd(),
isDev,
expectedOcrWorkerPoolSize: OCR_WORKER_POOL_SIZE,
};
}
async function resetOcrWorker() {
const broken = ocrWorkerPromise;
ocrWorkerPromise = null;
function ocrEngineFromOptions(options: CaptureOptions = {}): OcrWorkerEngine {
return options.ocrEngine === "ik-traineddata" ? "ik-traineddata" : "current";
}
function ikTessdataCandidates() {
const envDir = process.env.IK_TESSDATA_DIR;
return [
envDir,
path.resolve(process.cwd(), "data", "tessdata"),
path.resolve(process.cwd(), "work", "Inventory_Kamera", "InventoryKamera", "tessdata"),
path.resolve(process.cwd(), "work", "refs", "Inventory_Kamera", "InventoryKamera", "tessdata"),
path.resolve(process.cwd(), "..", "_ik_ref_fork", "InventoryKamera", "tessdata"),
path.resolve(process.cwd(), "..", "_ik_ref", "InventoryKamera", "tessdata"),
path.resolve(process.env.USERPROFILE ?? "", "Desktop", "_ik_ref_fork", "InventoryKamera", "tessdata"),
path.resolve(process.env.USERPROFILE ?? "", "Desktop", "_ik_ref", "InventoryKamera", "tessdata"),
process.resourcesPath ? path.resolve(process.resourcesPath, "tessdata") : "",
].filter(Boolean) as string[];
}
function findIkTessdataDir() {
return ikTessdataCandidates().find((candidate) => existsSync(path.join(candidate, `${IK_TRAINEDDATA_LANG}.traineddata`))) ?? "";
}
function getOcrWorkerOptions(engine: OcrWorkerEngine) {
if (engine !== "ik-traineddata") return { lang: "eng", options: undefined };
const langPath = findIkTessdataDir();
if (!langPath) {
throw new Error(`IK traineddata not found. Set IK_TESSDATA_DIR or place ${IK_TRAINEDDATA_LANG}.traineddata in data/tessdata.`);
}
return {
lang: IK_TRAINEDDATA_LANG,
options: {
langPath,
gzip: false,
cachePath: app.isReady() ? path.join(app.getPath("userData"), "tessdata-cache") : path.resolve(process.cwd(), "outputs", "tessdata-cache"),
},
};
}
function getOcrWorkerPool(engine: OcrWorkerEngine) {
const state = ocrWorkerPools[engine];
if (!state.poolPromise) {
const { lang, options } = getOcrWorkerOptions(engine);
state.poolPromise = Promise.all(
Array.from({ length: OCR_WORKER_POOL_SIZE }, async () => ({
worker: await createWorker(lang, 1, options),
parametersKey: "",
})),
);
}
return state.poolPromise;
}
async function resetOcrWorker(engine?: OcrWorkerEngine) {
const engines: OcrWorkerEngine[] = engine ? [engine] : ["current", "ik-traineddata"];
await Promise.all(engines.map(async (engineId) => {
const state = ocrWorkerPools[engineId];
const broken = state.poolPromise;
state.poolPromise = null;
state.runQueue = Promise.resolve();
ocrWarmupStatuses[engineId] = { engine: engineId, status: "cold", workerPoolSize: OCR_WORKER_POOL_SIZE };
if (broken) {
try {
const worker = await broken;
await worker.terminate();
const pool = await broken;
await Promise.all(pool.map((slot) => slot.worker.terminate().catch(() => undefined)));
} catch {
// Worker never initialized; nothing to clean up.
// Workers never initialized; nothing to clean up.
}
}
}));
}
function ocrParametersForCrop(cropId: string) {
switch (cropId) {
case "artifact-level":
return {
tessedit_pageseg_mode: PSM.SINGLE_WORD,
tessedit_char_whitelist: "0123456789+",
};
case "artifact-main-stat-value":
return {
tessedit_pageseg_mode: PSM.SINGLE_LINE,
tessedit_char_whitelist: "0123456789.,%+",
};
case "inventory-count":
return {
tessedit_pageseg_mode: PSM.SINGLE_LINE,
tessedit_char_whitelist: "0123456789/",
};
case "artifact-name":
case "artifact-slot":
case "artifact-main-stat-label":
case "artifact-footer":
return {
tessedit_pageseg_mode: PSM.SINGLE_LINE,
tessedit_char_whitelist: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 .:'%-",
};
case "artifact-main-stat":
return {
tessedit_pageseg_mode: PSM.SINGLE_BLOCK,
tessedit_char_whitelist: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 .,%+",
};
case "artifact-substats":
return {
tessedit_pageseg_mode: PSM.SINGLE_BLOCK,
tessedit_char_whitelist: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 .,%+-",
};
default:
return {
tessedit_pageseg_mode: PSM.SINGLE_BLOCK,
tessedit_char_whitelist: "",
};
}
}
async function runOcrOnCrops(crops: Array<{ id: string; label: string; dataUrl: string }>) {
try {
const worker = await getOcrWorker();
const results = [];
for (const crop of crops) {
const recognized = await worker.recognize(dataUrlToBuffer(crop.dataUrl));
results.push({
id: crop.id,
label: crop.label,
text: cleanOcrText(crop.id, recognized.data.text),
confidence: Math.round(recognized.data.confidence),
function ocrParametersKey(cropId: string) {
const params = ocrParametersForCrop(cropId);
return `${params.tessedit_pageseg_mode}:${params.tessedit_char_whitelist}`;
}
async function applyOcrParameters(slot: OcrWorkerSlot, cropId: string) {
const params = ocrParametersForCrop(cropId);
const key = ocrParametersKey(cropId);
if (key === slot.parametersKey) return;
await slot.worker.setParameters(params);
slot.parametersKey = key;
}
function warmOcrWorkerPool(engine: OcrWorkerEngine = "current") {
const current = ocrWarmupStatuses[engine];
if (current.status === "warming" || current.status === "ready") return Promise.resolve(current);
const started = Date.now();
ocrWarmupStatuses[engine] = {
engine,
status: "warming",
workerPoolSize: OCR_WORKER_POOL_SIZE,
startedAt: new Date(started).toISOString(),
};
return getOcrWorkerPool(engine)
.then(async (pool) => {
await Promise.all(pool.map((slot) => applyOcrParameters(slot, "artifact-name")));
const readyAt = Date.now();
ocrWarmupStatuses[engine] = {
engine,
status: "ready",
workerPoolSize: OCR_WORKER_POOL_SIZE,
startedAt: ocrWarmupStatuses[engine].startedAt,
readyAt: new Date(readyAt).toISOString(),
elapsedMs: readyAt - started,
};
return ocrWarmupStatuses[engine];
})
.catch((error: unknown) => {
ocrWarmupStatuses[engine] = {
engine,
status: "error",
workerPoolSize: OCR_WORKER_POOL_SIZE,
startedAt: ocrWarmupStatuses[engine].startedAt,
elapsedMs: Date.now() - started,
error: error instanceof Error ? error.message : String(error),
};
return ocrWarmupStatuses[engine];
});
}
function getOcrWarmupStatus() {
return {
current: ocrWarmupStatuses.current,
"ik-traineddata": ocrWarmupStatuses["ik-traineddata"],
};
}
async function runOcrOnCrops(crops: OcrCropPayload[], engine: OcrWorkerEngine) {
try {
const pool = await getOcrWorkerPool(engine);
const results: OcrResult[] = new Array(crops.length);
let nextCropIndex = 0;
const workers = pool.slice(0, Math.min(pool.length, crops.length || 1));
await Promise.all(workers.map(async (slot) => {
while (nextCropIndex < crops.length) {
const index = nextCropIndex++;
const crop = crops[index];
if (!crop) continue;
results[index] = await recognizeCropWithSlot(slot, crop);
}
}));
return results;
} catch (error) {
await resetOcrWorker();
await resetOcrWorker(engine);
throw error;
}
}
async function runOcrOnCropsWithTimeout(crops: Array<{ id: string; label: string; dataUrl: string }>, timeoutMs = 6500) {
async function recognizeCropWithSlot(slot: OcrWorkerSlot, crop: OcrCropPayload): Promise<OcrResult> {
const startedAt = Date.now();
await applyOcrParameters(slot, crop.id);
const recognized = await slot.worker.recognize(crop.image);
return {
id: crop.id,
label: crop.label,
text: cleanOcrText(crop.id, recognized.data.text),
confidence: Math.round(recognized.data.confidence),
elapsedMs: Date.now() - startedAt,
};
}
function runQueuedOcrOnCrops(crops: OcrCropPayload[], engine: OcrWorkerEngine) {
const state = ocrWorkerPools[engine];
const run = state.runQueue.catch(() => undefined).then(() => runOcrOnCrops(crops, engine));
state.runQueue = run.catch(() => undefined);
return run;
}
async function runOcrOnCropsWithTimeout(crops: OcrCropPayload[], engine: OcrWorkerEngine, timeoutMs = 6500) {
let timeout: NodeJS.Timeout | undefined;
try {
return await Promise.race([
runOcrOnCrops(crops).then((ocr) => ({ ocr, timedOut: false })),
runQueuedOcrOnCrops(crops, engine).then((ocr) => ({ ocr, timedOut: false })),
new Promise<{ ocr: Awaited<ReturnType<typeof runOcrOnCrops>>; timedOut: boolean }>((resolve) => {
timeout = setTimeout(() => resolve({ ocr: [], timedOut: true }), timeoutMs);
}),
@@ -569,7 +833,7 @@ function cleanOcrText(cropId: string, text: string) {
return match ? `Equipped: ${match[1].replace(/[^A-Za-z'\-\s]/g, "").trim()}` : equipped;
}
if (cropId === "artifact-title") {
if (cropId === "artifact-title" || cropId === "artifact-name" || cropId === "artifact-slot" || cropId === "artifact-main-stat-label") {
return normalized
.filter((line) => /[A-Za-z]/.test(line))
.slice(0, 2)
@@ -583,10 +847,23 @@ function cleanOcrText(cropId: string, text: string) {
.join("\n");
}
if (cropId === "artifact-main-stat-value") {
return normalized
.map((line) => line.replace(/[^0-9.,%]/g, ""))
.find((line) => /[0-9]/.test(line)) ?? "";
}
if (cropId === "artifact-level") {
const value = normalized
.map((line) => line.replace(/[^0-9+]/g, ""))
.find((line) => /[0-9]/.test(line)) ?? "";
return value.startsWith("+") || value === "" ? value : `+${value}`;
}
if (cropId === "artifact-substats") {
return normalized
.filter((line) => /(\+|CRIT|ATK|DEF|HP|Energy|Elemental)/i.test(line))
.slice(0, 5)
.slice(0, 6)
.join("\n");
}
@@ -660,6 +937,169 @@ function isArtifactTextColor(bitmap: Buffer, index: number) {
return green >= 180 && red >= 140 && blue <= 95 && green > blue + 35 && red > blue + 10;
}
function hasEquippedFooterMarker(bitmap: Buffer, imageSize: { width: number; height: number }, rect: Electron.Rectangle) {
const safeRect = clampCaptureRect(rect, imageSize);
const strideX = Math.max(1, Math.floor(safeRect.width / 48));
const strideY = Math.max(1, Math.floor(safeRect.height / 18));
let hits = 0;
for (let y = safeRect.y; y < safeRect.y + safeRect.height; y += strideY) {
const rowOffset = y * imageSize.width * 4;
for (let x = safeRect.x; x < safeRect.x + safeRect.width; x += strideX) {
if (isEquippedFooterYellow(bitmap, rowOffset + x * 4)) {
hits++;
if (hits >= 10) return true;
}
}
}
return false;
}
function isSanctifiedArtifactPurple(bitmap: Buffer, index: number) {
const blue = bitmap[index];
const green = bitmap[index + 1];
const red = bitmap[index + 2];
return blue >= 220 && red >= 180 && red <= 245 && green >= 155 && green <= 220 && blue > red + 10;
}
function detectSanctifiedArtifactDetail(bitmap: Buffer, imageSize: { width: number; height: number }, detailRect: Electron.Rectangle) {
const safeRect = clampCaptureRect(detailRect, imageSize);
const x0 = Math.max(safeRect.x, Math.round(safeRect.x + safeRect.width * 0.0));
const x1 = Math.min(imageSize.width - 1, Math.round(safeRect.x + safeRect.width * 0.0606));
const y0 = Math.max(safeRect.y, Math.round(safeRect.y + safeRect.height * 0.3333));
const y1 = Math.min(imageSize.height - 1, Math.round(y0 + safeRect.height * 0.0526));
const strideX = Math.max(1, Math.floor(Math.max(1, x1 - x0) / 12));
const strideY = Math.max(1, Math.floor(Math.max(1, y1 - y0) / 10));
let hits = 0;
for (let y = y0; y <= y1; y += strideY) {
const rowOffset = y * imageSize.width * 4;
for (let x = x0; x <= x1; x += strideX) {
if (isSanctifiedArtifactPurple(bitmap, rowOffset + x * 4)) {
hits++;
if (hits >= 3) return true;
}
}
}
return false;
}
function analyzeArtifactDetailPanel(bitmap: Buffer, imageSize: { width: number; height: number }, rect: Electron.Rectangle) {
const safeRect = clampCaptureRect(rect, imageSize);
const strideX = Math.max(1, Math.floor(safeRect.width / 96));
const strideY = Math.max(1, Math.floor(safeRect.height / 160));
let orangeHits = 0;
let greenHits = 0;
let textHits = 0;
let titleOrangeHits = 0;
let upperTextHits = 0;
let lowerGreenHits = 0;
for (let y = safeRect.y; y < safeRect.y + safeRect.height; y += strideY) {
const rowOffset = y * imageSize.width * 4;
for (let x = safeRect.x; x < safeRect.x + safeRect.width; x += strideX) {
const index = rowOffset + x * 4;
const relativeY = (y - safeRect.y) / safeRect.height;
const isOrange = isArtifactTitleOrange(bitmap, index);
const isGreen = isSetTitleGreen(bitmap, index);
const isText = isArtifactTextColor(bitmap, index);
if (isOrange) {
orangeHits++;
if (relativeY <= 0.085) titleOrangeHits++;
}
if (isGreen) {
greenHits++;
if (relativeY >= 0.37 && relativeY <= 0.84) lowerGreenHits++;
}
if (isText) {
textHits++;
if (relativeY >= 0.08 && relativeY <= 0.37) upperTextHits++;
}
}
}
const confidence = Math.max(
0,
Math.min(100, Math.round(titleOrangeHits * 1.6 + upperTextHits * 1.2 + lowerGreenHits * 0.7)),
);
return {
present: titleOrangeHits >= 40 && (upperTextHits >= 20 || lowerGreenHits >= 40) && confidence >= 58,
confidence,
orangeHits,
greenHits,
textHits,
titleOrangeHits,
upperTextHits,
lowerGreenHits,
};
}
function isPaimonProfileLight(bitmap: Buffer, index: number) {
const blue = bitmap[index];
const green = bitmap[index + 1];
const red = bitmap[index + 2];
return red > 150 && green > 150 && blue > 150;
}
function isPaimonProfileCream(bitmap: Buffer, index: number) {
const blue = bitmap[index];
const green = bitmap[index + 1];
const red = bitmap[index + 2];
return red >= 195 && green >= 185 && blue >= 155;
}
function isPaimonMenuTileDark(bitmap: Buffer, index: number) {
const blue = bitmap[index];
const green = bitmap[index + 1];
const red = bitmap[index + 2];
return red >= 45 && red <= 115 && green >= 55 && green <= 125 && blue >= 70 && blue <= 150;
}
function sampleScreenZone(
bitmap: Buffer,
imageSize: { width: number; height: number },
zone: { x0: number; x1: number; y0: number; y1: number },
predicate: (bitmap: Buffer, index: number) => boolean,
) {
const x0 = Math.max(0, Math.floor(imageSize.width * zone.x0));
const x1 = Math.min(imageSize.width, Math.ceil(imageSize.width * zone.x1));
const y0 = Math.max(0, Math.floor(imageSize.height * zone.y0));
const y1 = Math.min(imageSize.height, Math.ceil(imageSize.height * zone.y1));
const strideX = Math.max(1, Math.floor((x1 - x0) / 135));
const strideY = Math.max(1, Math.floor((y1 - y0) / 90));
let hits = 0;
let samples = 0;
for (let y = y0; y < y1; y += strideY) {
const rowOffset = y * imageSize.width * 4;
for (let x = x0; x < x1; x += strideX) {
samples++;
if (predicate(bitmap, rowOffset + x * 4)) hits++;
}
}
return samples > 0 ? (hits / samples) * 100 : 0;
}
function analyzePaimonMenu(bitmap: Buffer, imageSize: { width: number; height: number }) {
const profileZone = { x0: 0.05, x1: 0.40, y0: 0, y1: 0.30 };
const menuTileZone = { x0: 0.06, x1: 0.39, y0: 0.32, y1: 0.98 };
const profileLightPct = sampleScreenZone(bitmap, imageSize, profileZone, isPaimonProfileLight);
const profileCreamPct = sampleScreenZone(bitmap, imageSize, profileZone, isPaimonProfileCream);
const menuTileDarkPct = sampleScreenZone(bitmap, imageSize, menuTileZone, isPaimonMenuTileDark);
const confidence = Math.max(0, Math.min(100, Math.round((profileLightPct - 20) * 1.2 + (profileCreamPct - 12) * 1.4 + (menuTileDarkPct - 25) * 1.1)));
return {
present: profileLightPct >= 35 && profileCreamPct >= 20 && menuTileDarkPct >= 40,
confidence,
profileLightPct: Math.round(profileLightPct * 10) / 10,
profileCreamPct: Math.round(profileCreamPct * 10) / 10,
menuTileDarkPct: Math.round(menuTileDarkPct * 10) / 10,
};
}
function waitDelay(ms: number) {
return new Promise<void>((resolve) => setTimeout(resolve, Math.max(0, Math.floor(ms))));
}
@@ -708,18 +1148,40 @@ function imageCropDataUrl(sourceImage: NativeImage, rect: Electron.Rectangle, im
return sourceImage.crop(safeRect).toDataURL();
}
function imageCropFingerprint(sourceImage: NativeImage, rect: Electron.Rectangle, imageSize: { width: number; height: number }) {
const safeRect = clampCaptureRect(rect, imageSize);
const bitmap = sourceImage.crop(safeRect).getBitmap();
let hash = 2166136261;
const stride = Math.max(4, Math.floor(bitmap.length / 4096) * 4);
for (let index = 0; index < bitmap.length; index += stride) {
hash ^= bitmap[index] ?? 0;
hash = Math.imul(hash, 16777619);
hash ^= bitmap[index + 1] ?? 0;
hash = Math.imul(hash, 16777619);
hash ^= bitmap[index + 2] ?? 0;
hash = Math.imul(hash, 16777619);
}
return `${safeRect.width}x${safeRect.height}:${(hash >>> 0).toString(16)}`;
}
// Preprocessed copy of a crop for OCR (ADR-009): upscale for more pixels, then
// grayscale + Otsu-binarize with inversion (artifact text is the bright
// foreground). The original crop is kept separately for the diagnostics UI.
function preprocessedCropDataUrl(sourceImage: NativeImage, rect: Electron.Rectangle, imageSize: { width: number; height: number }) {
// OCR receives a PNG buffer directly to avoid DataURL encode/decode churn in
// the auto-scan hot path.
function preprocessedCropPngBuffer(sourceImage: NativeImage, rect: Electron.Rectangle, imageSize: { width: number; height: number }, cropId = "") {
const safeRect = clampCaptureRect(rect, imageSize);
const upscaled = sourceImage.crop(safeRect).resize({ width: Math.max(1, safeRect.width * 2), quality: "best" });
const scale = cropId === "artifact-level" ? 3 : 2;
const upscaled = sourceImage.crop(safeRect).resize({ width: Math.max(1, safeRect.width * scale), quality: "best" });
const size = upscaled.getSize();
if (!size.width || !size.height) return upscaled.toDataURL();
const binarized = binarizeForOcr({ data: upscaled.getBitmap(), width: size.width, height: size.height });
if (!size.width || !size.height) return upscaled.toPNG();
const binarized = binarizeForOcr(
{ data: upscaled.getBitmap(), width: size.width, height: size.height },
cropId === "artifact-level" ? { contrast: 80 } : {},
);
return nativeImage
.createFromBitmap(Buffer.from(binarized.data), { width: binarized.width, height: binarized.height })
.toDataURL();
.toPNG();
}
function createCrops(
@@ -727,10 +1189,27 @@ function createCrops(
imageSize: { width: number; height: number },
detailRect: Electron.Rectangle,
inventoryRect: Electron.Rectangle,
options: CaptureOptions = {},
bitmap?: Buffer,
cropOptions: { sanctified?: boolean; skipOcr?: boolean } = {},
) {
const templates: CropTemplate[] = detailCropRects(detailRect, imageSize);
const isArtifactScanMode = options.ocrMode === "artifact";
const fastArtifactProfile = isArtifactScanMode && options.ocrProfile === "fast";
const omitCropImages = Boolean(options.omitCropImages);
const skipCropOcr = Boolean(cropOptions.skipOcr);
const templates: CropTemplate[] = detailCropRects(detailRect, imageSize, { ...cropOptions, fastProfile: fastArtifactProfile })
.filter((template) => {
if (fastArtifactProfile && (
template.id === "artifact-set-effects" ||
template.id === "artifact-slot" ||
template.id === "artifact-main-stat-value"
)) return false;
if (template.id === "artifact-footer" && (options.omitEquippedOcr || fastArtifactProfile)) return false;
if (!isArtifactScanMode || template.id !== "artifact-footer" || !bitmap) return true;
return hasEquippedFooterMarker(bitmap, imageSize, template.rect);
});
if (inventoryRect.width > 120 && inventoryRect.height > 80) {
if (!isArtifactScanMode && inventoryRect.width > 120 && inventoryRect.height > 80) {
templates.push({
id: "inventory-count",
label: "Inventory count",
@@ -741,12 +1220,14 @@ function createCrops(
return templates
.map((template) => {
const rect = clampCaptureRect(template.rect, imageSize);
const ocrEnabled = !skipCropOcr;
return {
id: template.id,
label: template.label,
rect,
dataUrl: imageCropDataUrl(sourceImage, rect, imageSize),
ocrDataUrl: preprocessedCropDataUrl(sourceImage, rect, imageSize),
dataUrl: omitCropImages ? undefined : imageCropDataUrl(sourceImage, rect, imageSize),
ocrImage: ocrEnabled ? preprocessedCropPngBuffer(sourceImage, rect, imageSize, template.id) : undefined,
ocrEnabled,
};
})
.filter((crop) => crop.rect.width > 0 && crop.rect.height > 0);
@@ -815,6 +1296,7 @@ async function buildCaptureResult(
captureTarget: CaptureResult["captureTarget"],
options: CaptureOptions = {},
) {
const buildStartedAt = Date.now();
const size = sourceImage.getSize();
if (!size.width || !size.height) {
throw new Error("Capture produced an empty image.");
@@ -822,22 +1304,52 @@ async function buildCaptureResult(
const bitmap = sourceImage.getBitmap();
const detailRect = inferDetailRect(bitmap, size);
const artifactDetail = analyzeArtifactDetailPanel(bitmap, size, detailRect);
const paimonMenu = analyzePaimonMenu(bitmap, size);
const sanctified = detectSanctifiedArtifactDetail(bitmap, size, detailRect);
const skipOcrForMissingDetail = Boolean(options.skipOcrUnlessArtifactDetail && !artifactDetail.present);
const shouldSkipOcr = Boolean(options.skipOcr || skipOcrForMissingDetail);
const inventoryRect = inferInventoryRect(size, detailRect);
const crops = createCrops(sourceImage, size, detailRect, inventoryRect);
const omitCrops = Boolean(options.omitCrops);
const crops = omitCrops
? []
: createCrops(sourceImage, size, detailRect, inventoryRect, options, bitmap, { sanctified, skipOcr: shouldSkipOcr });
const locked = options.omitLockState
? undefined
: (() => {
const lockRect = clampCaptureRect(lockIconCropRect(detailRect, size), size);
const lockImage = sourceImage.crop(lockRect);
const lockSize = lockImage.getSize();
const locked = lockSize.width > 0 && lockSize.height > 0
return lockSize.width > 0 && lockSize.height > 0
? detectLockState({ data: lockImage.getBitmap(), width: lockSize.width, height: lockSize.height })
: undefined;
const croppedPayload = crops.map((crop) => ({
})();
const omitFullFrame = Boolean(options.omitFullFrame || options.ocrMode === "artifact");
const omitDetailPreview = Boolean(options.omitDetailPreview);
const omitInventoryPreview = Boolean(options.omitInventoryPreview || options.ocrMode === "artifact");
const detailFingerprint = imageCropFingerprint(sourceImage, detailRect, size);
const inventoryFingerprint = imageCropFingerprint(sourceImage, inventoryRect, size);
const croppedPayload = crops
.filter((crop) => crop.ocrEnabled !== false)
.map((crop) => {
return crop.ocrImage
? {
id: crop.id,
label: crop.label,
// OCR reads the preprocessed (upscaled + binarized) crop; the original is
// kept below for the diagnostics UI.
dataUrl: crop.ocrDataUrl ?? crop.dataUrl,
}));
const recognized = options.skipOcr ? { ocr: [], timedOut: false } : await runOcrOnCropsWithTimeout(croppedPayload);
image: crop.ocrImage,
}
: null;
})
.filter((crop): crop is OcrCropPayload => Boolean(crop));
const prepareMs = Date.now() - buildStartedAt;
const ocrEngine = ocrEngineFromOptions(options);
const ocrStartedAt = Date.now();
const recognized = shouldSkipOcr
? { ocr: [], timedOut: false }
: await runOcrOnCropsWithTimeout(croppedPayload, ocrEngine);
const ocrMs = Date.now() - ocrStartedAt;
const totalMs = Date.now() - buildStartedAt;
const captureOcrEngine: CaptureOptions["ocrEngine"] = ocrEngine === "ik-traineddata" ? "ik-traineddata" : "current";
const count = parseInventoryCount(recognized.ocr);
return {
@@ -845,14 +1357,16 @@ async function buildCaptureResult(
name: sourceName,
width: size.width,
height: size.height,
dataUrl: sourceImage.toDataURL(),
dataUrl: omitFullFrame ? "" : sourceImage.toDataURL(),
capturedAt: new Date().toISOString(),
captureTarget,
detailDataUrl: imageCropDataUrl(sourceImage, detailRect, size),
inventoryDataUrl: imageCropDataUrl(sourceImage, inventoryRect, size),
detailDataUrl: omitDetailPreview ? undefined : imageCropDataUrl(sourceImage, detailRect, size),
inventoryDataUrl: omitInventoryPreview ? undefined : imageCropDataUrl(sourceImage, inventoryRect, size),
detailFingerprint,
inventoryFingerprint,
ocr: recognized.ocr,
ocrTimedOut: recognized.timedOut,
ocrSkipped: Boolean(options.skipOcr),
ocrSkipped: shouldSkipOcr,
crops: crops.map((crop) => ({
id: crop.id,
label: crop.label,
@@ -865,13 +1379,30 @@ async function buildCaptureResult(
dataUrl: crop.dataUrl,
})),
inventoryGrid: inferInventoryGrid(size, detailRect),
artifactDetail,
paimonMenu,
inventoryCount: count,
locked,
sanctified,
layout: {
aspect: aspectRatioLabel(size),
isSixteenNine: isSixteenNine(size),
warning: layoutSupportWarning(size),
},
timings: {
totalMs,
prepareMs,
ocrMs: shouldSkipOcr ? 0 : ocrMs,
cropCount: croppedPayload.length,
ocrEngine: captureOcrEngine,
ocrWorkerPoolSize: OCR_WORKER_POOL_SIZE,
ocrProfile: options.ocrProfile ?? "full",
ocrFieldMs: recognized.ocr.reduce<Record<string, number>>((fields, result) => {
if (typeof result.elapsedMs === "number") fields[result.id] = result.elapsedMs;
return fields;
}, {}),
ocrSkipped: shouldSkipOcr,
},
};
}
@@ -993,12 +1524,14 @@ function initializeAppLifecycle() {
) => captureSource(id, delayMs, focus, captureOptions),
clickScreen: (x: number, y: number) => clickScreenCommand(x, y),
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => scrollScreenCommand(notches, anchorX, anchorY),
keyPress: (key: string) => keyPressCommand(key),
getAutomationGuard: () => getAutomationGuardCommand(),
});
createMainWindow();
registerScannerHotkeys();
startDevControlServer();
void warmOcrWorkerPool("current");
});
app.on("activate", () => {
@@ -1025,4 +1558,3 @@ function initializeAppLifecycle() {
}
initializeAppLifecycle();
electron/preload.cjs
+1
View File
@@ -8,6 +8,7 @@ contextBridge.exposeInMainWorld("assistantApi", {
captureSource: (sourceId, delayMs = 0, focusGenshin = false, options) => ipcRenderer.invoke("capture:captureSource", sourceId, delayMs, focusGenshin, options),
clickScreen: (x, y) => ipcRenderer.invoke("automation:clickScreen", x, y),
scrollScreen: (notches, anchorX, anchorY) => ipcRenderer.invoke("automation:scrollScreen", notches, anchorX, anchorY),
keyPress: (key) => ipcRenderer.invoke("automation:keyPress", key),
getAutomationGuard: () => ipcRenderer.invoke("automation:getGuard"),
focusMainWindow: () => ipcRenderer.invoke("app:focusMainWindow"),
focusGenshin: () => ipcRenderer.invoke("automation:focusGenshin"),
electron/preload.ts
+1
View File
@@ -11,6 +11,7 @@ contextBridge.exposeInMainWorld("assistantApi", {
captureSource: (sourceId: string, delayMs = 0, focusGenshin = false, options?: CaptureOptions) => ipcRenderer.invoke("capture:captureSource", sourceId, delayMs, focusGenshin, options),
clickScreen: (x: number, y: number) => ipcRenderer.invoke("automation:clickScreen", x, y),
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => ipcRenderer.invoke("automation:scrollScreen", notches, anchorX, anchorY),
keyPress: (key: string) => ipcRenderer.invoke("automation:keyPress", key),
getAutomationGuard: () => ipcRenderer.invoke("automation:getGuard"),
focusMainWindow: () => ipcRenderer.invoke("app:focusMainWindow"),
focusGenshin: () => ipcRenderer.invoke("automation:focusGenshin"),
electron/repositories/reviewSamplesRepository.ts
+24 -3
View File
@@ -3,6 +3,9 @@ import path from "node:path";
import type { ReviewSampleListResult, ReviewSamplesRepositoryPort, ReviewSamplePayload } from "./contracts.js";
import type { ReviewSampleRecord, SaveResultWithPath } from "../../src/types/global.js";
const SMALL_FILE_LIMIT_BYTES = 8 * 1024 * 1024;
const TAIL_READ_LIMIT_BYTES = 32 * 1024 * 1024;
export class ReviewSamplesRepository implements ReviewSamplesRepositoryPort {
private readonly filePath: string;
@@ -12,9 +15,12 @@ export class ReviewSamplesRepository implements ReviewSamplesRepositoryPort {
async list(limit = 50): Promise<ReviewSampleListResult> {
try {
const raw = await fs.readFile(this.filePath, "utf8");
const lines = raw.split(/\r?\n/).filter(Boolean);
const safeLimit = Math.max(1, Math.min(200, Number(limit) || 50));
const stats = await fs.stat(this.filePath);
const raw = stats.size <= SMALL_FILE_LIMIT_BYTES
? await fs.readFile(this.filePath, "utf8")
: await readTailText(this.filePath, stats.size, TAIL_READ_LIMIT_BYTES);
const lines = raw.split(/\r?\n/).filter(Boolean);
const samples = lines
.slice(-safeLimit)
.map((line) => {
@@ -25,7 +31,8 @@ export class ReviewSamplesRepository implements ReviewSamplesRepositoryPort {
}
})
.filter(Boolean) as ReviewSampleRecord[];
return { ok: true, samples: samples.reverse(), total: lines.length, path: this.filePath };
const total = stats.size <= SMALL_FILE_LIMIT_BYTES ? lines.length : Math.max(samples.length, lines.length);
return { ok: true, samples: samples.reverse(), total, path: this.filePath };
} catch {
return { ok: true, samples: [], total: 0, path: this.filePath };
}
@@ -37,3 +44,17 @@ export class ReviewSamplesRepository implements ReviewSamplesRepositoryPort {
return { ok: true, path: this.filePath };
}
}
async function readTailText(filePath: string, fileSize: number, maxBytes: number) {
const bytesToRead = Math.min(fileSize, maxBytes);
const handle = await fs.open(filePath, "r");
try {
const buffer = Buffer.alloc(bytesToRead);
await handle.read(buffer, 0, bytesToRead, fileSize - bytesToRead);
const text = buffer.toString("utf8");
const firstNewline = text.indexOf("\n");
return fileSize > bytesToRead && firstNewline >= 0 ? text.slice(firstNewline + 1) : text;
} finally {
await handle.close();
}
}
electron/services/inputHelper.ts
+58
View File
@@ -7,6 +7,7 @@ import type {
FocusGenshinResult,
GdiCaptureResult,
HelperOperationResponse,
KeyPressResult,
WindowBounds,
RuntimeInfo,
ScrollResult,
@@ -115,6 +116,32 @@ function Send-MouseClickBatch {
return [Native.InputHelper]::SendInput(2, [Native.InputHelper+INPUT[]]@($down, $up), $inputSize)
}
function Send-KeyPressBatch {
param([int]$virtualKey)
$down = New-Object Native.InputHelper+INPUT
$down.type = 1
$down.mi.dx = $virtualKey
$up = New-Object Native.InputHelper+INPUT
$up.type = 1
$up.mi.dx = $virtualKey
# Same union bytes as KEYBDINPUT: dx low word = wVk, dy = dwFlags.
$up.mi.dy = 0x0002
return [Native.InputHelper]::SendInput(2, [Native.InputHelper+INPUT[]]@($down, $up), $inputSize)
}
function Resolve-VirtualKey {
param([string]$key)
switch ($key.ToUpperInvariant()) {
"ESC" { return 27 }
"ESCAPE" { return 27 }
"ENTER" { return 13 }
"B" { return 66 }
"C" { return 67 }
"1" { return 49 }
default { throw "unsupported key: $key" }
}
}
function Get-CursorPoint {
$pt = New-Object Native.InputHelper+POINT
[Native.InputHelper]::GetCursorPos([ref]$pt) | Out-Null
@@ -358,6 +385,21 @@ while ($true) {
$response.notchesSent = $sentTotal
$response.inputBlocked = (($count -gt 0) -and ($sentTotal -eq 0))
}
"key" {
$focusInfo = Focus-GenshinWindow
if ($focusInfo.focused -and -not $focusInfo.alreadyForeground) {
Start-Sleep -Milliseconds 120
}
$vk = Resolve-VirtualKey -key "$($cmd.key)"
$sent = Send-KeyPressBatch -virtualKey $vk
$response.key = "$($cmd.key)"
$response.focused = $focusInfo.focused
$response.foregroundProcess = $focusInfo.foregroundProcess
$response.targetProcess = $focusInfo.targetProcess
$response.isElevated = Get-CurrentProcessElevation
$response.eventsSent = $sent
$response.inputBlocked = ($sent -lt 2)
}
"bounds" {
$clientBounds = Get-GenshinClientBounds
if ($null -eq $clientBounds) {
@@ -545,6 +587,7 @@ export interface InputHelperService {
getGenshinWindowBounds(): Promise<WindowBounds | null>;
clickScreen(x: number, y: number): Promise<ClickResult>;
scrollScreen(notches: number, anchorX?: number, anchorY?: number): Promise<ScrollResult>;
keyPress(key: string): Promise<KeyPressResult>;
getAutomationGuard(): Promise<AutomationGuard>;
capturePrimaryScreenViaGdi(): Promise<GdiCaptureResult>;
dispose(): void;
@@ -641,6 +684,20 @@ export function createInputHelperService(options: { userDataPath: string; exePat
};
}
async function keyPress(key: string) {
const result = (await request("key", { key }, 8000)) as HelperOperationResponse;
return {
ok: Boolean(result.ok) && Number(result.eventsSent ?? 0) >= 2,
key,
focused: Boolean(result.focused),
foregroundProcess: typeof result.foregroundProcess === "string" ? result.foregroundProcess : undefined,
targetProcess: typeof result.targetProcess === "string" ? result.targetProcess : undefined,
isElevated: typeof result.isElevated === "boolean" ? result.isElevated : undefined,
inputBlocked: Boolean(result.inputBlocked),
eventsSent: Number(result.eventsSent ?? 0),
};
}
async function getAutomationGuard() {
const result = await request("cursor", {}, 4000);
return {
@@ -688,6 +745,7 @@ export function createInputHelperService(options: { userDataPath: string; exePat
getGenshinWindowBounds,
clickScreen,
scrollScreen,
keyPress,
getAutomationGuard,
capturePrimaryScreenViaGdi,
dispose: () => inputHelper.dispose(),
native/input-helper/Program.cs
+42
View File
@@ -169,6 +169,23 @@ internal static class Program
break;
}
case "key":
{
var info = FocusGenshinWindow();
if (info.Focused && !info.AlreadyForeground) Thread.Sleep(120);
var key = GetString(root, "key");
var sent = SendKeyPressBatch(ResolveVirtualKey(key));
response["key"] = key;
response["focused"] = info.Focused;
response["foregroundProcess"] = info.ForegroundProcess;
response["targetProcess"] = info.TargetProcess;
response["isElevated"] = IsElevated();
response["eventsSent"] = sent;
response["inputBlocked"] = sent < 2;
break;
}
case "bounds":
{
var bounds = GetGenshinClientBounds();
@@ -423,6 +440,30 @@ internal static class Program
return Native.SendInput(1, inputs, Marshal.SizeOf<Native.INPUT>());
}
private static uint SendKeyPressBatch(int virtualKey)
{
var inputs = new Native.INPUT[2];
inputs[0].type = 1; // INPUT_KEYBOARD
inputs[0].mi.dx = virtualKey; // same union bytes as KEYBDINPUT.wVk
inputs[1].type = 1;
inputs[1].mi.dx = virtualKey;
inputs[1].mi.dy = Native.KEYEVENTF_KEYUP; // same union bytes as KEYBDINPUT.dwFlags
return Native.SendInput(2, inputs, Marshal.SizeOf<Native.INPUT>());
}
private static int ResolveVirtualKey(string key)
{
return key.Trim().ToUpperInvariant() switch
{
"ESC" or "ESCAPE" => 0x1B,
"ENTER" => 0x0D,
"B" => 0x42,
"C" => 0x43,
"1" => 0x31,
_ => throw new ArgumentOutOfRangeException(nameof(key), $"unsupported key: {key}")
};
}
private static string GetString(JsonElement root, string name)
=> root.TryGetProperty(name, out var value) ? value.ToString() : "";
@@ -445,6 +486,7 @@ internal static class Native
public const uint MOUSEEVENTF_LEFTDOWN = 0x0002;
public const uint MOUSEEVENTF_LEFTUP = 0x0004;
public const uint MOUSEEVENTF_WHEEL = 0x0800;
public const int KEYEVENTF_KEYUP = 0x0002;
public const uint SPI_GETFOREGROUNDLOCKTIMEOUT = 0x2000;
public const uint SPI_SETFOREGROUNDLOCKTIMEOUT = 0x2001;
public const uint SPIF_SENDCHANGE = 0x0002;
package.json
+6
View File
@@ -16,6 +16,12 @@
"lint": "tsc --noEmit",
"test": "vitest run",
"eval": "vitest run src/eval/ocrEval.test.ts",
"scan:soak": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1",
"scan:goal": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun",
"scan:goal:current": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun -ScanEngine current",
"scan:goal:ik": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun -ScanEngine ik-traineddata",
"scan:goal:compare": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -GoalRun -ScanEngine compare",
"scan:assessment:test": "powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\live-soak.ps1 -SelfTestAssessment",
"helper:build": "dotnet publish native/input-helper/InputHelper.csproj -c Release -o native/input-helper/bin/publish",
"data:genshin": "node scripts/generate-genshin-data.cjs"
},
scripts/dev-admin-start.ps1
+16
View File
@@ -56,6 +56,22 @@ try {
& (Join-Path $PSScriptRoot "kill-stale-instances.ps1")
$devPort = 5173
$devControlPort = 17317
$devControlInUse = $null
try {
$devControlInUse = Get-NetTCPConnection -LocalPort $devControlPort -State Listen -ErrorAction Stop | Select-Object -First 1
} catch {
# Get-NetTCPConnection kann auf manchen Systemen fehlen; das ist kein Fehler.
$devControlInUse = $null
}
if ($devControlInUse) {
$portOwner = Get-Process -Id $devControlInUse.OwningProcess -ErrorAction SilentlyContinue
Write-Host ""
Write-Host "FEHLER: Dev-Control-Port $devControlPort ist noch belegt (Prozess: $($portOwner.ProcessName), PID $($devControlInUse.OwningProcess))." -ForegroundColor Red
Write-Host "Das wuerde Live-Tests gegen eine alte App-Instanz laufen lassen. Schliesse die alte App oder beende diesen Prozess und starte npm run dev:admin erneut." -ForegroundColor Red
throw "Dev-Control-Port $devControlPort ist durch PID $($devControlInUse.OwningProcess) belegt."
}
$portInUse = $null
try {
$portInUse = Get-NetTCPConnection -LocalPort $devPort -State Listen -ErrorAction Stop | Select-Object -First 1
scripts/generate-genshin-data.cjs
+106
View File
@@ -44,6 +44,9 @@ const artifactPieces = artifacts
const slotByPiece = Object.fromEntries(artifactPieces.map((piece) => [piece.name, piece.slot]));
const setByPiece = Object.fromEntries(artifactPieces.map((piece) => [piece.name, piece.setName]));
const setToPieces = Object.fromEntries(
artifacts.map((set) => [set.name, artifactPieces.filter((piece) => piece.setName === set.name).map((piece) => piece.name)]),
);
const mainStats = [
'Elemental Mastery',
@@ -198,6 +201,23 @@ const data = {
'Qiqi ': 'Qiqi',
},
},
lookup: {
normalizedKeys: {
sets: normalizedMap(artifacts.map((set) => set.name)),
pieces: normalizedMap(artifactPieces.map((piece) => piece.name)),
slots: normalizedMap(['Flower of Life', 'Plume of Death', 'Sands of Eon', 'Goblet of Eonothem', 'Circlet of Logos']),
stats: normalizedMap([...mainStats, ...substats]),
characters: normalizedMap(characters.map((character) => character.name)),
},
goodKeys: {
sets: Object.fromEntries(artifacts.map((set) => [set.name, goodKey(set.name)])),
pieces: Object.fromEntries(artifactPieces.map((piece) => [piece.name, goodKey(piece.name)])),
stats: Object.fromEntries([...mainStats, ...substats].map((stat) => [stat, goodStatKey(stat)])),
characters: Object.fromEntries(characters.map((character) => [character.name, goodKey(character.name)])),
},
setToPieces,
validation: validateLookupPackage({ artifacts, artifactPieces, characters, mainStats, substats, setToPieces }),
},
uiProfiles: {
artifactDetailEn: {
language: 'English',
@@ -231,3 +251,89 @@ function slotFromRelicType(relicType) {
return '';
}
}
function normalizeLookupKey(value) {
return String(value)
.toLowerCase()
.normalize('NFKD')
.replace(/['’]/g, '')
.replace(/[^a-z0-9]+/g, '');
}
function normalizedMap(values) {
return Object.fromEntries(values.filter(Boolean).map((value) => [normalizeLookupKey(value), value]));
}
function goodKey(value) {
return String(value)
.replace(/['’]/g, '')
.replace(/[^A-Za-z0-9]+(.)/g, (_match, next) => String(next).toUpperCase())
.replace(/^[a-z]/, (first) => first.toUpperCase())
.replace(/[^A-Za-z0-9]/g, '');
}
function goodStatKey(stat) {
switch (stat) {
case 'HP':
return 'hp';
case 'HP%':
return 'hp_';
case 'ATK':
return 'atk';
case 'ATK%':
return 'atk_';
case 'DEF':
return 'def';
case 'DEF%':
return 'def_';
case 'Elemental Mastery':
return 'eleMas';
case 'Energy Recharge':
return 'enerRech_';
case 'CRIT Rate':
return 'critRate_';
case 'CRIT DMG':
return 'critDMG_';
case 'Healing Bonus':
return 'heal_';
case 'Physical DMG Bonus':
return 'physical_dmg_';
default:
return stat.toLowerCase().replace(' dmg bonus', '_dmg_').replace(/\s+/g, '');
}
}
function validateLookupPackage({ artifacts, artifactPieces, characters, mainStats, substats, setToPieces }) {
const errors = [];
const warnings = [];
const setNames = new Set(artifacts.map((set) => set.name));
const slotNames = new Set(['Flower of Life', 'Plume of Death', 'Sands of Eon', 'Goblet of Eonothem', 'Circlet of Logos']);
const goodSetKeys = new Set();
for (const set of artifacts) {
const key = goodKey(set.name);
if (goodSetKeys.has(key)) errors.push(`Duplicate GOOD set key: ${key}`);
goodSetKeys.add(key);
if ((setToPieces[set.name] ?? []).length === 0) warnings.push(`Set has no pieces: ${set.name}`);
}
for (const piece of artifactPieces) {
if (!setNames.has(piece.setName)) errors.push(`Piece ${piece.name} references missing set ${piece.setName}`);
if (!slotNames.has(piece.slot)) errors.push(`Piece ${piece.name} references missing slot ${piece.slot}`);
}
if (!characters.length) warnings.push('No characters generated.');
if (!mainStats.length || !substats.length) errors.push('Stats were not generated.');
return {
valid: errors.length === 0,
errors,
warnings,
summary: {
artifactSets: artifacts.length,
artifactPieces: artifactPieces.length,
characters: characters.length,
stats: mainStats.length + substats.length,
},
};
}
scripts/kill-stale-instances.ps1
+53 -4
View File
@@ -16,8 +16,25 @@
$ErrorActionPreference = "Stop"
$project = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..")).Path
$electronPath = Join-Path $project "node_modules\electron\dist\electron.exe"
$devControlPort = 17317
$killed = 0
$candidatePids = @{}
try {
$health = Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:$devControlPort/health" -TimeoutSec 2 -ErrorAction Stop
if ($health.appBuild -and $health.appBuild.cwd -eq $project) {
Write-Host "Dev-Control-Port $devControlPort gehoert zu diesem Projekt (PID $($health.appBuild.pid), Signatur $($health.appBuild.signature)). Versuche Self-Shutdown..."
try {
Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:$devControlPort/dev/shutdown?reason=restart" -TimeoutSec 2 -ErrorAction Stop | Out-Null
Start-Sleep -Milliseconds 900
} catch {
Write-Host "Self-Shutdown nicht verfuegbar oder fehlgeschlagen: $($_.Exception.Message)" -ForegroundColor Yellow
}
}
} catch {
# No dev-control server or an older/stuck process; continue with process scan.
}
Get-CimInstance Win32_Process |
Where-Object {
@@ -26,15 +43,47 @@ Get-CimInstance Win32_Process |
($_.Name -eq "powershell.exe" -and $_.CommandLine -like "*input-helper.ps1*")
} |
ForEach-Object {
Write-Host "Beende alte Instanz: $($_.Name) (PID $($_.ProcessId))"
Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue
$candidatePids[[int]$_.ProcessId] = $_.Name
}
try {
Get-NetTCPConnection -LocalPort $devControlPort -State Listen -ErrorAction Stop |
ForEach-Object {
if ($_.OwningProcess -gt 0) {
$owner = Get-Process -Id $_.OwningProcess -ErrorAction SilentlyContinue
$candidatePids[[int]$_.OwningProcess] = if ($owner) { "$($owner.ProcessName) port $devControlPort" } else { "port $devControlPort owner" }
}
}
} catch {
# Get-NetTCPConnection can be unavailable on some machines; process matching
# above still handles the normal non-elevated path.
}
$failed = 0
foreach ($entry in $candidatePids.GetEnumerator()) {
Write-Host "Beende alte Instanz: $($entry.Value) (PID $($entry.Key))"
try {
Stop-Process -Id $entry.Key -Force -ErrorAction Stop
$killed++
} catch {
Write-Host "WARNUNG: Konnte PID $($entry.Key) nicht beenden: $($_.Exception.Message)" -ForegroundColor Yellow
$failed++
}
}
if ($killed -gt 0) {
# Windows braucht einen Moment, um Ports/Handles wirklich freizugeben.
Start-Sleep -Milliseconds 500
Write-Host "$killed alte Prozess(e) beendet."
} else {
Write-Host "Keine alten Instanzen gefunden."
}
if ($failed -gt 0) {
Write-Host "$failed alte Prozess(e) konnten nicht beendet werden. Wenn das ein Admin-Prozess ist, starte npm run dev:admin und bestaetige UAC oder schliesse die alte App manuell." -ForegroundColor Yellow
throw "$failed alte Prozess(e) konnten nicht beendet werden."
}
if ($killed -eq 0 -and $failed -eq 0) {
Write-Host "Keine alten Instanzen gefunden."
} else {
Start-Sleep -Milliseconds 300
}
scripts/live-soak.ps1
+685
View File
@@ -0,0 +1,685 @@
param(
[string]$BaseUrl = "http://127.0.0.1:17317",
[int[]]$ProbeIndices = @(1, 3),
[int[]]$Limits = @(2, 5, 10, 20),
[int]$PollIntervalSeconds = 2,
[int]$TimeoutSeconds = 600,
[string]$OutputRoot = (Join-Path (Resolve-Path -LiteralPath ".").Path "outputs\live-soak"),
[switch]$GoalRun,
[ValidateSet("current", "ik-traineddata", "compare")]
[string]$ScanEngine = "current",
[switch]$BenchmarkOcr,
[int]$BenchmarkLimit = 5,
[ValidateSet("current", "ik-traineddata", "compare")]
[string]$BenchmarkEngine = "compare",
[ValidateSet("fast", "full")]
[string]$BenchmarkProfile = "fast",
[switch]$SkipSmartCapture,
[switch]$SaveFullReviewSamples,
[switch]$AllowStaleBuild,
[string]$ExpectedAppSignature,
[switch]$ContinueAfterBlocked,
[switch]$SelfTestAssessment
)
$ErrorActionPreference = "Stop"
if ([string]::IsNullOrWhiteSpace($ExpectedAppSignature)) {
$mainPath = Join-Path (Resolve-Path -LiteralPath ".").Path "electron\main.ts"
if (Test-Path -LiteralPath $mainPath) {
$mainSource = Get-Content -LiteralPath $mainPath -Raw
$match = [regex]::Match($mainSource, 'APP_RUNTIME_SIGNATURE\s*=\s*"([^"]+)"')
if ($match.Success) {
$ExpectedAppSignature = $match.Groups[1].Value
}
}
}
function ConvertTo-SafeFilePart([string]$Value) {
$safe = $Value -replace "[^A-Za-z0-9._-]+", "-"
$safe = $safe.Trim("-")
if ($safe.Length -eq 0) { return "item" }
if ($safe.Length -gt 80) { return $safe.Substring(0, 80) }
return $safe
}
function Invoke-DevJson([string]$Path) {
$uri = if ($Path.StartsWith("http")) { $Path } else { "$BaseUrl$Path" }
Invoke-RestMethod -Method Get -Uri $uri -TimeoutSec 60
}
function Save-Json([string]$Name, [object]$Payload) {
$path = Join-Path $RunDir "$(ConvertTo-SafeFilePart $Name).json"
$Payload | ConvertTo-Json -Depth 30 | Set-Content -LiteralPath $path -Encoding UTF8
return $path
}
function Assert-CurrentAppBuild([object]$HealthPayload) {
if ($AllowStaleBuild) { return }
if ($null -eq $HealthPayload.appBuild) {
throw "Dev endpoint is stale: /health has no appBuild signature. Close the old elevated Genshin Artifact Assistant/Electron instance, restart with npm run dev:admin, then rerun this script. Use -AllowStaleBuild only for debugging old instances."
}
if ([string]::IsNullOrWhiteSpace([string]$HealthPayload.appBuild.signature)) {
throw "Dev endpoint is stale: appBuild.signature is empty. Restart the elevated app before live scanner timing."
}
if (-not [string]::IsNullOrWhiteSpace($ExpectedAppSignature) -and [string]$HealthPayload.appBuild.signature -ne $ExpectedAppSignature) {
throw "Dev endpoint is stale: appBuild.signature='$($HealthPayload.appBuild.signature)' but current source expects '$ExpectedAppSignature'. Close the old Electron instance, restart with npm run dev:admin, then rerun this script. Use -AllowStaleBuild only for debugging old instances."
}
}
function Get-ScannerStatus {
Invoke-DevJson "/scanner/status"
}
function Test-ProbeSucceeded([object]$ProbePayload) {
if ($ProbePayload.ok) { return $true }
if ($ProbePayload.changed) { return $true }
return $false
}
function Get-CompletedScanSummary([object]$StatusPayload, [int]$Limit) {
if ($null -ne $StatusPayload.status.summary) {
return $StatusPayload.status.summary
}
$reviewStatus = [string]$StatusPayload.status.reviewStatus
$status = "done"
if ($reviewStatus -match "blockiert") {
$status = "blocked"
} elseif ($reviewStatus -match "gestoppt") {
$status = "stopped"
}
$stats = $StatusPayload.status.stats
return [pscustomobject]@{
mode = "Automatischer Scan"
status = $status
clicked = [int]$stats.clicked
attempted = [int]$stats.attempted
verified = [int]$stats.verified
parsed = [int]$stats.parsed
stored = [int]$stats.stored
review = [int]$stats.review
duplicates = [int]$stats.duplicates
misses = [int]$stats.misses
pages = [int]$stats.pages
targetCount = $Limit
gridLabel = $reviewStatus
}
}
function Convert-ScanTimingSummary([object]$StatusPayload, [int]$Limit, [string]$Engine) {
$summary = Get-CompletedScanSummary $StatusPayload $Limit
$stats = $StatusPayload.status.stats
return [pscustomobject]@{
engine = $Engine
limit = $Limit
status = $summary.status
clicked = $summary.clicked
attempted = $summary.attempted
verified = $summary.verified
parsed = $summary.parsed
stored = $summary.stored
review = $summary.review
duplicates = $summary.duplicates
misses = $summary.misses
pages = $summary.pages
elapsedMs = [int]$stats.elapsedMs
activeScanMs = [int]$stats.activeScanMs
writeFlushMs = [int]$stats.writeFlushMs
averageMsPerParsed = [int]$stats.averageMsPerParsed
activeAverageMsPerParsed = [int]$stats.activeAverageMsPerParsed
artifactsPerMinute = [double]$stats.artifactsPerMinute
activeArtifactsPerMinute = [double]$stats.activeArtifactsPerMinute
projectedMsFor100 = [int]$stats.projectedMsFor100
activeProjectedMsFor100 = [int]$stats.activeProjectedMsFor100
averageCaptureMs = [int]$stats.averageCaptureMs
averageOcrMs = [int]$stats.averageOcrMs
captureP50Ms = [int]$stats.captureP50Ms
captureP90Ms = [int]$stats.captureP90Ms
ocrP50Ms = [int]$stats.ocrP50Ms
ocrP90Ms = [int]$stats.ocrP90Ms
cardReadyMs = [int]$stats.cardReadyMs
cardReadyCount = [int]$stats.cardReadyCount
averageCardReadyMs = [int]$stats.averageCardReadyMs
scrollReadyMs = [int]$stats.scrollReadyMs
scrollReadyCount = [int]$stats.scrollReadyCount
averageScrollReadyMs = [int]$stats.averageScrollReadyMs
captureMs = [int]$stats.captureMs
ocrMs = [int]$stats.ocrMs
reviewStatus = [string]$StatusPayload.status.reviewStatus
}
}
function Write-ScanTimingLine([object]$Timing) {
Write-Host ("engine={0} limit={1} status={2} parsed={3} review={4} miss={5} elapsed={6}ms active={7}ms flush={8}ms avg={9}ms activeAvg={10}ms captureAvg={11}ms captureP50={12}ms captureP90={13}ms ocrAvg={14}ms ocrP50={15}ms ocrP90={16}ms cardReadyAvg={17}ms scrollReadyAvg={18}ms ppm={19} activePpm={20} projected100={21}ms activeProjected100={22}ms" -f `
$Timing.engine,
$Timing.limit,
$Timing.status,
$Timing.parsed,
$Timing.review,
$Timing.misses,
$Timing.elapsedMs,
$Timing.activeScanMs,
$Timing.writeFlushMs,
$Timing.averageMsPerParsed,
$Timing.activeAverageMsPerParsed,
$Timing.averageCaptureMs,
$Timing.captureP50Ms,
$Timing.captureP90Ms,
$Timing.averageOcrMs,
$Timing.ocrP50Ms,
$Timing.ocrP90Ms,
$Timing.averageCardReadyMs,
$Timing.averageScrollReadyMs,
$Timing.artifactsPerMinute,
$Timing.activeArtifactsPerMinute,
$Timing.projectedMsFor100,
$Timing.activeProjectedMsFor100)
}
function Get-TimingBottleneck([object]$Timing) {
$parts = @(
[pscustomobject]@{ name = "ocr"; value = [int]$Timing.averageOcrMs },
[pscustomobject]@{ name = "capture"; value = [int]$Timing.averageCaptureMs },
[pscustomobject]@{ name = "card-ready"; value = [int]$Timing.averageCardReadyMs },
[pscustomobject]@{ name = "scroll-ready"; value = [int]$Timing.averageScrollReadyMs }
) | Sort-Object -Property value -Descending
if ($parts.Count -eq 0 -or $parts[0].value -le 0) { return "unknown" }
return $parts[0].name
}
function Get-TimingRecommendation([object]$Timing) {
$bottleneck = Get-TimingBottleneck $Timing
switch ($bottleneck) {
"ocr" { return "OCR dominates; compare engine, crop count, worker pool, and parser-derived fields first." }
"capture" { return "Capture dominates; reduce payloads/crops and avoid full-frame or Base64 work in the hot loop." }
"card-ready" { return "Card-ready dominates; tune detail fingerprint polling against IK's 200ms item wait." }
"scroll-ready" { return "Scroll-ready dominates; tune page fingerprint polling against IK's 100ms fast-scroll wait." }
default { return "No dominant timing component detected; inspect misses/review/duplicates and raw diagnostic events." }
}
}
function Get-ScanQuality([object]$Timing) {
$parsed = [int]$Timing.parsed
$misses = [int]$Timing.misses
$review = [int]$Timing.review
$target = [int]$Timing.limit
$processed = [Math]::Max(1, $parsed + $misses)
$parseCoverage = if ($target -gt 0) { [Math]::Round($parsed / $target, 4) } else { 0 }
$missRate = [Math]::Round($misses / $processed, 4)
$reviewRate = if ($parsed -gt 0) { [Math]::Round($review / $parsed, 4) } else { 1 }
$statusOk = [string]$Timing.status -eq "done"
$coverageOk = $parsed -ge $target
$missOk = $missRate -le 0.02
$reviewOk = $reviewRate -le 0.15
$qualityPenalty = [Math]::Round(($missRate * 1000000) + ($reviewRate * 250000) + ((1 - $parseCoverage) * 1000000), 0)
$qualityDecision = if (-not $statusOk) {
"not-qualified: scan did not finish cleanly"
} elseif (-not $coverageOk) {
"not-qualified: parsed fewer artifacts than requested"
} elseif (-not $missOk) {
"not-qualified: miss rate above 2%"
} elseif (-not $reviewOk) {
"not-qualified: review rate above 15%"
} else {
"qualified"
}
return [pscustomobject]@{
parseCoverage = $parseCoverage
missRate = $missRate
reviewRate = $reviewRate
qualityDecision = $qualityDecision
qualityPenalty = $qualityPenalty
qualified = ($statusOk -and $coverageOk -and $missOk -and $reviewOk)
}
}
function New-PerformanceAssessment([object[]]$Summaries) {
$limitReports = @()
foreach ($group in ($Summaries | Group-Object -Property limit | Sort-Object { [int]$_.Name })) {
$entries = @($group.Group | ForEach-Object {
$quality = Get-ScanQuality $_
$_ | Add-Member -NotePropertyName parseCoverage -NotePropertyValue $quality.parseCoverage -Force
$_ | Add-Member -NotePropertyName missRate -NotePropertyValue $quality.missRate -Force
$_ | Add-Member -NotePropertyName reviewRate -NotePropertyValue $quality.reviewRate -Force
$_ | Add-Member -NotePropertyName qualityDecision -NotePropertyValue $quality.qualityDecision -Force
$_ | Add-Member -NotePropertyName qualityPenalty -NotePropertyValue $quality.qualityPenalty -Force
$_ | Add-Member -NotePropertyName qualified -NotePropertyValue $quality.qualified -Force
$_
} | Sort-Object -Property @{ Expression = "qualified"; Descending = $true }, qualityPenalty, activeAverageMsPerParsed, averageMsPerParsed)
if ($entries.Count -eq 0) { continue }
$winner = $entries[0]
$engineReports = @()
foreach ($entry in $entries) {
$engineReports += [pscustomobject]@{
engine = $entry.engine
status = $entry.status
parsed = $entry.parsed
review = $entry.review
misses = $entry.misses
parseCoverage = $entry.parseCoverage
missRate = $entry.missRate
reviewRate = $entry.reviewRate
qualified = $entry.qualified
qualityDecision = $entry.qualityDecision
qualityPenalty = $entry.qualityPenalty
activeAverageMsPerParsed = $entry.activeAverageMsPerParsed
activeProjectedMsFor100 = $entry.activeProjectedMsFor100
averageOcrMs = $entry.averageOcrMs
averageCaptureMs = $entry.averageCaptureMs
averageCardReadyMs = $entry.averageCardReadyMs
averageScrollReadyMs = $entry.averageScrollReadyMs
bottleneck = Get-TimingBottleneck $entry
recommendation = Get-TimingRecommendation $entry
}
}
$limitReports += [pscustomobject]@{
limit = [int]$group.Name
winnerEngine = $winner.engine
winnerQualified = $winner.qualified
winnerMissRate = $winner.missRate
winnerReviewRate = $winner.reviewRate
winnerActiveAverageMsPerParsed = $winner.activeAverageMsPerParsed
winnerActiveProjectedMsFor100 = $winner.activeProjectedMsFor100
engines = $engineReports
}
}
$goal100 = @($limitReports | Where-Object { $_.limit -eq 100 } | Select-Object -First 1)
return [pscustomobject]@{
createdAt = (Get-Date).ToString("o")
goal100 = if ($goal100.Count -gt 0) { $goal100[0] } else { $null }
limits = $limitReports
}
}
function Write-PerformanceAssessment([object]$Assessment) {
foreach ($limit in @($Assessment.limits)) {
Write-Host ("assessment limit={0}: winner={1} qualified={2} missRate={3:P1} reviewRate={4:P1} activeAvg={5}ms projected100={6}ms" -f `
$limit.limit,
$limit.winnerEngine,
$limit.winnerQualified,
$limit.winnerMissRate,
$limit.winnerReviewRate,
$limit.winnerActiveAverageMsPerParsed,
$limit.winnerActiveProjectedMsFor100)
foreach ($engine in @($limit.engines)) {
Write-Host (" {0}: qualified={1}, missRate={2:P1}, reviewRate={3:P1}, bottleneck={4}, activeAvg={5}ms, ocr={6}ms, capture={7}ms, cardReady={8}ms, scrollReady={9}ms, decision={10}" -f `
$engine.engine,
$engine.qualified,
$engine.missRate,
$engine.reviewRate,
$engine.bottleneck,
$engine.activeAverageMsPerParsed,
$engine.averageOcrMs,
$engine.averageCaptureMs,
$engine.averageCardReadyMs,
$engine.averageScrollReadyMs,
$engine.qualityDecision)
}
}
}
function Invoke-AssessmentSelfTest {
$synthetic = @(
[pscustomobject]@{
engine = "current"
limit = 100
status = "done"
parsed = 100
review = 22
misses = 0
activeAverageMsPerParsed = 700
averageMsPerParsed = 760
activeProjectedMsFor100 = 70000
averageOcrMs = 260
averageCaptureMs = 160
averageCardReadyMs = 190
averageScrollReadyMs = 60
},
[pscustomobject]@{
engine = "ik-traineddata"
limit = 100
status = "done"
parsed = 100
review = 4
misses = 0
activeAverageMsPerParsed = 820
averageMsPerParsed = 870
activeProjectedMsFor100 = 82000
averageOcrMs = 210
averageCaptureMs = 170
averageCardReadyMs = 205
averageScrollReadyMs = 80
},
[pscustomobject]@{
engine = "broken-fast"
limit = 45
status = "done"
parsed = 45
review = 0
misses = 3
activeAverageMsPerParsed = 300
averageMsPerParsed = 330
activeProjectedMsFor100 = 30000
averageOcrMs = 100
averageCaptureMs = 90
averageCardReadyMs = 40
averageScrollReadyMs = 20
},
[pscustomobject]@{
engine = "current"
limit = 45
status = "done"
parsed = 45
review = 1
misses = 0
activeAverageMsPerParsed = 600
averageMsPerParsed = 650
activeProjectedMsFor100 = 60000
averageOcrMs = 230
averageCaptureMs = 130
averageCardReadyMs = 160
averageScrollReadyMs = 50
}
)
$assessment = New-PerformanceAssessment -Summaries $synthetic
$goal100 = $assessment.goal100
$limit45 = @($assessment.limits | Where-Object { $_.limit -eq 45 } | Select-Object -First 1)[0]
if ($goal100.winnerEngine -ne "ik-traineddata") {
throw "Assessment self-test failed: expected ik-traineddata to win limit=100, got '$($goal100.winnerEngine)'."
}
if (-not $goal100.winnerQualified) {
throw "Assessment self-test failed: expected limit=100 winner to be qualified."
}
if ($limit45.winnerEngine -ne "current") {
throw "Assessment self-test failed: expected current to win limit=45, got '$($limit45.winnerEngine)'."
}
if (@($goal100.engines | Where-Object { $_.engine -eq "current" })[0].qualityDecision -ne "not-qualified: review rate above 15%") {
throw "Assessment self-test failed: expected high-review current run to be rejected."
}
if (@($limit45.engines | Where-Object { $_.engine -eq "broken-fast" })[0].qualityDecision -ne "not-qualified: miss rate above 2%") {
throw "Assessment self-test failed: expected broken-fast run to be rejected for miss rate."
}
Write-PerformanceAssessment $assessment
Write-Host "Assessment self-test passed." -ForegroundColor Green
return $assessment
}
function Invoke-OcrBenchmark {
param(
[int]$Limit,
[string]$Engine,
[string]$Profile
)
Write-Host "Warming current OCR workers..."
$warmCurrent = Invoke-DevJson "/scanner/ocr/warmup?engine=current"
Save-Json "benchmark-warmup-current" $warmCurrent | Out-Null
if ($Engine -eq "compare" -or $Engine -eq "ik-traineddata") {
Write-Host "Warming IK-traineddata OCR workers..."
$warmIk = Invoke-DevJson "/scanner/ocr/warmup?engine=ik-traineddata"
Save-Json "benchmark-warmup-ik-traineddata" $warmIk | Out-Null
}
Write-Host "Running OCR benchmark engine=$Engine profile=$Profile limit=$Limit"
$benchmark = Invoke-DevJson "/scanner/benchmark-ocr?limit=$Limit&engine=$Engine&profile=$Profile"
Save-Json "benchmark-ocr-$Engine-$Profile-limit-$Limit" $benchmark | Out-Null
if ($benchmark.summary.mode -eq "compare") {
foreach ($engineSummary in @($benchmark.summary.engines)) {
Write-Host ("benchmark {0}: avg={1}ms ocrAvg={2}ms p50={3}ms p90={4}ms projected100={5}ms skipped={6} pool={7}" -f `
$engineSummary.engine,
$engineSummary.averageMs,
$engineSummary.averageOcrMs,
$engineSummary.p50Ms,
$engineSummary.p90Ms,
$engineSummary.projectedMs.artifacts100,
$engineSummary.skippedOcrCaptures,
$engineSummary.workerPoolSize)
}
} else {
$summary = $benchmark.summary
Write-Host ("benchmark {0}: avg={1}ms ocrAvg={2}ms p50={3}ms p90={4}ms projected100={5}ms skipped={6} pool={7}" -f `
$summary.engine,
$summary.averageMs,
$summary.averageOcrMs,
$summary.p50Ms,
$summary.p90Ms,
$summary.projectedMs.artifacts100,
$summary.skippedOcrCaptures,
$summary.workerPoolSize)
}
return $benchmark
}
function Convert-ReviewSamplesSummary([object]$ReviewPayload) {
$samples = @()
foreach ($record in @($ReviewPayload.samples)) {
$parsed = $record.sample.parsed
$capture = $record.sample.capture
$ocr = @()
foreach ($entry in @($capture.ocr)) {
$ocr += [pscustomobject]@{
id = $entry.id
label = $entry.label
text = $entry.text
confidence = $entry.confidence
}
}
$samples += [pscustomobject]@{
savedAt = $record.savedAt
reason = $record.sample.reason
parsed = [pscustomobject]@{
name = $parsed.name
slot = $parsed.slot
level = $parsed.level
mainStat = $parsed.mainStat
mainValue = $parsed.mainValue
setName = $parsed.setName
equipped = $parsed.equipped
confidence = $parsed.confidence
notes = $parsed.notes
fields = $parsed.fields
}
capture = [pscustomobject]@{
name = $capture.name
width = $capture.width
height = $capture.height
captureTarget = $capture.captureTarget
capturedAt = $capture.capturedAt
inventoryGrid = if ($capture.inventoryGrid) {
[pscustomobject]@{
rows = $capture.inventoryGrid.rows
cols = $capture.inventoryGrid.cols
confidence = $capture.inventoryGrid.confidence
source = $capture.inventoryGrid.source
}
} else { $null }
inventoryCount = $capture.inventoryCount
locked = $capture.locked
ocr = $ocr
}
}
}
return [pscustomobject]@{
ok = $ReviewPayload.ok
total = $ReviewPayload.total
path = $ReviewPayload.path
samples = $samples
}
}
function Wait-ForScannerIdle([int]$Limit, [string]$Engine) {
$startedAt = Get-Date
$pollIndex = 0
$lastStatus = $null
while ($true) {
Start-Sleep -Seconds $PollIntervalSeconds
$pollIndex += 1
$statusPayload = Get-ScannerStatus
$lastStatus = $statusPayload
Save-Json "scan-$Engine-limit-$Limit-poll-$pollIndex" $statusPayload | Out-Null
$running = [bool]$statusPayload.status.running
if (-not $running) {
return $statusPayload
}
$elapsed = ((Get-Date) - $startedAt).TotalSeconds
if ($elapsed -gt $TimeoutSeconds) {
$stop = Invoke-DevJson "/scanner/stop"
Save-Json "scan-$Engine-limit-$Limit-timeout-stop" $stop | Out-Null
throw "Scanner timed out after $TimeoutSeconds seconds for limit=$Limit engine=$Engine. Stop command was sent."
}
}
}
if ($SelfTestAssessment) {
Invoke-AssessmentSelfTest | Out-Null
exit 0
}
$stamp = Get-Date -Format "yyyy-MM-ddTHH-mm-ss"
$RunDir = Join-Path $OutputRoot $stamp
New-Item -ItemType Directory -Force -Path $RunDir | Out-Null
if ($GoalRun) {
$Limits = @(2, 5, 20, 45, 100)
$BenchmarkOcr = $true
}
$ScanEngines = if ($ScanEngine -eq "compare") { @("current", "ik-traineddata") } else { @($ScanEngine) }
$RunSummaries = @()
$transcriptPath = Join-Path $RunDir "transcript.log"
try {
Start-Transcript -Path $transcriptPath -Append | Out-Null
} catch {
Write-Warning "Could not start transcript: $($_.Exception.Message)"
}
try {
Write-Host "Live soak output: $RunDir"
Write-Host "Checking dev control server at $BaseUrl"
$health = Invoke-DevJson "/health"
Save-Json "00-health" $health | Out-Null
Assert-CurrentAppBuild $health
if ($health.appBuild) {
Write-Host "App build: signature=$($health.appBuild.signature), pid=$($health.appBuild.pid), startedAt=$($health.appBuild.startedAt), ocrWorkers=$($health.appBuild.expectedOcrWorkerPoolSize)"
if ($health.appBuild.expectedOcrWorkerPoolSize -lt 4) {
Write-Host "WARNUNG: OCR worker pool is below 4. This is valid for constrained debugging, but not ideal for IK-speed comparison." -ForegroundColor Yellow
}
}
$initialStatus = Get-ScannerStatus
Save-Json "01-status-before" $initialStatus | Out-Null
if ($initialStatus.status.runtimeInfo) {
$runtime = $initialStatus.status.runtimeInfo
Write-Host "Runtime: elevated=$($runtime.isElevated), genshinFound=$($runtime.genshinFound), target=$($runtime.targetProcess)"
} else {
Write-Host "Runtime info is not published yet. Open the app scanner view before running broad scans." -ForegroundColor Yellow
}
if ($BenchmarkOcr) {
Invoke-OcrBenchmark -Limit $BenchmarkLimit -Engine $BenchmarkEngine -Profile $BenchmarkProfile | Out-Null
}
if (-not $SkipSmartCapture) {
Write-Host "Capturing smart preflight snapshot without OCR..."
$capture = Invoke-DevJson "/capture/smart?skipOcr=1"
Save-Json "02-smart-capture-skip-ocr" $capture | Out-Null
}
foreach ($index in $ProbeIndices) {
Write-Host "Running probe click index=$index"
$probe = Invoke-DevJson "/automation/probe-click?index=$index"
Save-Json "probe-index-$index" $probe | Out-Null
if (-not (Test-ProbeSucceeded $probe)) {
Write-Host "Probe index=$index did not fully pass. Review the saved JSON before broader scans." -ForegroundColor Yellow
if (-not $ContinueAfterBlocked) {
throw "Stopping after failed probe index=$index. Re-run with -ContinueAfterBlocked only if you deliberately want to continue."
}
} elseif (-not $probe.ok -and $probe.changed) {
Write-Host "Probe index=$index changed the detail panel even though helper cursor/click readback was not clean; continuing." -ForegroundColor Yellow
}
}
foreach ($engine in $ScanEngines) {
foreach ($limit in $Limits) {
if ($limit -lt 1) { continue }
Write-Host "Starting bounded scanner run limit=$limit engine=$engine"
$start = Invoke-DevJson "/scanner/start?limit=$limit&engine=$engine"
Save-Json "scan-$engine-limit-$limit-start" $start | Out-Null
$finalStatus = Wait-ForScannerIdle -Limit $limit -Engine $engine
Save-Json "scan-$engine-limit-$limit-final" $finalStatus | Out-Null
$summary = Get-CompletedScanSummary $finalStatus $limit
Write-Host "engine=$engine limit=$limit summary: status=$($summary.status), attempted=$($summary.attempted), verified=$($summary.verified), parsed=$($summary.parsed), stored=$($summary.stored), review=$($summary.review), misses=$($summary.misses), pages=$($summary.pages)"
$timing = Convert-ScanTimingSummary $finalStatus $limit $engine
$RunSummaries += $timing
Write-ScanTimingLine $timing
if (($summary.status -eq "blocked" -or $summary.status -eq "stopped") -and -not $ContinueAfterBlocked) {
throw "Stopping after scan status '$($summary.status)' for limit=$limit engine=$engine."
}
}
}
$review = Invoke-DevJson "/review/samples?limit=30"
Save-Json "review-samples-tail-summary" (Convert-ReviewSamplesSummary $review) | Out-Null
if ($SaveFullReviewSamples) {
Save-Json "review-samples-tail-full" $review | Out-Null
}
$afterStatus = Get-ScannerStatus
Save-Json "99-status-after" $afterStatus | Out-Null
Save-Json "scan-run-summary" ([pscustomobject]@{
createdAt = (Get-Date).ToString("o")
goalRun = [bool]$GoalRun
scanEngine = $ScanEngine
scanEngines = $ScanEngines
benchmarkOcr = [bool]$BenchmarkOcr
limits = $Limits
summaries = $RunSummaries
}) | Out-Null
if ($RunSummaries.Count -gt 0) {
$assessment = New-PerformanceAssessment -Summaries $RunSummaries
Save-Json "scan-performance-assessment" $assessment | Out-Null
Write-PerformanceAssessment $assessment
$csvPath = Join-Path $RunDir "scan-run-summary.csv"
$RunSummaries | Export-Csv -LiteralPath $csvPath -NoTypeInformation -Encoding UTF8
Write-Host "Timing summary CSV: $csvPath"
}
Write-Host "Live soak complete. Output: $RunDir" -ForegroundColor Green
} catch {
Write-Host "Live soak failed: $($_.Exception.Message)" -ForegroundColor Red
Write-Host "Output so far: $RunDir" -ForegroundColor Yellow
exit 1
} finally {
try {
Stop-Transcript | Out-Null
} catch {
}
}
src/data/genshinGameData.json
+1429 -1
View File
File diff suppressed because it is too large Load Diff
src/features/scan/components/DiagnosticsView.tsx
+141 -1
View File
@@ -1,5 +1,5 @@
import { useState } from "react";
import { AlertTriangle, Download, Play, Upload, Wrench } from "lucide-react";
import { AlertTriangle, BadgeCheck, Camera, ClipboardList, Download, Gauge, Play, Target, Upload, Wrench } from "lucide-react";
import type { CaptureResult } from "../../../types/global";
import type { ScanViewControllerResult } from "../types";
import { FieldConfidenceList } from "./ScanResultCards";
@@ -14,6 +14,53 @@ interface DiagnosticsViewProps {
canDemoScan?: boolean;
}
const appDiagnosisSections = [
{
title: "Was die App kann",
tone: "ok",
icon: BadgeCheck,
items: [
"Genshin-Fenster erkennen, Smart Capture ausfuehren und fokussierte Artifact-Crops erzeugen.",
"Artifact-Felder deterministisch gegen das lokale Genshin-Datenpaket parsen.",
"Auto-Scan read-only aus der sichtbaren Inventory-Seite starten, inklusive Grid, Verifikation, Dedupe und Store.",
"Review-Samples, lokale Text-Lernregeln, GOOD Import/Export und Lock-Status im Store nutzen.",
],
},
{
title: "Was noch fehlt",
tone: "warn",
icon: ClipboardList,
items: [
"Paimon-Menue-Einstieg ist gebaut, aber live noch nicht mit 2/20/45 Limits validiert.",
"Native/IK-Tesseract ist nur als Benchmark-Pfad vorbereitet, noch nicht Standard.",
"Positive locked=true Probe an einem sicher gesperrten Artifact fehlt.",
"Empfehlungen bleiben Nebenfunktion, bis Scanner-Vertrauen und Review-Rate stabil genug sind.",
],
},
{
title: "Wo es Probleme macht",
tone: "risk",
icon: AlertTriangle,
items: [
"OCR ist weiterhin der Haupt-Risikofaktor; einige Felder landen noch in Fallback, Ableitung oder Review.",
"Bild-Preprocessing kann nur an echten Captures bewertet werden, nicht allein mit Text-Eval.",
"Auto-Scan braucht bei erhoehtem Genshin auch eine erhoehte App-Laufzeit.",
"Groessere Runs brauchen weiter Beobachtung auf Scroll-Uebergaenge, Wiederholseiten und Review-Quote.",
],
},
{
title: "Naechste Verbesserungen",
tone: "next",
icon: Target,
items: [
"Review-Corpus aus echten Samples vergroessern und mit `npm run eval` messbar halten.",
"OCR-Benchmark gegen identische Crops fahren und erst danach Engine-Standard wechseln.",
"Paimon-Menue-Pfad live pruefen und bei Blockade sichtbar auf visible-inventory zurueckfallen.",
"Diagnose weiter als Operator-Cockpit halten: Live-Status, Evidenz und naechster sicherer Schritt.",
],
},
];
// All developer / diagnostic surfaces live here, separated from the Scan
// workspace: runtime + rights, grid detection, learning + data-package status,
// fingerprint, auto-scan counters, the automation log, and the raw crop/OCR/
@@ -39,6 +86,7 @@ export function DiagnosticsView({ controller, latestCapture, captureStatus, onDe
playerProgress,
reviewStatus,
automationLogLines,
diagnosticEvents,
canSaveReviewSample,
handleSaveReviewSample,
} = useScanDiagnosticsModalModel({
@@ -106,6 +154,37 @@ export function DiagnosticsView({ controller, latestCapture, captureStatus, onDe
</div>
</div>
<div className="diagnose-card app-diagnosis-card">
<div className="diagnose-card-heading">
<div>
<p className="eyebrow">Aktueller App-Stand</p>
<h3>Scanner zuerst, Empfehlungen danach</h3>
</div>
<span className="diagnosis-source">
<Gauge size={14} />
Quelle: Docs + Live-Status
</span>
</div>
<div className="app-diagnosis-grid">
{appDiagnosisSections.map((section) => {
const Icon = section.icon;
return (
<article className={`app-diagnosis-section ${section.tone}`} key={section.title}>
<div className="app-diagnosis-title">
<Icon size={16} />
<strong>{section.title}</strong>
</div>
<ul>
{section.items.map((item) => (
<li key={item}>{item}</li>
))}
</ul>
</article>
);
})}
</div>
</div>
<div className="diagnose-grid">
<div className="diagnose-card">
<p className="eyebrow">Status</p>
@@ -196,6 +275,67 @@ export function DiagnosticsView({ controller, latestCapture, captureStatus, onDe
</div>
</div>
<div className="diagnose-card">
<div className="diagnose-card-heading">
<div>
<p className="eyebrow">Evidence timeline</p>
<h3>Scan-Flugschreiber</h3>
</div>
<span className="diagnosis-source">
<Camera size={14} />
letzte {diagnosticEvents.length}
</span>
</div>
{diagnosticEvents.length > 0 ? (
<div className="scan-evidence-timeline">
{diagnosticEvents.slice().reverse().map((event) => (
<article className={`scan-evidence-event ${event.severity}`} key={event.id}>
<div className="scan-evidence-header">
<span>{new Date(event.at).toLocaleTimeString()}</span>
<strong>{event.phase}</strong>
<em>{event.severity}</em>
</div>
<p>{event.message}</p>
{event.details && (
<div className="scan-evidence-details">
{Object.entries(event.details).map(([key, value]) => (
<span key={key}>{key}: {String(value ?? "-")}</span>
))}
</div>
)}
{event.capture && (
<div className="scan-evidence-capture">
<div>
<strong>{event.capture.name}</strong>
<span>{event.capture.width}x{event.capture.height} · {event.capture.target ?? "capture"} · fp {event.capture.fingerprint}</span>
{event.capture.grid && (
<span>grid {event.capture.grid.cols}x{event.capture.grid.rows} · {event.capture.grid.targets} targets · {event.capture.grid.confidence}% · {event.capture.grid.source}</span>
)}
{event.capture.count && (
<span>count {event.capture.count.current}/{event.capture.count.total || "?"} · {event.capture.count.confidence}% · {event.capture.count.text || "-"}</span>
)}
{event.capture.artifactDetail && (
<span>detail {event.capture.artifactDetail.present ? "yes" : "no"} · {event.capture.artifactDetail.confidence}% · orange {event.capture.artifactDetail.orangeHits} · text {event.capture.artifactDetail.textHits}</span>
)}
{event.capture.paimonMenu && (
<span>paimon {event.capture.paimonMenu.present ? "yes" : "no"} · {event.capture.paimonMenu.confidence}%</span>
)}
{event.capture.layoutWarning && <span className="evidence-warning">{event.capture.layoutWarning}</span>}
</div>
<div className="scan-evidence-images">
{event.capture.screenshots?.inventory && <img src={event.capture.screenshots.inventory} alt={`${event.phase} inventory`} />}
{event.capture.screenshots?.detail && <img src={event.capture.screenshots.detail} alt={`${event.phase} detail`} />}
</div>
</div>
)}
</article>
))}
</div>
) : (
<p className="result-empty">Noch keine Evidence-Events. Starte einen Capture oder Auto-Scan, dann erscheinen hier Schritte mit Screenshots.</p>
)}
</div>
<div className="diagnose-card">
<div className="diagnose-card-heading">
<p className="eyebrow">Crops, OCR &amp; Confidence</p>
src/features/scan/components/ScanMainSection.tsx
+5 -15
View File
@@ -7,14 +7,9 @@ export function ScanMainSection({
latestCapture,
captureStatus,
parsedArtifact,
sourceLabel,
gridLabel,
inventoryLabel,
activeTargetCount,
storedTotal,
reviewSampleTotal,
learningRulesLoaded,
learningRuleCount,
setDetailsOpen,
autoScanRunning,
canOpenReviewQueue,
@@ -27,29 +22,25 @@ export function ScanMainSection({
captureImageSrc,
captureImageAlt,
hasCapture,
captureModeText,
resultHeading,
noArtifactText,
noCaptureMessage,
targetLabel,
dbLabel,
reviewLabel,
rulesLabel,
} = useScanMainSectionModel({
latestCapture,
parsedArtifact,
activeTargetCount,
storedTotal,
reviewSampleTotal,
learningRulesLoaded,
learningRuleCount,
setDetailsOpen,
openReviewQueue,
});
return (
<div className="scanner-main-grid">
<div className="capture-stage-shell">
<div className={`capture-stage-shell ${hasCapture ? "has-capture" : "is-empty"}`}>
<div className="capture-stage">
{hasCapture ? (
<img src={captureImageSrc} alt={captureImageAlt} />
@@ -61,10 +52,6 @@ export function ScanMainSection({
</div>
)}
</div>
<div className="capture-stage-meta">
<div><span>Quelle</span><strong>{sourceLabel}</strong></div>
<div><span>Inventar</span><strong>{inventoryLabel}</strong></div>
</div>
</div>
<aside className="scanner-result-panel">
@@ -81,9 +68,12 @@ export function ScanMainSection({
<span>{reviewLabel}</span>
</div>
<div className="scanner-result-actions">
<button className="ghost-button" onClick={handleOpenDetails} disabled={!canOpenDetails}>
Details
</button>
<button className="ghost-button" onClick={handleOpenReviewQueue} disabled={autoScanRunning || !canOpenReviewQueue}>
<AlertTriangle size={15} />
Review Queue
Review
</button>
</div>
<p className="scanner-result-caption">{captureStatus}</p>
src/features/scan/components/ScanTopControlsSection.tsx
+10 -12
View File
@@ -32,7 +32,7 @@ export function ScanTopControlsSection({
openDiagnostics,
captureSingleArtifact,
stopScan,
runVisibleGridScan,
runGuidedAutoScan,
runAutoReviewScan,
bridgeStatusText,
bridgePillClass,
@@ -46,7 +46,6 @@ export function ScanTopControlsSection({
refreshCaptureSourcesTitle,
diagnosticsButtonTitle,
scanSetupButtonTitle,
showPlayerProgress,
progressWidth,
progressStats,
} = useScanTopControlsModel({
@@ -64,8 +63,7 @@ export function ScanTopControlsSection({
<div className="scanner-header">
<div>
<p className="eyebrow">Scanner</p>
<h2>Artifact capture workspace</h2>
<p className="scanner-subcopy">Quelle waehlen, Auto-Scan starten, Ergebnis rechts pruefen. Dev-Details im Diagnose-Tab.</p>
<h2>Artifact Scan</h2>
</div>
<div className="scanner-header-pills">
<span className={`runtime-pill ${bridgePillClass}`}>{bridgeStatusText}</span>
@@ -119,10 +117,11 @@ export function ScanTopControlsSection({
</button>
</div>
<div className="player-scan-lower">
<div className="player-scan-actions">
<button
className="primary-button scan-cta"
onClick={runVisibleGridScan}
onClick={runGuidedAutoScan}
disabled={!canStartAutoScan}
title={autoScanButtonTitle}
>
@@ -145,7 +144,7 @@ export function ScanTopControlsSection({
title={captureSingleButtonTitle}
>
<Camera size={15} />
Einzelnes Artifact lesen
Einzelnes Artifact
</button>
{autoScanRunning && (
<button className="stop-button" onClick={stopScan}>
@@ -154,11 +153,6 @@ export function ScanTopControlsSection({
)}
</div>
<p className="player-status">
{playerStatusText}
</p>
{showPlayerProgress && (
<div className="player-progress">
<div className="player-progress-bar">
<div style={{ width: `${progressWidth}%` }} />
@@ -171,7 +165,11 @@ export function ScanTopControlsSection({
))}
</div>
</div>
)}
</div>
<p className="player-status">
{playerStatusText}
</p>
</div>
</>
);
src/features/scan/components/hooks/useScanMainSectionModel.ts
-10
View File
@@ -8,14 +8,12 @@ export interface ScanMainSectionModel {
captureImageSrc: string;
captureImageAlt: string;
hasCapture: boolean;
captureModeText: string;
resultHeading: string;
noArtifactText: string;
noCaptureMessage: string;
targetLabel: string;
dbLabel: string;
reviewLabel: string;
rulesLabel: string;
}
type UseScanMainSectionModelProps = Pick<
@@ -25,8 +23,6 @@ type UseScanMainSectionModelProps = Pick<
| "activeTargetCount"
| "storedTotal"
| "reviewSampleTotal"
| "learningRulesLoaded"
| "learningRuleCount"
| "setDetailsOpen"
| "openReviewQueue"
>;
@@ -37,8 +33,6 @@ export function useScanMainSectionModel({
activeTargetCount,
storedTotal,
reviewSampleTotal,
learningRulesLoaded,
learningRuleCount,
setDetailsOpen,
openReviewQueue,
}: UseScanMainSectionModelProps): ScanMainSectionModel {
@@ -53,14 +47,12 @@ export function useScanMainSectionModel({
const captureImageAlt = latestCapture
? `Latest capture from ${latestCapture.name}`
: "Latest capture is not available yet";
const captureModeText = latestCapture ? "Erkannt" : "Warte";
const resultHeading = parsedArtifact ? parsedArtifact.name : "Noch kein Artifact";
const noArtifactText = "Oeffne ein Artifact in Genshin und nutze \"Einzelnes Artifact lesen\" - oder starte direkt den Auto-Scan.";
const noCaptureMessage = noArtifactText;
const targetLabel = `Ziel ${activeTargetCount}`;
const dbLabel = `DB ${storedTotal ?? "-"}`;
const reviewLabel = `Review ${reviewSampleTotal}`;
const rulesLabel = `Regeln ${learningRulesLoaded ? learningRuleCount : "..."}`;
return {
canOpenDetails,
@@ -69,13 +61,11 @@ export function useScanMainSectionModel({
captureImageSrc,
captureImageAlt,
hasCapture: Boolean(latestCapture),
captureModeText,
resultHeading,
noArtifactText,
noCaptureMessage,
targetLabel,
dbLabel,
reviewLabel,
rulesLabel,
};
}
src/features/scan/components/hooks/useScanSummaryFooterModel.ts
+11 -3
View File
@@ -1,4 +1,12 @@
import type { ScanSummaryFooterProps } from "../types";
import type { ScanSummaryFooterProps } from "../types";
function formatDuration(ms: number) {
if (ms <= 0) return "0s";
const seconds = Math.round(ms / 1000);
const minutes = Math.floor(seconds / 60);
const rest = seconds % 60;
return minutes > 0 ? `${minutes}m ${rest}s` : `${rest}s`;
}
export interface ScanSummaryFooterModel {
summaryCopy: string;
@@ -12,10 +20,10 @@ export function useScanSummaryFooterModel({
}: ScanSummaryFooterProps): ScanSummaryFooterModel {
const summaryCopy = scanSummary.status === "blocked" && scanSummary.clicked === 0 && scanSummary.mode !== "Manueller Scan"
? "Es wurden keine Klicks ausgefuehrt. Grund siehe oben."
: `${scanSummary.attempted} Positionen bearbeitet, ${scanSummary.verified} Ansichten verifiziert, ${scanSummary.parsed} Artifact${scanSummary.parsed === 1 ? "" : "s"} gelesen. Deine Sammlung: ${storedTotal ?? "?"} Artifacts.`;
: `${scanSummary.attempted} Positionen bearbeitet, ${scanSummary.verified} Ansichten verifiziert, ${scanSummary.parsed} Artifact${scanSummary.parsed === 1 ? "" : "s"} gelesen in ${formatDuration(scanSummary.elapsedMs)} (${scanSummary.averageMsPerParsed || 0} ms/Artifact). Deine Sammlung: ${storedTotal ?? "?"} Artifacts.`;
const devCopy = devMode
? `clicked ${scanSummary.clicked} · attempted ${scanSummary.attempted} · verified ${scanSummary.verified} · parsed ${scanSummary.parsed} · misses ${scanSummary.misses} · pages ${scanSummary.pages}`
? `clicked ${scanSummary.clicked} | attempted ${scanSummary.attempted} | verified ${scanSummary.verified} | parsed ${scanSummary.parsed} | misses ${scanSummary.misses} | pages ${scanSummary.pages} | active ${formatDuration(scanSummary.activeScanMs)} | flush ${scanSummary.writeFlushMs}ms | capture ${scanSummary.averageCaptureMs}ms | ocr ${scanSummary.averageOcrMs}ms | ${scanSummary.artifactsPerMinute}/min | active ${scanSummary.activeArtifactsPerMinute}/min | 100 projected ${formatDuration(scanSummary.projectedMsFor100)}`
: null;
return {
src/features/scan/components/hooks/useScanTopControlsModel.ts
+18 -6
View File
@@ -15,7 +15,7 @@ export interface ScanTopControlsModel {
openDiagnostics: () => void;
captureSingleArtifact: () => void;
stopScan: () => void;
runVisibleGridScan: () => void;
runGuidedAutoScan: () => void;
runAutoReviewScan: () => void;
bridgeStatusText: string;
bridgePillClass: string;
@@ -47,7 +47,7 @@ export function useScanTopControlsModel({
setSettingsOpen,
setDiagnosticsOpen,
requestScanStop,
runVisibleGridScan,
runGuidedAutoScan,
runAutoReviewScan,
autoScanRunning,
canCaptureSource,
@@ -77,17 +77,20 @@ export function useScanTopControlsModel({
const openDiagnostics = useCallback(() => setDiagnosticsOpen(true), [setDiagnosticsOpen]);
const captureSingleArtifact = useCallback(() => captureSelectedSource(0, true), [captureSelectedSource]);
const stopScan = useCallback(() => requestScanStop("Stop-Button gedrueckt."), [requestScanStop]);
const startGuidedAutoScan = useCallback(() => {
void runGuidedAutoScan();
}, [runGuidedAutoScan]);
const bridgeStatusText = bridgeReady ? "Bridge verbunden" : "Bridge fehlt";
const bridgePillClass = bridgeReady ? "elevated" : "standard";
const runtimeStatusText = runtimeInfo?.isElevated ? "Admin bereit" : "Standard";
const runtimePillClass = runtimeInfo?.isElevated ? "elevated" : "standard";
const playerStatusText = reviewStatus
|| (runtimeInfo?.isElevated
? "App laeuft als Administrator. Oeffne in Genshin das Artifact-Inventar und starte den Auto-Scan."
? "App laeuft als Administrator. Auto-Scan prueft den Screen ohne OCR und startet erst, wenn eine Artifact-Detailkarte offen ist."
: "App laeuft im Standard-Modus - Auto-Scan braucht Administrator-Rechte. Bitte die App schliessen und als Administrator neu starten.");
const autoScanButtonTitle = requiresAdminForAutoScan
? "App laeuft nicht als Administrator. Bitte die App als Administrator neu starten."
: "Klickt und scrollt automatisch durch das sichtbare Artifact-Inventar.";
: "Prueft zuerst ohne OCR den Screen, oeffnet bei Bedarf per Inventory-Kamera-Sequenz das Artifact-Inventar und scannt erst mit sichtbarer Detailkarte.";
const autoScanButtonLabel = autoScanRunning ? "Scan laeuft..." : "Auto-Scan starten";
const manualScanButtonTitle = "Du klickst die Artifacts in Genshin selbst an; die App liest nur mit. Kein Auto-Klick, kein Scrollen.";
const captureSingleButtonTitle = "Liest das gerade in Genshin geoeffnete Artifact einmalig.";
@@ -104,11 +107,20 @@ export function useScanTopControlsModel({
{ label: "Klicks", value: autoScanStats.clicked },
{ label: "Positionen", value: autoScanStats.attempted },
{ label: "Verifiziert", value: autoScanStats.verified },
{ label: "ms/Artifact", value: autoScanStats.averageMsPerParsed || "-" },
{ label: "Gespeichert", value: autoScanStats.stored },
{ label: "Review", value: autoScanStats.review },
{ label: "Sammlung", value: storedTotal ?? "-", extraClass: "collection" },
],
[autoScanStats.clicked, autoScanStats.attempted, autoScanStats.verified, autoScanStats.stored, autoScanStats.review, storedTotal],
[
autoScanStats.clicked,
autoScanStats.attempted,
autoScanStats.verified,
autoScanStats.averageMsPerParsed,
autoScanStats.stored,
autoScanStats.review,
storedTotal,
],
);
return {
@@ -124,7 +136,7 @@ export function useScanTopControlsModel({
openDiagnostics,
captureSingleArtifact,
stopScan,
runVisibleGridScan,
runGuidedAutoScan: startGuidedAutoScan,
runAutoReviewScan,
bridgeStatusText,
bridgePillClass,
src/features/scan/components/modals/ScanSettingsModal.tsx
+52 -28
View File
@@ -1,5 +1,6 @@
import type { ScanSettingsModalProps } from "./types";
import { useScanSettingsModalModel } from "./hooks/useScanSettingsModalModel";
import type { StepperControlModel } from "./hooks/useScanSettingsModalModel";
export function ScanSettingsModal({
open,
@@ -16,9 +17,9 @@ export function ScanSettingsModal({
const {
closeSettings,
handleScanLimitChange,
handleSkipRowsChange,
stopPropagation,
scanLimitControl,
skipRowsControl,
inventoryCountText,
inventoryClassName,
scanLimitClassName,
@@ -50,33 +51,17 @@ export function ScanSettingsModal({
<button className="ghost-button" onClick={closeSettings}>Schliessen</button>
</div>
<div className="modal-body">
<div className="scan-config-strip">
<label>
<span>Anzahl Artifacts</span>
<input
type="number"
min={1}
max={1800}
value={scanLimit}
onChange={handleScanLimitChange}
/>
</label>
<label>
<span>Zeilen ueberspringen</span>
<input
type="number"
min={0}
max={8}
value={skipRows}
onChange={handleSkipRowsChange}
/>
</label>
<p>
Die App uebernimmt die erkannte Inventar-Anzahl nur als Startwert und Deckel nach oben. Dein manuell gesetztes Ziel bleibt erhalten.
Mit "Zeilen ueberspringen" kannst du den Startverzug korrigieren, falls du nicht am Anfang der Liste beginnst.
</p>
<div className="scan-settings-layout">
<div className="scan-settings-controls">
<StepperControl control={scanLimitControl} />
<StepperControl control={skipRowsControl} />
</div>
<div className="scanner-preflight diagnostics-preflight">
<div className="scan-settings-note">
<strong>Manuelle Werte bleiben erhalten.</strong>
<span>Die erkannte Inventar-Anzahl ist nur ein Vorschlag und oberer Deckel. Startzeilen brauchst du nur, wenn du nicht oben beginnst.</span>
</div>
</div>
<div className="scanner-preflight settings-preflight">
<div className={inventoryClassName}>
<span>Inventarzaehler</span>
<strong>{inventoryCountText}</strong>
@@ -97,3 +82,42 @@ export function ScanSettingsModal({
</div>
);
}
function StepperControl({ control }: { control: StepperControlModel }) {
return (
<section className="settings-stepper" aria-label={control.label}>
<div className="settings-stepper-head">
<div>
<span>{control.label}</span>
<small>{control.helper}</small>
</div>
</div>
<div className="settings-stepper-row">
<button type="button" className="stepper-button" onClick={control.decrement} aria-label={`${control.label} verringern`}>
−
</button>
<input
inputMode="numeric"
pattern="[0-9]*"
min={control.min}
max={control.max}
value={control.value}
onChange={control.onChange}
onBlur={control.onBlur}
onKeyDown={control.onKeyDown}
aria-label={control.label}
/>
<button type="button" className="stepper-button" onClick={control.increment} aria-label={`${control.label} erhoehen`}>
+
</button>
</div>
<div className="settings-presets" aria-label={`${control.label} Presets`}>
{control.presets.map((value) => (
<button type="button" key={value} onClick={() => control.applyPreset(value)}>
{value}
</button>
))}
</div>
</section>
);
}
src/features/scan/components/modals/hooks/useScanDetailsModalModel.ts
+13 -10
View File
@@ -33,21 +33,24 @@ export function useScanDetailsModalModel({
const crops = latestCapture?.crops ?? [];
const ocr = latestCapture?.ocr ?? [];
const cropRows = crops
.filter((crop) => Boolean(crop.dataUrl))
.map((crop) => ({
id: crop.id,
dataUrl: crop.dataUrl ?? "",
label: crop.label,
x: crop.rect.x,
y: crop.rect.y,
width: crop.rect.width,
height: crop.rect.height,
}));
return {
closeDetails,
stopPropagation,
parsedNotes,
showParsedNotes: parsedNotes.length > 0,
cropRows: crops.map((crop) => ({
id: crop.id,
dataUrl: crop.dataUrl,
label: crop.label,
x: crop.rect.x,
y: crop.rect.y,
width: crop.rect.width,
height: crop.rect.height,
})),
cropRows,
ocrRows: ocr.map((entry) => ({
id: entry.id,
label: entry.label,
@@ -55,7 +58,7 @@ export function useScanDetailsModalModel({
text: entry.text || "No text detected",
})),
debugText: `Debug: crops ${crops.length} / ocr ${ocr.length}`,
showCrops: crops.length > 0,
showCrops: cropRows.length > 0,
showOcr: ocr.length > 0,
};
}
src/features/scan/components/modals/hooks/useScanDiagnosticsModalModel.ts
+47 -1
View File
@@ -1,8 +1,10 @@
import { detailFingerprint } from "../../../../../lib/autoScanLoop";
import { dataGeneratedAt, sourceVersion } from "../../../../../lib/genshinData";
import { dataPackageStatus } from "../../../../../lib/dataPackageStatus";
import { validateLookupPackage } from "../../../../../lib/genshinLookup";
import { useCallback, useMemo, type MouseEvent } from "react";
import type { ScanDiagnosticsModalProps } from "../types";
import type { ScanDiagnosticEvent } from "../../../../../lib/scanDiagnosticsLog";
export interface ScanDiagnosticsModelProgress {
width: number;
@@ -37,6 +39,7 @@ export interface ScanDiagnosticsModalModel {
showDevRows: boolean;
reviewStatus: string;
automationLogLines: string[];
diagnosticEvents: ScanDiagnosticEvent[];
canSaveReviewSample: boolean;
}
@@ -98,7 +101,11 @@ export function useScanDiagnosticsModalModel({
const learningRulesText = controller.learningRulesLoaded ? `${controller.learningRuleCount} local rules` : "loading";
const dataStaleness = dataPackageStatus(dataGeneratedAt, sourceVersion);
const learningRulesSubtext = `Review samples feed deterministic OCR fixes before each new parse. Data package: ${sourceVersion}${dataStaleness.warning ? ` - ${dataStaleness.warning}` : ""}`;
const lookupStatus = validateLookupPackage();
const lookupText = lookupStatus.valid
? `Lookup OK: ${lookupStatus.summary.artifactSets} sets / ${lookupStatus.summary.artifactPieces} pieces`
: `Lookup invalid: ${lookupStatus.errors[0] ?? "unknown error"}`;
const learningRulesSubtext = `Review samples feed deterministic OCR fixes before each new parse. ${lookupText}. Data package: ${sourceVersion}${dataStaleness.warning ? ` - ${dataStaleness.warning}` : ""}`;
const playerProgress = useMemo(() => {
const width = Math.min(
@@ -128,6 +135,25 @@ export function useScanDiagnosticsModalModel({
{ label: "duplicates", value: controller.autoScanStats.duplicates },
{ label: "misses", value: controller.autoScanStats.misses },
{ label: "pages", value: controller.autoScanStats.pages },
{ label: "elapsedMs", value: controller.autoScanStats.elapsedMs },
{ label: "activeScanMs", value: controller.autoScanStats.activeScanMs },
{ label: "writeFlushMs", value: controller.autoScanStats.writeFlushMs },
{ label: "avgMs", value: controller.autoScanStats.averageMsPerParsed },
{ label: "activeAvgMs", value: controller.autoScanStats.activeAverageMsPerParsed },
{ label: "avgCaptureMs", value: controller.autoScanStats.averageCaptureMs },
{ label: "captureP50Ms", value: controller.autoScanStats.captureP50Ms },
{ label: "captureP90Ms", value: controller.autoScanStats.captureP90Ms },
{ label: "avgOcrMs", value: controller.autoScanStats.averageOcrMs },
{ label: "ocrP50Ms", value: controller.autoScanStats.ocrP50Ms },
{ label: "ocrP90Ms", value: controller.autoScanStats.ocrP90Ms },
{ label: "cardReadyAvgMs", value: controller.autoScanStats.averageCardReadyMs },
{ label: "cardReadyCount", value: controller.autoScanStats.cardReadyCount },
{ label: "scrollReadyAvgMs", value: controller.autoScanStats.averageScrollReadyMs },
{ label: "scrollReadyCount", value: controller.autoScanStats.scrollReadyCount },
{ label: "perMin x10", value: Math.round(controller.autoScanStats.artifactsPerMinute * 10) },
{ label: "activePerMin x10", value: Math.round(controller.autoScanStats.activeArtifactsPerMinute * 10) },
{ label: "projected100Ms", value: controller.autoScanStats.projectedMsFor100 },
{ label: "activeProjected100Ms", value: controller.autoScanStats.activeProjectedMsFor100 },
],
[
controller.autoScanStats.clicked,
@@ -139,6 +165,25 @@ export function useScanDiagnosticsModalModel({
controller.autoScanStats.duplicates,
controller.autoScanStats.misses,
controller.autoScanStats.pages,
controller.autoScanStats.elapsedMs,
controller.autoScanStats.activeScanMs,
controller.autoScanStats.writeFlushMs,
controller.autoScanStats.averageMsPerParsed,
controller.autoScanStats.activeAverageMsPerParsed,
controller.autoScanStats.averageCaptureMs,
controller.autoScanStats.captureP50Ms,
controller.autoScanStats.captureP90Ms,
controller.autoScanStats.averageOcrMs,
controller.autoScanStats.ocrP50Ms,
controller.autoScanStats.ocrP90Ms,
controller.autoScanStats.averageCardReadyMs,
controller.autoScanStats.cardReadyCount,
controller.autoScanStats.averageScrollReadyMs,
controller.autoScanStats.scrollReadyCount,
controller.autoScanStats.artifactsPerMinute,
controller.autoScanStats.activeArtifactsPerMinute,
controller.autoScanStats.projectedMsFor100,
controller.autoScanStats.activeProjectedMsFor100,
],
);
@@ -186,6 +231,7 @@ export function useScanDiagnosticsModalModel({
showDevRows: controller.devMode,
reviewStatus: controller.reviewStatus,
automationLogLines: controller.automationLog,
diagnosticEvents: controller.diagnosticEvents,
canSaveReviewSample: canSaveReviewSample && Boolean(controller.parsedArtifact),
};
}
src/features/scan/components/modals/hooks/useScanSettingsModalModel.ts
+109 -14
View File
@@ -1,12 +1,27 @@
import { useCallback, type ChangeEvent, type MouseEvent } from "react";
import { useCallback, useEffect, useState, type ChangeEvent, type FocusEvent, type KeyboardEvent, type MouseEvent } from "react";
import type { CaptureResult, RuntimeInfo } from "../../../../../types/global";
import { clampScanLimit, clampSkipRows } from "../../../../../lib/scannerSession";
export interface StepperControlModel {
label: string;
value: string;
helper: string;
min: number;
max: number;
presets: number[];
onChange: (event: ChangeEvent<HTMLInputElement>) => void;
onBlur: (event: FocusEvent<HTMLInputElement>) => void;
onKeyDown: (event: KeyboardEvent<HTMLInputElement>) => void;
decrement: () => void;
increment: () => void;
applyPreset: (value: number) => void;
}
export interface ScanSettingsModalModel {
closeSettings: () => void;
handleScanLimitChange: (event: ChangeEvent<HTMLInputElement>) => void;
handleSkipRowsChange: (event: ChangeEvent<HTMLInputElement>) => void;
stopPropagation: (event: MouseEvent<HTMLDivElement>) => void;
scanLimitControl: StepperControlModel;
skipRowsControl: StepperControlModel;
inventoryCountText: string;
inventoryClassName: string;
scanLimitClassName: string;
@@ -36,17 +51,71 @@ export function useScanSettingsModalModel({
setScanLimitTouched: (touched: boolean) => void;
setSkipRows: (rows: number) => void;
}): ScanSettingsModalModel {
const [scanLimitText, setScanLimitText] = useState(String(scanLimit));
const [skipRowsText, setSkipRowsText] = useState(String(skipRows));
useEffect(() => {
setScanLimitText(String(scanLimit));
}, [scanLimit]);
useEffect(() => {
setSkipRowsText(String(skipRows));
}, [skipRows]);
const closeSettings = useCallback(() => setSettingsOpen(false), [setSettingsOpen]);
const handleScanLimitChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
setScanLimitTouched(true);
setScanLimit(clampScanLimit(Number(event.target.value)));
}, [setScanLimit, setScanLimitTouched]);
const handleSkipRowsChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
setSkipRows(clampSkipRows(Number(event.target.value)));
}, [setSkipRows]);
const stopPropagation = useCallback((event: MouseEvent<HTMLDivElement>) => {
event.stopPropagation();
}, []);
const applyScanLimit = useCallback((value: number) => {
const next = clampScanLimit(value);
setScanLimitTouched(true);
setScanLimit(next);
setScanLimitText(String(next));
}, [setScanLimit, setScanLimitTouched]);
const applySkipRows = useCallback((value: number) => {
const next = clampSkipRows(value);
setSkipRows(next);
setSkipRowsText(String(next));
}, [setSkipRows]);
const handleScanLimitChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
setScanLimitText(event.target.value.replace(/\D/g, "").slice(0, 4));
}, []);
const handleSkipRowsChange = useCallback((event: ChangeEvent<HTMLInputElement>) => {
setSkipRowsText(event.target.value.replace(/\D/g, "").slice(0, 2));
}, []);
const commitScanLimit = useCallback((rawValue: string) => {
applyScanLimit(rawValue.trim() === "" ? scanLimit : Number(rawValue));
}, [applyScanLimit, scanLimit]);
const commitSkipRows = useCallback((rawValue: string) => {
applySkipRows(rawValue.trim() === "" ? skipRows : Number(rawValue));
}, [applySkipRows, skipRows]);
const handleScanLimitBlur = useCallback((event: FocusEvent<HTMLInputElement>) => {
commitScanLimit(event.target.value);
}, [commitScanLimit]);
const handleSkipRowsBlur = useCallback((event: FocusEvent<HTMLInputElement>) => {
commitSkipRows(event.target.value);
}, [commitSkipRows]);
const handleScanLimitKeyDown = useCallback((event: KeyboardEvent<HTMLInputElement>) => {
if (event.key !== "Enter") return;
commitScanLimit(event.currentTarget.value);
event.currentTarget.blur();
}, [commitScanLimit]);
const handleSkipRowsKeyDown = useCallback((event: KeyboardEvent<HTMLInputElement>) => {
if (event.key !== "Enter") return;
commitSkipRows(event.currentTarget.value);
event.currentTarget.blur();
}, [commitSkipRows]);
const detectedInventoryCount = latestCapture?.inventoryCount?.current;
const detectedInventoryTotal = latestCapture?.inventoryCount?.total;
const inventoryClassName = detectedInventoryCount ? "ok" : "neutral";
@@ -56,14 +125,40 @@ export function useScanSettingsModalModel({
const scanLimitClassName = scanLimit !== detectedInventoryCount ? "ok" : "standard";
const skipRowsClassName = skipRows > 0 ? "ok" : "standard";
const focusModeText = runtimeInfo?.isElevated ? "Admin" : "Standard";
const activeTargetText = `Aktive Zielvorgabe ${activeTargetCount} und Fokusmodus ${focusModeText}.`;
const scanSummaryText = "Der Auto-Scan zaehlt \"Positionen\" und \"verified\", bevor die Datenbank in den Save-Pfad laeuft. So wird \"scanned\" nicht mit \"erfolgreich gespeichert\" verwechselt.";
const activeTargetText = `${activeTargetCount} Ziele · ${focusModeText}`;
const scanSummaryText = "Auto-Scan zaehlt Positionen, verifizierte Ansichten und gespeicherte Artifacts getrennt.";
return {
closeSettings,
handleScanLimitChange,
handleSkipRowsChange,
stopPropagation,
scanLimitControl: {
label: "Scan-Ziel",
value: scanLimitText,
helper: "Wie viele sichtbare Positionen verarbeitet werden.",
min: 1,
max: 1800,
presets: [16, 20, 50, 100],
onChange: handleScanLimitChange,
onBlur: handleScanLimitBlur,
onKeyDown: handleScanLimitKeyDown,
decrement: () => applyScanLimit(scanLimit - 1),
increment: () => applyScanLimit(scanLimit + 1),
applyPreset: applyScanLimit,
},
skipRowsControl: {
label: "Startzeilen ueberspringen",
value: skipRowsText,
helper: "Nur nutzen, wenn du mitten in der Liste beginnst.",
min: 0,
max: 8,
presets: [0, 1, 2, 3],
onChange: handleSkipRowsChange,
onBlur: handleSkipRowsBlur,
onKeyDown: handleSkipRowsKeyDown,
decrement: () => applySkipRows(skipRows - 1),
increment: () => applySkipRows(skipRows + 1),
applyPreset: applySkipRows,
},
inventoryCountText,
inventoryClassName,
scanLimitClassName,
src/features/scan/hooks/scanViewControllerService.ts
+8 -1
View File
@@ -3,6 +3,7 @@ import type { ScannerLearningRules } from "../../../lib/scannerLearning";
import type { ParsedArtifactCandidate } from "../../../lib/artifactOcrParser";
import type { AutoScanStats, ScanSummary } from "../../../lib/scannerSession";
import type { ScanActionContext } from "./scanViewScanActions";
import type { createScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import type {
ReviewStateContext,
} from "./scanViewReviewHelpers";
@@ -42,6 +43,7 @@ export interface ScanActionContextInput {
setReviewStatus: Dispatch<SetStateAction<string>>;
appendAutomationLog: (line: string) => void;
appendClickDiagnostics: (result: ClickResult, prefix?: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
parseArtifact: (capture: CaptureResult | null) => ParsedArtifactCandidate | null;
persistParsedArtifact: (
capture: CaptureResult | null,
@@ -60,7 +62,11 @@ export interface ScanActionContextInput {
focusGenshin?: boolean,
options?: CaptureOptions,
) => Promise<CaptureResult | null>;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean) => Promise<CaptureResult | null>;
captureFastSelectedSource: (
delayMs?: number,
focusGenshin?: boolean,
options?: CaptureOptions,
) => Promise<CaptureResult | null>;
}
export function createReviewContext(input: ReviewStateContextInput): ReviewStateContext {
@@ -98,6 +104,7 @@ export function createScanActionContext(input: ScanActionContextInput): ScanActi
setReviewStatus: input.setReviewStatus,
appendAutomationLog: input.appendAutomationLog,
appendClickDiagnostics: input.appendClickDiagnostics,
appendDiagnosticEvent: input.appendDiagnosticEvent,
parseArtifact: input.parseArtifact,
persistParsedArtifact: input.persistParsedArtifact,
shouldFlagArtifactForReview: (parsed) => (parsed ? shouldFlagArtifactForReview(parsed) : false),
src/features/scan/hooks/scanViewReviewHelpers.ts
+27 -4
View File
@@ -277,9 +277,28 @@ export async function saveReviewSample(
}
if (!capture) return { result: null, recoveredParsed: null, recoveredToDb: false };
const result = await reviewSamplesRepo.saveSample({
reason,
capture: {
const compactAutomaticSample = /^automatic:/i.test(reason);
const sampleCapture = compactAutomaticSample
? {
id: capture.id,
name: capture.name,
width: capture.width,
height: capture.height,
detailDataUrl: capture.detailDataUrl,
captureTarget: capture.captureTarget,
capturedAt: capture.capturedAt,
crops: capture.crops?.map((crop: NonNullable<CaptureResult["crops"]>[number]) => ({
id: crop.id,
label: crop.label,
rect: crop.rect,
dataUrl: crop.dataUrl,
})),
inventoryGrid: capture.inventoryGrid,
inventoryCount: capture.inventoryCount,
locked: capture.locked,
ocr: capture.ocr,
}
: {
id: capture.id,
name: capture.name,
width: capture.width,
@@ -299,7 +318,11 @@ export async function saveReviewSample(
inventoryCount: capture.inventoryCount,
locked: capture.locked,
ocr: capture.ocr,
},
};
const result = await reviewSamplesRepo.saveSample({
reason,
capture: sampleCapture,
parsed,
});
src/features/scan/hooks/scanViewScanActions.ts
+346 -7
View File
@@ -1,8 +1,10 @@
import { automationBlockReason, requiresAdminForAutomation } from "../../../lib/automationPlanner";
import { artifactTabClickTarget, keyPressBlocked, validateAutoScanEntryPreflight, type ScanEntryMode } from "../../../lib/autoScanEntry";
import { captureRejectionReason } from "../../../lib/scannerCaptureQuality";
import { runAutoScanLoop } from "../../../lib/autoScanLoop";
import { clampScanLimit, emptyAutoScanStats, resolveScanTargetCount, type AutoScanStats, type ScanSummary } from "../../../lib/scannerSession";
import { addCaptureTiming, clampScanLimit, emptyAutoScanStats, resolveScanTargetCount, updateScanTiming, type AutoScanStats, type ScanSummary } from "../../../lib/scannerSession";
import { getAutoReviewReason, wait } from "../../../lib/scanReviewUtils";
import { summarizeClickResult, summarizeKeyPressResult, type createScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import type { AutomationRepositoryPort, RuntimeRepositoryPort } from "../../../infrastructure/repositories/rendererBridgeRepositories";
import type {
AutomationGuard,
@@ -36,8 +38,9 @@ export interface ScanActionContext {
setReviewStatus: Dispatch<SetStateAction<string>>;
appendAutomationLog: (line: string) => void;
appendClickDiagnostics: (result: ClickResult, prefix?: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
captureSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean) => Promise<CaptureResult | null>;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
parseArtifact: (capture: CaptureResult | null) => ParsedArtifactCandidate | null;
persistParsedArtifact: (capture: CaptureResult | null, parsed: ParsedArtifactCandidate, source: string, needsReview: boolean) => Promise<boolean>;
saveReviewSample: (capture: CaptureResult | null, parsed: ParsedArtifactCandidate | null, reason?: string) => Promise<BooleanResult | null>;
@@ -47,6 +50,9 @@ export interface ScanActionContext {
export interface VisibleGridScanOptions {
scanLimit?: number;
scanEntryMode?: ScanEntryMode;
processInitialSelection?: boolean;
ocrEngine?: CaptureOptions["ocrEngine"];
}
function buildScanSignature(parsed: ParsedArtifactCandidate) {
@@ -83,6 +89,11 @@ export async function runAutoReviewScan(context: ScanActionContext): Promise<voi
const seen = new Set<string>();
const stats: AutoScanStats = { ...emptyAutoScanStats, pages: 1 };
const startedAt = Date.now();
const updateManualStats = () => {
updateScanTiming(stats, startedAt);
setAutoScanStats({ ...stats });
};
let idleTicks = 0;
const maxArtifacts = resolveScanTargetCount(scanLimit, detectedInventoryCount);
const maxIdleTicks = 90;
@@ -96,7 +107,7 @@ export async function runAutoReviewScan(context: ScanActionContext): Promise<voi
if (capture && rejection) {
await saveReviewSample(capture, parsed, `manual:capture-rejected`);
stats.review++;
setAutoScanStats({ ...stats });
updateManualStats();
}
idleTicks++;
setReviewStatus(`Manueller Scan wartet auf ein lesbares Artifact... (${stats.parsed}/${maxArtifacts})${rejection ? ` ${rejection}` : ""}`);
@@ -117,6 +128,7 @@ export async function runAutoReviewScan(context: ScanActionContext): Promise<voi
stats.attempted++;
stats.verified++;
stats.parsed++;
addCaptureTiming(stats, capture.timings);
const reason = getAutoReviewReason(capture, parsed);
const needsReview = shouldFlagArtifactForReview(parsed);
@@ -127,12 +139,13 @@ export async function runAutoReviewScan(context: ScanActionContext): Promise<voi
if (await persistParsedArtifact(capture, parsed, "manual-scan", needsReview)) {
stats.stored++;
}
setAutoScanStats({ ...stats });
updateManualStats();
setReviewStatus(`Manueller Scan: neues Artifact erkannt (${stats.parsed}/${maxArtifacts}). Klicke das naechste Artifact an oder druecke Stop.`);
await wait(700);
}
setAutoScanRunning(false);
updateScanTiming(stats, startedAt);
const status: ScanSummary["status"] = stopVisibleScanRef.current ? "stopped" : "done";
const idleSuffix = idleTicks >= maxIdleTicks ? " Keine neuen Artifacts erkannt; manueller Scan beendet." : "";
await focusDashboard();
@@ -165,6 +178,7 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
setReviewStatus,
appendAutomationLog,
appendClickDiagnostics,
appendDiagnosticEvent,
captureSelectedSource,
captureFastSelectedSource,
parseArtifact,
@@ -177,8 +191,14 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
focusDashboard,
} = context;
const scanLimit = typeof options.scanLimit === "number" ? clampScanLimit(options.scanLimit) : configuredScanLimit;
const scanEntryMode = options.scanEntryMode ?? "visible-inventory";
const ocrEngine = options.ocrEngine === "ik-traineddata" ? "ik-traineddata" : "current";
if (autoScanRunning || !bridgeReady || !selectedSourceId || !automationRepo?.focusGenshinForScanStart || !automationRepo?.focusGenshin || !automationRepo?.clickScreen || !automationRepo?.scrollScreen) return;
if (scanEntryMode !== "visible-inventory" && !automationRepo.keyPress) {
setReviewStatus("Auto-Scan-Einstieg ist nicht verfuegbar: Keypress-Bridge fehlt.");
return;
}
const requiresAdminForAutoScan = requiresAdminForAutomation(runtimeInfo);
if (requiresAdminForAutoScan) {
@@ -200,6 +220,12 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
setScanSummary(null);
setAutoScanStats(emptyAutoScanStats);
setReviewStatus("Automatischer Scan startet. ESC gedrueckt halten oder Stop klicken beendet den Scan sofort.");
appendDiagnosticEvent({
phase: "scan-start",
severity: "info",
message: `Auto-scan start requested (${scanEntryMode}, ${ocrEngine})`,
details: { scanLimit, skipRows, detectedInventoryCount, ocrEngine },
});
const freshRuntime = await runtimeRepo?.getRuntimeInfo().catch(() => null);
const adminBlockReason = automationBlockReason(freshRuntime);
@@ -207,6 +233,12 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
setAutoScanRunning(false);
setReviewStatus(adminBlockReason);
appendAutomationLog("blocked: App laeuft nicht als Administrator, keine In-Game-Klicks ausgefuehrt");
appendDiagnosticEvent({
phase: "preflight",
severity: "error",
message: adminBlockReason,
details: { elevated: freshRuntime?.isElevated, genshinFound: freshRuntime?.genshinFound },
});
setScanSummary({
mode: "Automatischer Scan",
status: "blocked",
@@ -220,6 +252,17 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
if (freshRuntime) {
const required = freshRuntime.genshinFound ? `found:${freshRuntime.targetProcess || "genshin"}` : "not-found";
appendAutomationLog(`runtime ping: elevated=${freshRuntime.isElevated} ${required}`);
appendDiagnosticEvent({
phase: "runtime",
severity: freshRuntime.genshinFound ? "ok" : "warn",
message: `Runtime ping: ${required}`,
details: {
elevated: freshRuntime.isElevated,
foreground: freshRuntime.foregroundProcess,
target: freshRuntime.targetProcess,
helperPid: freshRuntime.helperPid,
},
});
}
const focusGenshinForScanStart = automationRepo.focusGenshinForScanStart ?? automationRepo.focusGenshin;
@@ -234,6 +277,18 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
appendAutomationLog(
`focus attempt ${attempt}/3: ${current.focused ? "ok" : "failed"} found:${current.genshinFound ? "yes" : "no"} setForeground:${current.setForegroundResult ?? "n/a"} target:${current.targetProcess || "?"} fg:${current.foregroundProcess || "?"}`,
);
appendDiagnosticEvent({
phase: "focus",
severity: current.focused ? "ok" : "warn",
message: `Focus attempt ${attempt}/3 ${current.focused ? "succeeded" : "failed"}`,
details: {
found: current.genshinFound,
setForeground: current.setForegroundResult,
target: current.targetProcess,
foreground: current.foregroundProcess,
alreadyForeground: current.alreadyForeground,
},
});
if (current.focused) break;
if (!current.genshinFound) break;
}
@@ -246,6 +301,16 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
? "Genshin-Prozess wurde nicht gefunden. Bitte pruefen, ob Genshin laeuft, und Auto-Scan erneut starten."
: "Genshin konnte nicht in den Vordergrund geholt werden. Bitte Genshin manuell anklicken/fokussieren und Auto-Scan erneut starten.";
setReviewStatus(reason);
appendDiagnosticEvent({
phase: "focus",
severity: "error",
message: reason,
details: {
found: focusResult?.genshinFound,
target: focusResult?.targetProcess,
foreground: focusResult?.foregroundProcess,
},
});
setScanSummary({
mode: "Automatischer Scan",
status: "blocked",
@@ -256,7 +321,46 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
return;
}
setReviewStatus("Genshin ist im Vordergrund. Automatischer Scan startet. ESC gedrueckt halten oder Stop klicken beendet den Scan sofort.");
setReviewStatus(scanEntryMode !== "visible-inventory"
? "Genshin ist im Vordergrund. Oeffne Artifact-Inventar und warte auf Detailkarte..."
: "Genshin ist im Vordergrund. Automatischer Scan startet. ESC gedrueckt halten oder Stop klicken beendet den Scan sofort.");
const entryCapture = await prepareAutoScanEntry({
mode: scanEntryMode,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
const entryPreflight = validateAutoScanEntryPreflight(entryCapture);
if (!entryPreflight.ok) {
setAutoScanRunning(false);
setReviewStatus(entryPreflight.reason);
appendAutomationLog(`entry blocked: ${entryPreflight.reason}`);
appendDiagnosticEvent({
phase: "entry-preflight",
severity: "error",
message: entryPreflight.reason,
capture: entryCapture,
});
setScanSummary({
mode: scanEntryMode === "visible-inventory" ? "Automatischer Scan" : `Automatischer Scan (${scanEntryMode})`,
status: "blocked",
...emptyAutoScanStats,
targetCount: resolveScanTargetCount(scanLimit, detectedInventoryCount),
gridLabel: entryPreflight.reason,
});
await focusDashboard();
return;
}
setReviewStatus("Artifact-Inventar ist bereit. Automatischer Scan startet. ESC gedrueckt halten oder Stop klicken beendet den Scan sofort.");
appendDiagnosticEvent({
phase: "entry-preflight",
severity: "ok",
message: "Artifact inventory preflight passed.",
capture: entryCapture,
});
const result = await runAutoScanLoop(
{
@@ -285,7 +389,7 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
automationRepo?.getAutomationGuard?.() ??
Promise.resolve<AutomationGuard>({ ok: false, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: (delayMs = 0, focusGenshin = false) => captureSelectedSource(delayMs, focusGenshin),
captureSelectedSource: (delayMs = 0, focusGenshin = false, options) => captureSelectedSource(delayMs, focusGenshin, options),
captureFastSelectedSource,
parseArtifact,
persistParsedArtifact,
@@ -302,6 +406,9 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
scanLimit,
skipRows,
detectedInventoryCount,
processInitialSelection: options.processInitialSelection ?? scanEntryMode !== "visible-inventory",
skipInitialGridTarget: scanEntryMode !== "visible-inventory",
ocrEngine,
},
);
@@ -310,10 +417,242 @@ export async function runVisibleGridScan(context: ScanActionContext, options: Vi
await focusDashboard();
setReviewStatus(`Automatischer Scan ${result.status === "stopped" ? "gestoppt" : result.status === "blocked" ? "blockiert" : "fertig"}. ${result.stats.clicked} Klicks, ${result.stats.attempted} Positionen bearbeitet, ${result.stats.verified} Ansichten verifiziert, ${result.stats.parsed} gelesen, ${result.stats.stored} in der Datenbank, ${result.stats.review} Review-Samples, ${result.stats.duplicates} Duplikate, ${result.stats.misses} Misses.${result.blockedReason ? ` ${result.blockedReason}` : ""}`);
setScanSummary({
mode: "Automatischer Scan",
mode: scanEntryMode === "visible-inventory" ? `Automatischer Scan [${ocrEngine}]` : `Automatischer Scan (${scanEntryMode}) [${ocrEngine}]`,
status: result.status,
...result.stats,
targetCount: result.targetCount,
gridLabel: result.gridLabel,
});
}
async function prepareAutoScanEntry({
mode,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
}: {
mode: ScanEntryMode;
automationRepo: AutomationRepositoryPort;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
appendAutomationLog: (line: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
}) {
if (mode === "visible-inventory") {
const capture = await captureFastSelectedSource(0, true);
appendDiagnosticEvent({
phase: "entry-visible",
severity: capture ? "ok" : "error",
message: capture ? "Visible inventory preflight capture ready." : "Visible inventory preflight capture failed.",
capture,
});
return capture;
}
if (mode === "direct-inventory") {
return tryInventoryEntrySequence({
label: "direct",
sendEscapeFirst: false,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
}
if (mode === "auto-entry") {
const directCapture = await tryInventoryEntrySequence({
label: "auto-direct",
sendEscapeFirst: false,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
const directPreflight = validateAutoScanEntryPreflight(directCapture);
if (directPreflight.ok) return directCapture;
appendAutomationLog(`auto-entry direct path failed: ${directPreflight.reason}`);
appendDiagnosticEvent({
phase: "entry-fallback",
severity: "info",
message: `Direct inventory entry did not reach an artifact detail card. Trying IK fallback. ${directPreflight.reason}`,
capture: directCapture,
});
}
return tryInventoryEntrySequence({
label: "paimon",
sendEscapeFirst: true,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
});
}
async function tryInventoryEntrySequence({
label,
sendEscapeFirst,
automationRepo,
captureFastSelectedSource,
appendAutomationLog,
appendDiagnosticEvent,
}: {
label: string;
sendEscapeFirst: boolean;
automationRepo: AutomationRepositoryPort;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
appendAutomationLog: (line: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
}) {
if (sendEscapeFirst) {
const escapeResult = await automationRepo.keyPress?.("ESC");
appendAutomationLog(`${label} entry key ESC: ${escapeResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(escapeResult) ? "error" : "ok",
message: `${label} entry key ESC`,
details: summarizeKeyPressResult(escapeResult),
});
if (keyPressBlocked(escapeResult)) return null;
const menuProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 750,
predicate: (capture) => Boolean(capture),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: menuProbe ? "info" : "warn",
message: `${label} capture after ESC step.`,
capture: menuProbe,
});
if (menuProbe?.paimonMenu?.present) {
const closeMenuResult = await automationRepo.keyPress?.("ESC");
appendAutomationLog(`${label} entry key ESC close menu: ${closeMenuResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(closeMenuResult) ? "error" : "ok",
message: `${label} entry key ESC close menu`,
details: summarizeKeyPressResult(closeMenuResult),
});
if (keyPressBlocked(closeMenuResult)) return menuProbe;
const worldProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 750,
predicate: (capture) => Boolean(capture && !capture.paimonMenu?.present),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: worldProbe ? "info" : "warn",
message: `${label} capture after closing Paimon menu.`,
capture: worldProbe,
});
if (worldProbe?.paimonMenu?.present) {
appendAutomationLog(`${label} entry stopped: Paimon menu still visible after second ESC`);
return worldProbe;
}
}
}
const inventoryResult = await automationRepo.keyPress?.("B");
appendAutomationLog(`${label} entry key B: ${inventoryResult?.ok ? "ok" : "blocked"}`);
appendDiagnosticEvent({
phase: "entry-key",
severity: keyPressBlocked(inventoryResult) ? "error" : "ok",
message: `${label} entry key B`,
details: summarizeKeyPressResult(inventoryResult),
});
if (keyPressBlocked(inventoryResult)) return null;
const tabProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 1200,
predicate: (capture) => Boolean(capture && !capture.paimonMenu?.present && capture.inventoryGrid && capture.inventoryGrid.source !== "missing"),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: tabProbe ? "info" : "warn",
message: `${label} capture after Inventory-key step.`,
capture: tabProbe,
});
if (tabProbe?.paimonMenu?.present) {
appendAutomationLog(`${label} entry stopped: Paimon menu still visible after Inventory key`);
appendDiagnosticEvent({
phase: "entry-capture",
severity: "info",
message: `${label} entry stopped before tab click because Paimon menu is still visible.`,
capture: tabProbe,
});
return tabProbe;
}
if (!tabProbe?.inventoryGrid || tabProbe.inventoryGrid.source === "missing") return tabProbe;
const target = artifactTabClickTarget(tabProbe);
appendAutomationLog(`${label} entry artifact tab -> ${target.x},${target.y}`);
const click = await automationRepo.clickScreen(target.x, target.y);
appendDiagnosticEvent({
phase: "entry-click",
severity: click.inputBlocked || click.clicked === false || click.moved === false ? "warn" : "ok",
message: `${label} artifact tab click at ${target.x},${target.y}`,
details: summarizeClickResult(click),
capture: tabProbe,
});
if (click.inputBlocked || click.clicked === false) return null;
const gridProbe = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (capture) => Boolean(capture?.inventoryGrid && capture.inventoryGrid.source !== "missing"),
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: gridProbe ? "info" : "warn",
message: `${label} capture after artifact-tab click before first tile selection.`,
capture: gridProbe,
});
const firstTarget = gridProbe?.inventoryGrid?.centers?.[0];
if (!firstTarget) return gridProbe;
appendAutomationLog(`${label} entry first artifact tile -> ${firstTarget.x},${firstTarget.y}`);
const firstTileClick = await automationRepo.clickScreen(firstTarget.x, firstTarget.y);
appendDiagnosticEvent({
phase: "entry-click",
severity: firstTileClick.inputBlocked || firstTileClick.clicked === false || firstTileClick.moved === false ? "warn" : "ok",
message: `${label} first artifact tile click at ${firstTarget.x},${firstTarget.y}`,
details: summarizeClickResult(firstTileClick),
capture: gridProbe,
});
if (firstTileClick.inputBlocked || firstTileClick.clicked === false) return null;
const finalCapture = await waitForEntryCapture({
captureFastSelectedSource,
timeoutMs: 900,
predicate: (capture) => validateAutoScanEntryPreflight(capture).ok,
});
appendDiagnosticEvent({
phase: "entry-capture",
severity: finalCapture ? "info" : "warn",
message: `${label} final capture after first artifact selection.`,
capture: finalCapture,
});
return finalCapture;
}
async function waitForEntryCapture({
captureFastSelectedSource,
timeoutMs,
pollMs = 150,
predicate,
}: {
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
timeoutMs: number;
pollMs?: number;
predicate: (capture: CaptureResult | null) => boolean;
}) {
const startedAt = Date.now();
let latest: CaptureResult | null = null;
while (Date.now() - startedAt <= timeoutMs) {
latest = await captureFastSelectedSource(0, true);
if (predicate(latest)) return latest;
const remaining = timeoutMs - (Date.now() - startedAt);
if (remaining <= 0) break;
await wait(Math.min(pollMs, remaining));
}
return latest;
}
src/features/scan/hooks/useScanCommandListener.ts
+9 -4
View File
@@ -1,6 +1,6 @@
import { useEffect } from "react";
import type { AutomationRepositoryPort } from "../../../infrastructure/repositories/rendererBridgeRepositoryTypes";
import type { ScannerCommand } from "../../../types/global";
import type { CaptureOptions, ScannerCommand } from "../../../types/global";
import type { VisibleGridScanOptions } from "./scanViewScanActions";
interface ScanCommandListenerInput {
@@ -9,6 +9,7 @@ interface ScanCommandListenerInput {
isScanning: boolean;
selectedSourceId: string;
requestScanStop: (reason: string) => void;
runGuidedAutoScan: (options?: { scanLimit?: number; ocrEngine?: CaptureOptions["ocrEngine"] }) => Promise<void>;
runVisibleGridScan: (options?: VisibleGridScanOptions) => Promise<void>;
}
@@ -18,6 +19,7 @@ export function useScanCommandListener({
isScanning,
selectedSourceId,
requestScanStop,
runGuidedAutoScan,
runVisibleGridScan,
}: ScanCommandListenerInput) {
useEffect(() => {
@@ -29,9 +31,12 @@ export function useScanCommandListener({
}
const commandType = typeof command === "string" ? command : command.type;
if (commandType === "start-auto" && !autoScanRunning && !isScanning && selectedSourceId) {
const options = typeof command === "string" ? undefined : { scanLimit: command.scanLimit };
void runVisibleGridScan(options);
if (typeof command === "string" || !command.scanEntryMode) {
void runGuidedAutoScan(typeof command === "string" ? undefined : { scanLimit: command.scanLimit, ocrEngine: command.ocrEngine });
return;
}
void runVisibleGridScan({ scanLimit: command.scanLimit, scanEntryMode: command.scanEntryMode, ocrEngine: command.ocrEngine });
}
});
}, [automationRepo, autoScanRunning, isScanning, selectedSourceId, requestScanStop, runVisibleGridScan]);
}, [automationRepo, autoScanRunning, isScanning, selectedSourceId, requestScanStop, runGuidedAutoScan, runVisibleGridScan]);
}
src/features/scan/hooks/useScanSnapshotPublisher.ts
+29
View File
@@ -4,6 +4,8 @@ import { type AutoScanStats, type ScanSummary } from "../../../lib/scannerSessio
import type { RuntimeInfo } from "../../../types/global";
import type { SnapshotRepositoryPort } from "../../../infrastructure/repositories/rendererBridgeRepositoryTypes";
import type { CaptureResult } from "../../../types/global";
import { validateLookupPackage } from "../../../lib/genshinLookup";
import type { ScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
type InventoryGrid = NonNullable<CaptureResult["inventoryGrid"]>;
@@ -17,6 +19,7 @@ interface ScanSnapshotPublisherInput {
snapshot: AppSnapshot;
latestInventoryGrid: InventoryGrid | null | undefined;
automationLog: string[];
diagnosticEvents: ScanDiagnosticEvent[];
runtimeInfo: RuntimeInfo | null;
storedTotal: number | null;
learningRuleCount: number;
@@ -33,6 +36,7 @@ export function useScanSnapshotPublisher({
snapshot,
latestInventoryGrid,
automationLog,
diagnosticEvents,
runtimeInfo,
storedTotal,
learningRuleCount,
@@ -52,9 +56,33 @@ export function useScanSnapshotPublisher({
snapshotBuilds: snapshot.builds.length,
grid: latestInventoryGrid ?? null,
automationLog: automationLog.slice(-12),
diagnosticEvents: diagnosticEvents.slice(-8).map((event) => ({
...event,
capture: event.capture
? {
...event.capture,
screenshots: event.capture.screenshots
? {
detail: event.capture.screenshots.detail ? "[detail screenshot available in Diagnose]" : undefined,
inventory: event.capture.screenshots.inventory ? "[inventory screenshot available in Diagnose]" : undefined,
full: event.capture.screenshots.full ? "[full screenshot omitted]" : undefined,
}
: undefined,
}
: undefined,
})),
runtimeInfo,
storedTotal,
learningRuleCount,
lookupStatus: validateLookupPackage(),
ocrEngine: scanSummary?.mode.includes("ik-traineddata") ? "ik-traineddata" : "current",
entryMode: scanSummary?.mode.includes("auto-entry")
? "auto-entry"
: scanSummary?.mode.includes("direct-inventory")
? "direct-inventory"
: scanSummary?.mode.includes("paimon-menu")
? "paimon-menu"
: "visible-inventory",
updatedAt: new Date().toISOString(),
}).catch(() => undefined);
}, [
@@ -67,6 +95,7 @@ export function useScanSnapshotPublisher({
snapshot,
latestInventoryGrid,
automationLog,
diagnosticEvents,
runtimeInfo,
storedTotal,
learningRuleCount,
src/features/scan/hooks/useScanViewActions.ts
+68 -4
View File
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useMemo } from "react";
import { useCallback, useEffect, useMemo, useRef } from "react";
import type { Dispatch, MutableRefObject, SetStateAction } from "react";
import { runAutoReviewScan as runAutoReviewScanAction, runVisibleGridScan as runVisibleGridScanAction, type VisibleGridScanOptions } from "./scanViewScanActions";
import {
@@ -21,6 +21,8 @@ import type {
} from "../../../infrastructure/repositories/rendererBridgeRepositoryTypes";
import type { AutoScanStats, ScanSummary } from "../../../lib/scannerSession";
import type { RuntimeInfo } from "../../../types/global";
import type { createScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import { validateAutoScanEntryPreflight } from "../../../lib/autoScanEntry";
type BooleanSetter = Dispatch<SetStateAction<boolean>>;
type NumberSetter = Dispatch<SetStateAction<number>>;
@@ -49,6 +51,7 @@ interface ScanViewActionInput {
setReviewStatus: StringSetter;
appendAutomationLog: (line: string) => void;
appendClickDiagnostics: (result: ClickResult, prefix?: string) => void;
appendDiagnosticEvent: (event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => void;
parseArtifact: (capture: CaptureResult | null) => ParsedArtifactCandidate | null;
artifactRepo?: ArtifactRepositoryPort;
reviewSamplesRepo?: ReviewSampleRepositoryPort;
@@ -77,6 +80,7 @@ export interface ScanViewActionResult {
loadReviewQueue: () => Promise<void>;
openReviewQueue: () => Promise<void>;
runAutoReviewScan: () => Promise<void>;
runGuidedAutoScan: (options?: { scanLimit?: number; ocrEngine?: CaptureOptions["ocrEngine"] }) => Promise<void>;
runVisibleGridScan: (options?: VisibleGridScanOptions) => Promise<void>;
}
@@ -99,6 +103,7 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
setReviewStatus,
appendAutomationLog,
appendClickDiagnostics,
appendDiagnosticEvent,
parseArtifact,
artifactRepo,
reviewSamplesRepo,
@@ -116,6 +121,7 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
canCaptureSource,
setReviewQueueOpen,
} = input;
const learningInitializedRef = useRef(false);
const requestScanStop = useCallback((reason = "Stop angefordert.") => {
stopVisibleScanRef.current = true;
@@ -154,8 +160,11 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
);
useEffect(() => {
if (learningInitializedRef.current) return;
if (!learningRepo && !reviewSamplesRepo && !artifactRepo) return;
learningInitializedRef.current = true;
void initializeLearningState(reviewContext);
}, [reviewContext]);
}, [artifactRepo, learningRepo, reviewContext, reviewSamplesRepo]);
const focusDashboard = useCallback(async () => {
try {
@@ -215,12 +224,25 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
setReviewStatus,
appendAutomationLog,
appendClickDiagnostics,
appendDiagnosticEvent,
parseArtifact,
persistParsedArtifact: parseArtifactAndPersist,
saveReviewSample: handleSaveReviewSample,
focusDashboard,
captureSelectedSource,
captureFastSelectedSource: (delayMs = 0, focusGenshin = false) => captureSelectedSource(delayMs, focusGenshin, { skipOcr: true }),
captureSelectedSource: (delayMs = 0, focusGenshin = false, options) => captureSelectedSource(delayMs, focusGenshin, {
ocrMode: "artifact",
omitFullFrame: true,
omitInventoryPreview: true,
...options,
}),
captureFastSelectedSource: (delayMs = 0, focusGenshin = false, options) => captureSelectedSource(delayMs, focusGenshin, {
skipOcr: true,
omitFullFrame: true,
omitCrops: true,
omitCropImages: true,
omitLockState: true,
...options,
}),
}),
[
autoScanRunning,
@@ -240,6 +262,7 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
setReviewStatus,
appendAutomationLog,
appendClickDiagnostics,
appendDiagnosticEvent,
parseArtifact,
parseArtifactAndPersist,
handleSaveReviewSample,
@@ -276,12 +299,52 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
scanActionContext,
]);
const runGuidedAutoScan = useCallback(async (options: { scanLimit?: number; ocrEngine?: CaptureOptions["ocrEngine"] } = {}) => {
if (autoScanRunning || !bridgeReady || !selectedSourceId || !automationRepo?.clickScreen || !automationRepo?.scrollScreen) {
return;
}
setReviewStatus("Auto-Scan prueft den Startzustand ohne OCR...");
const preflightCapture = await captureSelectedSource(0, true, {
skipOcr: true,
omitFullFrame: true,
omitCrops: true,
omitCropImages: true,
omitLockState: true,
});
const visibleInventoryReady = validateAutoScanEntryPreflight(preflightCapture).ok;
appendDiagnosticEvent({
phase: "guided-start",
severity: visibleInventoryReady ? "ok" : "info",
message: visibleInventoryReady
? "Artifact inventory detail view already visible; starting scan directly."
: "Artifact detail view is not ready; trying direct inventory entry, then Inventory Kamera fallback.",
capture: preflightCapture,
});
await runVisibleGridScanAction(scanActionContext, {
scanLimit: options.scanLimit,
scanEntryMode: visibleInventoryReady ? "visible-inventory" : "auto-entry",
processInitialSelection: visibleInventoryReady,
ocrEngine: options.ocrEngine,
});
}, [
autoScanRunning,
bridgeReady,
selectedSourceId,
automationRepo?.clickScreen,
automationRepo?.scrollScreen,
setReviewStatus,
captureSelectedSource,
appendDiagnosticEvent,
scanActionContext,
]);
useScanCommandListener({
automationRepo,
autoScanRunning,
isScanning,
selectedSourceId,
requestScanStop,
runGuidedAutoScan,
runVisibleGridScan,
});
@@ -291,6 +354,7 @@ export function useScanViewActions(input: ScanViewActionInput): ScanViewActionRe
loadReviewQueue: loadReviewQueueAction,
openReviewQueue: openReviewQueueModal,
runAutoReviewScan,
runGuidedAutoScan,
runVisibleGridScan,
};
}
src/features/scan/hooks/useScanViewController.ts
+24 -1
View File
@@ -14,6 +14,7 @@ import { useScanSnapshotPublisher } from "./useScanSnapshotPublisher";
import { useScanViewActions } from "./useScanViewActions";
import { useScanViewStateSync } from "./useScanViewStateSync";
import { emptyAutoScanStats, resolveScanTargetCount, type AutoScanStats, type ScanSummary } from "../../../lib/scannerSession";
import { createScanDiagnosticEvent, summarizeClickResult, type ScanDiagnosticEvent } from "../../../lib/scanDiagnosticsLog";
import type { ScanViewProps, ScanViewControllerResult } from "../types";
import type { CaptureResult, ClickResult, ReviewSampleRecord } from "../../../types/global";
import type { StoredArtifactRecord } from "../../../types/storage";
@@ -56,6 +57,7 @@ export function useScanViewController({
const [scanLimitTouched, setScanLimitTouched] = useState(false);
const [skipRows, setSkipRows] = useState(0);
const [automationLog, setAutomationLog] = useState<string[]>([]);
const [diagnosticEvents, setDiagnosticEvents] = useState<ScanDiagnosticEvent[]>([]);
const [storedTotal, setStoredTotal] = useState<number | null>(null);
const [devMode, setDevMode] = useState(() => localStorage.getItem("gaa-dev-mode") === "1");
const [scannerLearningRules, setScannerLearningRules] = useState<ScannerLearningRules>({ textReplacements: {} });
@@ -92,12 +94,28 @@ export function useScanViewController({
setAutomationLog((previous) => [...previous.slice(-11), `${new Date().toLocaleTimeString()} ${line}`]);
}, []);
const appendDiagnosticEvent = useCallback((event: Omit<Parameters<typeof createScanDiagnosticEvent>[0], "includeFullScreenshot">) => {
setDiagnosticEvents((previous) => [
...previous.slice(-23),
createScanDiagnosticEvent({
...event,
includeFullScreenshot: false,
}),
]);
}, []);
const appendClickDiagnostics = useCallback((result: ClickResult, prefix = "input") => {
const cursor = `${result.cursorX ?? "?"},${result.cursorY ?? "?"}`;
const focus = result.focused ? `fg:${result.foregroundProcess || "Genshin"}` : `fg-miss:${result.foregroundProcess || "?"}`;
const blocked = result.inputBlocked ? " input:blocked" : "";
appendAutomationLog(`${prefix}: ${focus}${blocked} cursor ${cursor} moved:${result.moved ? "yes" : "no"} clicked:${result.clicked ? "yes" : "no"}`);
}, [appendAutomationLog]);
appendDiagnosticEvent({
phase: "input",
severity: result.inputBlocked || result.clicked === false || result.moved === false ? "warn" : "ok",
message: `${prefix}: click ${result.clicked ? "sent" : "not sent"}`,
details: summarizeClickResult(result),
});
}, [appendAutomationLog, appendDiagnosticEvent]);
const toggleDevMode = useCallback(() => {
setDevMode((previous) => {
@@ -118,6 +136,7 @@ export function useScanViewController({
loadReviewQueue: loadReviewQueueAction,
openReviewQueue: openReviewQueueModal,
runAutoReviewScan,
runGuidedAutoScan,
runVisibleGridScan,
} = useScanViewActions({
autoScanRunning,
@@ -137,6 +156,7 @@ export function useScanViewController({
setReviewStatus,
appendAutomationLog,
appendClickDiagnostics,
appendDiagnosticEvent,
parseArtifact,
artifactRepo,
reviewSamplesRepo,
@@ -209,6 +229,7 @@ export function useScanViewController({
snapshot,
latestInventoryGrid: latestCapture?.inventoryGrid ?? null,
automationLog,
diagnosticEvents,
runtimeInfo,
storedTotal,
learningRuleCount,
@@ -230,6 +251,7 @@ export function useScanViewController({
scanLimitTouched,
skipRows,
automationLog,
diagnosticEvents,
storedTotal,
devMode,
scannerLearningRules,
@@ -267,6 +289,7 @@ export function useScanViewController({
loadReviewQueue: loadReviewQueueAction,
openReviewQueue: openReviewQueueModal,
runAutoReviewScan,
runGuidedAutoScan,
runVisibleGridScan,
canGoodInterop,
exportGoodFromStore,
src/features/scan/types.ts
+3
View File
@@ -3,6 +3,7 @@ import type { BooleanResult, CaptureOptions, CaptureResult, CaptureSourceInfo, R
import type { StoredArtifactRecord } from "../../types/storage";
import type { AutoScanStats, ScanSummary } from "../../lib/scannerSession";
import type { ParsedArtifactCandidate } from "../../lib/artifactOcrParser";
import type { ScanDiagnosticEvent } from "../../lib/scanDiagnosticsLog";
import type { ScannerLearningRules } from "../../lib/scannerLearning";
import type { Dispatch, SetStateAction } from "react";
import type { analyzeReviewSamples } from "../../lib/reviewSampleAnalysis";
@@ -36,6 +37,7 @@ export interface ScanViewControllerResult {
scanLimitTouched: boolean;
skipRows: number;
automationLog: string[];
diagnosticEvents: ScanDiagnosticEvent[];
storedTotal: number | null;
devMode: boolean;
scannerLearningRules: ScannerLearningRules;
@@ -79,6 +81,7 @@ export interface ScanViewControllerResult {
loadReviewQueue: () => Promise<void>;
openReviewQueue: () => Promise<void>;
runAutoReviewScan: () => Promise<void>;
runGuidedAutoScan: (options?: { scanLimit?: number; ocrEngine?: CaptureOptions["ocrEngine"] }) => Promise<void>;
runVisibleGridScan: () => Promise<void>;
canGoodInterop: boolean;
exportGoodFromStore: () => Promise<{ ok: boolean; path?: string; count: number }>;
src/infrastructure/repositories/rendererBridgeRepositoryFactory.ts
+6
View File
@@ -22,6 +22,7 @@ import type {
RuntimeInfo,
SaveResultWithPath,
ScrollResult,
KeyPressResult,
AutomationGuard,
ClickResult,
ReviewSampleListResult,
@@ -111,6 +112,10 @@ function emptyScrollResult(): ScrollResult {
return { ok: false, notchesSent: 0, inputBlocked: false };
}
function emptyKeyPressResult(key = ""): KeyPressResult {
return { ok: false, key, inputBlocked: false, eventsSent: 0 };
}
function emptyBooleanResult(): BooleanResult {
return { ok: false };
}
@@ -203,6 +208,7 @@ export function createRendererRepositories(): RendererRepositories | null {
clickScreen: (x, y) => createBridgeSafeCall(() => bridge.clickScreen(x, y), emptyClickResult()),
scrollScreen: (notches, anchorX, anchorY) =>
createBridgeSafeCall(() => bridge.scrollScreen(notches, anchorX, anchorY), emptyScrollResult()),
keyPress: (key) => createBridgeSafeCall(() => bridge.keyPress(key), emptyKeyPressResult(key)),
onCommand: bridge.onScannerCommand,
};
src/infrastructure/repositories/rendererBridgeRepositoryTypes.ts
+2
View File
@@ -20,6 +20,7 @@ import type {
SaveResultWithPath,
ScrollResult,
ScannerLearningRulePayload,
KeyPressResult,
} from "../../types/global";
import type { AppSnapshot } from "../../types/domain";
import type { StoredArtifactRecord } from "../../types/storage";
@@ -64,6 +65,7 @@ export interface AutomationRepositoryPort {
focusMainWindow(): Promise<BooleanResult>;
clickScreen(x: number, y: number): Promise<ClickResult>;
scrollScreen(notches: number, anchorX?: number, anchorY?: number): Promise<ScrollResult>;
keyPress(key: string): Promise<KeyPressResult>;
onCommand(callback: (command: ScannerCommand) => void): () => void;
}
src/lib/artifactOcrParser.test.ts
+91
View File
@@ -95,6 +95,83 @@ describe("parseArtifactCandidate", () => {
expect(parsed?.mainValue).toBe("46.6%");
});
it("derives main stat value when the fast OCR profile skips the value crop", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Maidens Fading Beauty",
"artifact-slot": "Goblet of Eonothem",
"artifact-main-stat-label": "Cryo DMG Bonus",
"artifact-level": "+20",
"artifact-substats": "+ CRIT Rate+6.6%\n+ CRIT DMG+12.4%\n+ HP+269\n+ ATK+16.3%",
"artifact-footer": "Equipped: Skirk",
}));
expect(parsed?.mainStat).toBe("Cryo DMG Bonus");
expect(parsed?.mainValue).toBe("46.6%");
expect(parsed?.fields.mainValue.source).toBe("derived");
});
it("uses split main stat value OCR when level OCR is missing", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Heldenepos's Unspoken Tale",
"artifact-slot": "Goblet of Eonothem",
"artifact-main-stat-label": "Pyro DMG Bonus",
"artifact-main-stat-value": "46.6%",
"artifact-level": "",
"artifact-substats": "+ Energy Recharge+13.0%\n+ ATK+5.8%\n+ Elemental Mastery+58\n+ CRIT DMG+14.0%",
}));
expect(parsed?.slot).toBe("Goblet of Eonothem");
expect(parsed?.mainStat).toBe("Pyro DMG Bonus");
expect(parsed?.mainValue).toBe("46.6%");
expect(parsed?.fields.mainValue.source).toBe("ocr");
});
it("derives fixed flower and plume values without a value crop", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Gladiator's Nostalgia",
"artifact-slot": "Flower of Life",
"artifact-main-stat-label": "HP",
"artifact-level": "+20",
"artifact-substats": "+ CRIT DMG+7.0%\n+ Energy Recharge+6.5%\n+ Elemental Mastery+68",
}));
expect(parsed?.mainStat).toBe("HP");
expect(parsed?.mainValue).toBe("4,780");
expect(parsed?.fields.mainValue.source).toBe("derived");
});
it("derives slot and set from the piece name when fast auto-scan skips slot OCR", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Gladiator's Nostalgia",
"artifact-main-stat-label": "HP",
"artifact-level": "+20",
"artifact-substats": "+ CRIT DMG+7.0%\n+ Energy Recharge+6.5%\n+ Elemental Mastery+68",
}));
expect(parsed?.slot).toBe("Flower of Life");
expect(parsed?.fields.slot.source).toBe("derived");
expect(parsed?.setName).toBe("Gladiator's Finale");
expect(parsed?.fields.setName.source).toBe("derived");
expect(parsed?.mainStat).toBe("HP");
expect(parsed?.mainValue).toBe("4,780");
});
it("caps derived slot and set confidence when the piece name is fuzzy", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-name": "Gladiator Nostalg",
"artifact-main-stat-label": "HP",
"artifact-level": "+20",
"artifact-substats": "+ CRIT DMG+7.0%\n+ Energy Recharge+6.5%\n+ Elemental Mastery+68",
}));
expect(parsed?.name).toBe("Gladiator's Nostalgia");
expect(parsed?.fields.name.confidence).toBeLessThan(94);
expect(parsed?.slot).toBe("Flower of Life");
expect(parsed?.fields.slot.confidence).toBeLessThanOrEqual(parsed?.fields.name.confidence ?? 0);
expect(parsed?.setName).toBe("Gladiator's Finale");
expect(parsed?.fields.setName.confidence).toBeLessThanOrEqual(parsed?.fields.name.confidence ?? 0);
});
it("does not let substats override circlet crit main stats", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-title": "Holy Crown of the Believer\nCirclet of Logos",
@@ -342,4 +419,18 @@ describe("parseArtifactCandidate", () => {
expect(sands?.mainValue).toBe("46.6%");
expect(sands?.mainStat).toBe("Unknown main stat");
});
it("derives DEF percent for goblet max-value reads when the OCR stat label is garbled", () => {
const parsed = parseArtifactCandidate(captureFromOcr({
"artifact-title": "Vessel of Plenty",
"artifact-main-stat": "Goblet of Eonothem\nDLT\n58.3%",
"artifact-substats": "- DEF+58\n- Elemental Mastery+47\n+ CRIT Rate+5.8%\n+ HP+299",
"artifact-footer": "Equipped: I -",
}));
expect(parsed?.slot).toBe("Goblet of Eonothem");
expect(parsed?.mainValue).toBe("58.3%");
expect(parsed?.mainStat).toBe("DEF%");
expect(parsed?.setName).toBe("Night of the Sky's Unveiling");
});
});
src/lib/artifactOcrParser.ts
+45 -12
View File
@@ -19,6 +19,7 @@ import {
textReplacements,
} from "./genshinData.js";
import { fuzzyFindKnown, simplifyForMatch } from "./fuzzyMatch.js";
import { matchCharacter, matchPiece, matchSet, matchSlot, matchStat } from "./genshinLookup.js";
import { implausibleSubstats } from "./substatRolls.js";
type MainStatValueReference = { stat: string; base: number; max: number };
@@ -62,15 +63,22 @@ export function parseArtifactCandidate(capture: CaptureResult | null): ParsedArt
(capture.ocr ?? []).map((entry: (typeof capture.ocr)[number]) => [entry.id, normalizeText(entry.text)]),
);
const allText = normalizeText((capture.ocr ?? []).map((entry: (typeof capture.ocr)[number]) => entry.text).join("\n"));
const titleText = byId.get("artifact-title") ?? "";
const mainText = byId.get("artifact-main-stat") ?? "";
const nameText = byId.get("artifact-name") ?? "";
const slotOnlyText = byId.get("artifact-slot") ?? "";
const legacyTitleText = byId.get("artifact-title") ?? "";
const titleText = [nameText, slotOnlyText].filter(Boolean).join("\n") || legacyTitleText;
const mainLabelText = byId.get("artifact-main-stat-label") ?? "";
const mainValueText = byId.get("artifact-main-stat-value") ?? "";
const legacyMainText = byId.get("artifact-main-stat") ?? "";
const mainText = [mainLabelText, mainValueText].filter(Boolean).join("\n") || legacyMainText;
const levelText = byId.get("artifact-level") ?? "";
const substatText = byId.get("artifact-substats") ?? "";
const setText = byId.get("artifact-set-effects") ?? "";
const footerText = byId.get("artifact-footer") ?? "";
const nameField = parseArtifactName(titleText);
const slotField = parseSlot(titleText + "\n" + allText, nameField.value);
const levelField = parseArtifactLevel(substatText + "\n" + mainText + "\n" + allText);
const slotField = parseSlot([slotOnlyText, titleText, allText].filter(Boolean).join("\n"), nameField);
const levelField = parseArtifactLevel([levelText, substatText, mainText, allText].filter(Boolean).join("\n"));
const parsedLevel = levelField.value ? Number.parseInt(levelField.value, 10) : null;
const level = parsedLevel ?? 0;
let mainStatField = inferMainStat(slotField.value, mainText);
@@ -100,7 +108,7 @@ export function parseArtifactCandidate(capture: CaptureResult | null): ParsedArt
}
const substats = parseSubstats([substatText, leadingSetEffectText(setText)].filter(Boolean).join("\n"));
const substatsField = field(substats.join(", "), substats.length >= 4 ? 96 : substats.length >= 3 ? 82 : substats.length > 0 ? 55 : 0, substats.length ? "ocr" : "missing");
const setField = parseSetName(setText, nameField.value);
const setField = parseSetName(setText, nameField);
const equippedField = parseEquippedCharacter(footerText + "\n" + allText);
const notes: string[] = [];
@@ -161,6 +169,10 @@ function parseArtifactName(titleText: string): ParsedField {
.filter(Boolean);
for (const line of titleLines) {
const match = matchPiece(line);
if (match.value) {
return field(match.value, match.confidence, match.source === "exact" || match.source === "alias" ? "database" : "fallback");
}
const alias = normalizePieceAlias(line);
if (alias) return field(alias, 96, "database");
}
@@ -172,13 +184,17 @@ function parseArtifactName(titleText: string): ParsedField {
return fallback ? field(fallback, 50, "fallback") : field("", 0, "missing");
}
function parseSlot(text: string, artifactName: string): ParsedField {
function parseSlot(text: string, artifactName: ParsedField): ParsedField {
const slotLines = text
.split("\n")
.map((line) => cleanupOcrLabel(line))
.filter(Boolean);
for (const line of slotLines) {
const match = matchSlot(line);
if (match.value) {
return field(match.value, match.confidence, match.source === "fuzzy" ? "fallback" : "ocr");
}
const alias = normalizeSlotAlias(line);
if (alias) return field(alias, 96, "ocr");
}
@@ -186,18 +202,22 @@ function parseSlot(text: string, artifactName: string): ParsedField {
const directSlot = fuzzyFindKnown(text, slotNames, 0.68);
if (directSlot) return field(directSlot.value, Math.round(directSlot.score * 100), directSlot.score >= 0.95 ? "ocr" : "fallback");
const derivedSlot = artifactName ? pieceToSlot.get(artifactName) ?? "" : "";
return derivedSlot ? field(derivedSlot, 94, "derived") : field("", 0, "missing");
const derivedSlot = artifactName.value ? pieceToSlot.get(artifactName.value) ?? "" : "";
return derivedSlot ? field(derivedSlot, derivedConfidence(artifactName, 94), "derived") : field("", 0, "missing");
}
function parseSetName(setText: string, artifactName: string): ParsedField {
const setFromPiece = artifactName ? pieceToSet.get(artifactName) : undefined;
function parseSetName(setText: string, artifactName: ParsedField): ParsedField {
const setFromPiece = artifactName.value ? pieceToSet.get(artifactName.value) : undefined;
const candidateLines = setText
.split("\n")
.map((line) => line.trim().replace(/:$/, ""))
.filter((line) => line.length > 3 && !/^\d/.test(line) && !/piece set/i.test(line));
for (const line of candidateLines) {
const match = matchSet(line);
if (match.value) {
return field(match.value, match.confidence, match.source === "exact" || match.source === "alias" ? "database" : "fallback");
}
const alias = normalizeSetAlias(line);
if (alias) return field(alias, 96, "database");
}
@@ -206,10 +226,15 @@ function parseSetName(setText: string, artifactName: string): ParsedField {
const setFromText = fuzzyFindKnown(`${directLine ?? ""}\n${setText}`, knownSets, 0.64);
if (setFromText && (!setFromPiece || setFromText.score >= 0.78)) return field(setFromText.value, Math.round(setFromText.score * 100), setFromText.score >= 0.95 ? "ocr" : "fallback");
if (setFromPiece) return field(setFromPiece, 92, "derived");
if (setFromPiece) return field(setFromPiece, derivedConfidence(artifactName, 92), "derived");
return setFromText ? field(setFromText.value, Math.round(setFromText.score * 100), "fallback") : field("", 0, "missing");
}
function derivedConfidence(sourceField: ParsedField, maxConfidence: number) {
if (sourceField.source === "database" || sourceField.confidence >= maxConfidence) return maxConfidence;
return Math.max(45, Math.min(maxConfidence, sourceField.confidence));
}
function parseArtifactLevel(text: string): ParsedField {
const lines = normalizeText(text)
.split("\n")
@@ -294,6 +319,10 @@ function inferMainStat(slot: string, text: string): ParsedField {
if (direct) return field(promotePercentVariant(direct, text), 94, "ocr");
const allowedForSlot = sortLongestFirst(allowedMainStatsForSlot(slot));
const lookup = matchStat(text);
if (lookup.value && allowedForSlot.includes(lookup.value)) {
return field(promotePercentVariant(lookup.value, text), lookup.confidence, lookup.source === "fuzzy" ? "fallback" : "ocr");
}
const fuzzyAllowed = fuzzyFindKnown(text, allowedForSlot, 0.68);
if (fuzzyAllowed) return field(promotePercentVariant(fuzzyAllowed.value, text), Math.round(fuzzyAllowed.score * 100), "fallback");
@@ -332,6 +361,7 @@ function findDirectMainStat(text: string) {
if (hasPercentValue && /(^|\s)atk(\s|$)/i.test(text)) return "ATK%";
if (hasPercentValue && /(^|\s)hp(\s|$)/i.test(text)) return "HP%";
if (hasPercentValue && /(^|\s)def(\s|$)/i.test(text)) return "DEF%";
if (hasPercentValue && /(^|\s)dlt(\s|$)/i.test(text)) return "DEF%";
return "";
}
@@ -461,6 +491,10 @@ function parseEquippedCharacter(text: string): ParsedField {
if (!equippedLine) return field("Not detected", 45, "missing");
const afterLabel = cleanupCharacterNoise(equippedLine);
const match = afterLabel ? matchCharacter(afterLabel) : null;
if (match?.value) {
return field(match.value, match.confidence, match.source === "fuzzy" ? "fallback" : "database");
}
const alias = afterLabel ? normalizeCharacterAlias(afterLabel) : "";
if (alias) return field(alias, 96, "database");
const known = afterLabel ? fuzzyFindKnown(afterLabel, knownCharacters, 0.6) : null;
@@ -585,4 +619,3 @@ function sortLongestFirst(values: string[]) {
function escapeRegex(value: string) {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
src/lib/autoScanEntry.test.ts
+93
View File
@@ -0,0 +1,93 @@
import { describe, expect, it } from "vitest";
import type { CaptureResult } from "../types/global";
import { artifactTabClickTarget, buildAutoScanEntryPlan, keyPressBlocked, validateAutoScanEntryPreflight } from "./autoScanEntry";
function capture(overrides: Partial<CaptureResult> = {}): CaptureResult {
return {
id: "test",
name: "test",
width: 1920,
height: 1080,
dataUrl: "",
capturedAt: new Date(0).toISOString(),
inventoryGrid: {
centers: [{ x: 179, y: 254, row: 0, col: 0 }],
rows: 4,
cols: 8,
confidence: 76,
source: "detected",
},
captureTarget: "genshin-client",
artifactDetail: { present: true, confidence: 84, orangeHits: 14, greenHits: 8, textHits: 30 },
layout: { aspect: "1.78:1", isSixteenNine: true, warning: "" },
...overrides,
};
}
describe("autoScanEntry", () => {
it("plans only read-only keys/clicks for the Paimon entry", () => {
expect(buildAutoScanEntryPlan("paimon-menu")).toEqual([
{ type: "key", key: "ESC", label: "Inventory Kamera step: leave the already-open Paimon menu" },
{ type: "key", key: "ESC", label: "If Paimon is still visible, close it before opening inventory" },
{ type: "key", key: "B", label: "Inventory Kamera step: open inventory from world" },
{ type: "click-artifact-tab", label: "Inventory Kamera step: select artifact inventory tab" },
{ type: "click-first-artifact", label: "Select first visible artifact so the detail card is open" },
{ type: "capture-preflight", label: "Verify artifact inventory grid and detail card" },
]);
});
it("plans a guided auto entry with direct world path before IK fallback", () => {
expect(buildAutoScanEntryPlan("auto-entry")).toEqual([
{ type: "key", key: "B", label: "Try direct inventory entry from world" },
{ type: "click-artifact-tab", label: "Select artifact inventory tab if inventory opened" },
{ type: "click-first-artifact", label: "Select first visible artifact so the detail card is open" },
{ type: "key", key: "ESC", label: "Fallback: Inventory Kamera step from already-open Paimon menu" },
{ type: "key", key: "ESC", label: "Fallback: close Paimon menu if ESC opened or left it visible" },
{ type: "key", key: "B", label: "Fallback: open inventory from world" },
{ type: "click-artifact-tab", label: "Fallback: select artifact inventory tab" },
{ type: "click-first-artifact", label: "Fallback: select first visible artifact" },
{ type: "capture-preflight", label: "Verify artifact inventory grid and detail card" },
]);
});
it("uses Inventory Kamera's artifact-tab coordinate ratio", () => {
expect(artifactTabClickTarget(capture())).toEqual({ x: 672, y: 47 });
});
it("blocks invalid lookup and unsupported layouts before auto-scan", () => {
expect(validateAutoScanEntryPreflight(capture(), { valid: false, errors: ["bad"], warnings: [], summary: { artifactSets: 0, artifactPieces: 0, characters: 0, stats: 0, generatedAt: "", sourceVersion: "" } }).reason).toContain("Lookup invalid");
expect(validateAutoScanEntryPreflight(capture({ layout: { aspect: "2.39:1", isSixteenNine: false, warning: "nicht 16:9" } })).reason).toContain("nicht 16:9");
});
it("blocks primary-screen and missing detail-card starts", () => {
expect(validateAutoScanEntryPreflight(capture({ captureTarget: "primary-screen" })).reason).toContain("Primary Screen");
expect(validateAutoScanEntryPreflight(capture({ artifactDetail: { present: false, confidence: 12, orangeHits: 0, greenHits: 1, textHits: 4 } })).reason).toContain("Keine Artifact-Detailansicht");
});
it("blocks the Paimon menu before grid clicks or OCR", () => {
expect(validateAutoScanEntryPreflight(capture({
artifactDetail: {
present: false,
confidence: 8,
orangeHits: 12,
greenHits: 154,
textHits: 20,
titleOrangeHits: 4,
upperTextHits: 11,
lowerGreenHits: 0,
},
paimonMenu: {
present: true,
confidence: 81,
profileLightPct: 56,
profileCreamPct: 31.1,
menuTileDarkPct: 67.7,
},
})).reason).toContain("Paimon-Menue erkannt");
});
it("detects blocked key input", () => {
expect(keyPressBlocked({ ok: false, key: "B", inputBlocked: true })).toBe(true);
expect(keyPressBlocked({ ok: true, key: "B", inputBlocked: false })).toBe(false);
});
});
src/lib/autoScanEntry.ts
+89
View File
@@ -0,0 +1,89 @@
import type { CaptureResult, KeyPressResult } from "../types/global";
import { validateLookupPackage, type LookupValidationStatus } from "./genshinLookup";
export type ScanEntryMode = "visible-inventory" | "direct-inventory" | "paimon-menu" | "auto-entry";
export interface AutoScanEntryAction {
type: "key" | "click-artifact-tab" | "click-first-artifact" | "capture-preflight";
key?: "ESC" | "B";
label: string;
}
export interface AutoScanEntryPreflight {
ok: boolean;
reason: string;
}
export function buildAutoScanEntryPlan(mode: ScanEntryMode): AutoScanEntryAction[] {
if (mode === "visible-inventory") {
return [{ type: "capture-preflight", label: "Capture visible artifact inventory" }];
}
if (mode === "direct-inventory") {
return [
{ type: "key", key: "B", label: "Open inventory from the current world state" },
{ type: "click-artifact-tab", label: "Select artifact inventory tab" },
{ type: "click-first-artifact", label: "Select first visible artifact so the detail card is open" },
{ type: "capture-preflight", label: "Verify artifact inventory grid and detail card" },
];
}
if (mode === "auto-entry") {
return [
{ type: "key", key: "B", label: "Try direct inventory entry from world" },
{ type: "click-artifact-tab", label: "Select artifact inventory tab if inventory opened" },
{ type: "click-first-artifact", label: "Select first visible artifact so the detail card is open" },
{ type: "key", key: "ESC", label: "Fallback: Inventory Kamera step from already-open Paimon menu" },
{ type: "key", key: "ESC", label: "Fallback: close Paimon menu if ESC opened or left it visible" },
{ type: "key", key: "B", label: "Fallback: open inventory from world" },
{ type: "click-artifact-tab", label: "Fallback: select artifact inventory tab" },
{ type: "click-first-artifact", label: "Fallback: select first visible artifact" },
{ type: "capture-preflight", label: "Verify artifact inventory grid and detail card" },
];
}
return [
{ type: "key", key: "ESC", label: "Inventory Kamera step: leave the already-open Paimon menu" },
{ type: "key", key: "ESC", label: "If Paimon is still visible, close it before opening inventory" },
{ type: "key", key: "B", label: "Inventory Kamera step: open inventory from world" },
{ type: "click-artifact-tab", label: "Inventory Kamera step: select artifact inventory tab" },
{ type: "click-first-artifact", label: "Select first visible artifact so the detail card is open" },
{ type: "capture-preflight", label: "Verify artifact inventory grid and detail card" },
];
}
export function artifactTabClickTarget(capture: CaptureResult): { x: number; y: number } {
return {
x: Math.round(capture.width * (448 / 1280)),
y: Math.round(capture.height * (31 / 720)),
};
}
export function keyPressBlocked(result: KeyPressResult | null | undefined) {
return !result?.ok || Boolean(result.inputBlocked);
}
export function validateAutoScanEntryPreflight(
capture: CaptureResult | null,
lookupStatus: LookupValidationStatus = validateLookupPackage(),
): AutoScanEntryPreflight {
if (!lookupStatus.valid) {
return {
ok: false,
reason: `Lookup invalid: ${lookupStatus.errors[0] ?? "unknown lookup error"}`,
};
}
if (!capture) return { ok: false, reason: "Keine Capture-Daten fuer den Auto-Scan-Start." };
if (capture.captureTarget === "primary-screen") {
return { ok: false, reason: "Auto-Scan blockiert: Capture stammt vom Primary Screen statt vom Genshin-Client." };
}
if (capture.layout?.warning) return { ok: false, reason: capture.layout.warning };
if (capture.paimonMenu?.present) {
return { ok: false, reason: `Paimon-Menue erkannt (${capture.paimonMenu.confidence}%). Auto-Scan startet erst im Artifact-Inventar mit sichtbarer Detailkarte.` };
}
if (!capture.inventoryGrid || capture.inventoryGrid.source === "missing" || capture.inventoryGrid.centers.length === 0) {
return { ok: false, reason: "Kein verlaessliches Artifact-Grid erkannt. Artifact-Inventar sichtbar lassen." };
}
if (!capture.artifactDetail?.present) {
const confidence = capture.artifactDetail ? ` (${capture.artifactDetail.confidence}% Detail-Marker)` : "";
return { ok: false, reason: `Keine Artifact-Detailansicht erkannt${confidence}. Artifact-Inventar mit sichtbarer Detailkarte oeffnen.` };
}
return { ok: true, reason: "" };
}
src/lib/autoScanLoop.test.ts
+460 -2
View File
@@ -50,6 +50,7 @@ function capture(overrides: Partial<ScanTestCapture> = {}): ScanTestCapture {
detailDataUrl: "data:image/png;base64,DETAIL-AAA",
inventoryDataUrl: "data:image/png;base64,GRID-AAA",
inventoryGrid: sampleGrid,
artifactDetail: { present: true, confidence: 84, orangeHits: 12, greenHits: 8, textHits: 28 },
...overrides,
};
}
@@ -81,6 +82,11 @@ describe("autoScanLoop fingerprints", () => {
expect(detailFingerprint(captureA as never)).toBe(detailFingerprint(captureB as never));
});
it("prefers native detail fingerprints when preview images are omitted", () => {
expect(detailFingerprint({ detailFingerprint: "native-detail", dataUrl: "data:image/png;base64,FULL" } as never)).toBe("native-detail");
expect(screenFingerprint({ inventoryFingerprint: "native-inventory", dataUrl: "data:image/png;base64,FULL" } as never)).toBe("native-inventory");
});
it("uses the inventory preview for scroll verification when available", () => {
const captureA = {
inventoryDataUrl: "data:image/png;base64," + "GRID-A".repeat(64),
@@ -101,7 +107,7 @@ describe("autoScanLoop fingerprints", () => {
expect(isRepeatedProcessedPageFingerprint("", seen, 3)).toBe(false);
});
it("does not block before first click when start capture is from the primary screen", async () => {
it("blocks before first click when start capture is from the primary screen", async () => {
const startCapture = capture({
captureTarget: "primary-screen",
detailDataUrl: `data:image/png;base64,${"D".repeat(600)}`,
@@ -127,7 +133,7 @@ describe("autoScanLoop fingerprints", () => {
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => capture({ detailDataUrl: `data:image/png;base64,${"E".repeat(600)}` }),
captureFastSelectedSource: async () => startCapture,
captureFastSelectedSource: async () => (clicked > 0 ? capture({ detailDataUrl: `data:image/png;base64,${"E".repeat(600)}` }) : startCapture),
parseArtifact: () => sampleParse,
persistParsedArtifact: async () => false,
saveReviewSample: async () => ({ ok: true }),
@@ -141,8 +147,460 @@ describe("autoScanLoop fingerprints", () => {
};
const result = await runAutoScanLoop(deps, { scanLimit: 1, skipRows: 0, detectedInventoryCount: null });
expect(result.blockedReason).toContain("Primary Screen");
expect(result.status).toBe("blocked");
expect(clicked).toBe(0);
});
it("blocks before first click when no artifact detail card is visible", async () => {
let clicked = 0;
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async () => {
clicked += 1;
return {
ok: true,
x: 0,
y: 0,
clicked: true,
moved: true,
focused: true,
};
},
scrollScreen: async () => ({ ok: true, notchesSent: 0 }),
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => capture(),
captureFastSelectedSource: async () => capture({ artifactDetail: { present: false, confidence: 10, orangeHits: 0, greenHits: 0, textHits: 3 } }),
parseArtifact: () => sampleParse,
persistParsedArtifact: async () => false,
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: () => undefined,
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
const result = await runAutoScanLoop(deps, { scanLimit: 1, skipRows: 0, detectedInventoryCount: null, ocrEngine: "ik-traineddata" });
expect(result.blockedReason).toContain("Keine Artifact-Detailansicht");
expect(result.status).toBe("blocked");
expect(clicked).toBe(0);
});
it("continues when detail verification proves a helper-reported click miss still changed selection", async () => {
const startCapture = capture({
detailDataUrl: `data:image/png;base64,${"D".repeat(600)}`,
});
let clicked = 0;
const selectedCaptureOptions: unknown[] = [];
const selectedFocusFlags: boolean[] = [];
const fastFocusFlags: boolean[] = [];
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async () => {
clicked += 1;
if (clicked === 1) {
return {
ok: true,
x: 80,
y: 90,
cursorX: 960,
cursorY: 539,
clicked: false,
moved: false,
focused: true,
isElevated: true,
};
}
return {
ok: true,
x: 80,
y: 90,
cursorX: 80,
cursorY: 90,
clicked: true,
moved: true,
focused: true,
isElevated: true,
};
},
scrollScreen: async () => ({ ok: true, notchesSent: 0 }),
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async (_delayMs, focusGenshin, options) => {
selectedFocusFlags.push(Boolean(focusGenshin));
selectedCaptureOptions.push(options);
return capture({ detailDataUrl: `data:image/png;base64,${"E".repeat(600)}` });
},
captureFastSelectedSource: async (_delayMs, focusGenshin) => {
fastFocusFlags.push(Boolean(focusGenshin));
return clicked > 0 ? capture({ detailDataUrl: `data:image/png;base64,${"E".repeat(600)}` }) : startCapture;
},
parseArtifact: () => sampleParse,
persistParsedArtifact: async () => true,
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: () => undefined,
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
const result = await runAutoScanLoop(deps, { scanLimit: 1, skipRows: 0, detectedInventoryCount: null, ocrEngine: "ik-traineddata" });
expect(result.blockedReason).toBe("");
expect(result.status).toBe("done");
expect(clicked).toBe(1);
expect(result.stats.verified).toBe(1);
expect(result.stats.parsed).toBe(1);
expect(selectedFocusFlags).toEqual([false]);
expect(fastFocusFlags.every((flag) => flag === false)).toBe(true);
expect(selectedCaptureOptions).toContainEqual({
ocrMode: "artifact",
ocrProfile: "fast",
ocrEngine: "ik-traineddata",
omitFullFrame: true,
omitInventoryPreview: true,
omitCropImages: true,
omitEquippedOcr: true,
skipOcrUnlessArtifactDetail: true,
});
});
it("processes an initial Paimon-selected artifact before clicking the next tile", async () => {
let clicked = 0;
let persisted = 0;
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async () => {
clicked += 1;
return {
ok: true,
x: 80,
y: 90,
cursorX: 80,
cursorY: 90,
clicked: true,
moved: true,
focused: true,
};
},
scrollScreen: async () => ({ ok: true, notchesSent: 0 }),
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => capture({ detailDataUrl: `data:image/png;base64,${"I".repeat(600)}` }),
captureFastSelectedSource: async () => capture({ detailDataUrl: `data:image/png;base64,${"I".repeat(600)}` }),
parseArtifact: () => sampleParse,
persistParsedArtifact: async () => {
persisted += 1;
return true;
},
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: () => undefined,
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
const result = await runAutoScanLoop(deps, {
scanLimit: 1,
skipRows: 0,
detectedInventoryCount: null,
processInitialSelection: true,
});
expect(result.status).toBe("done");
expect(result.stats.parsed).toBe(1);
expect(result.stats.stored).toBe(1);
expect(clicked).toBe(0);
expect(persisted).toBe(1);
});
it("starts after the already selected initial tile", async () => {
let clicked = 0;
const clickedTargets: Array<{ x: number; y: number }> = [];
const logs: string[] = [];
const twoTileGrid = {
centers: [
{ x: 179, y: 254, row: 0, col: 0 },
{ x: 325, y: 254, row: 0, col: 1 },
],
rows: 1,
cols: 2,
confidence: 86,
source: "detected" as const,
};
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async (x, y) => {
clicked += 1;
clickedTargets.push({ x, y });
return {
ok: true,
x: 80,
y: 90,
cursorX: 80,
cursorY: 90,
clicked: true,
moved: true,
focused: true,
};
},
scrollScreen: async () => ({ ok: true, notchesSent: 0 }),
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => capture({ inventoryGrid: twoTileGrid, detailDataUrl: `data:image/png;base64,${"I".repeat(600)}` }),
captureFastSelectedSource: async () => capture({ inventoryGrid: twoTileGrid, detailDataUrl: `data:image/png;base64,${"I".repeat(600)}` }),
parseArtifact: () => sampleParse,
persistParsedArtifact: async () => true,
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: (line) => logs.push(line),
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
await runAutoScanLoop(deps, {
scanLimit: 2,
skipRows: 0,
detectedInventoryCount: null,
processInitialSelection: true,
skipInitialGridTarget: true,
});
expect(clicked).toBe(1);
expect(clickedTargets[0]).toEqual({ x: 325, y: 254 });
expect(logs.some((line) => line.includes("r0 c0"))).toBe(false);
});
it("does not skip the first grid tile when the visible inventory selection was user-made", async () => {
let clicked = 0;
const clickedTargets: Array<{ x: number; y: number }> = [];
const twoTileGrid = {
centers: [
{ x: 179, y: 254, row: 0, col: 0 },
{ x: 325, y: 254, row: 0, col: 1 },
],
rows: 1,
cols: 2,
confidence: 86,
source: "detected" as const,
};
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async (x, y) => {
clicked += 1;
clickedTargets.push({ x, y });
return {
ok: true,
x,
y,
cursorX: x,
cursorY: y,
clicked: true,
moved: true,
focused: true,
};
},
scrollScreen: async () => ({ ok: true, notchesSent: 0 }),
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => capture({ inventoryGrid: twoTileGrid, detailDataUrl: `data:image/png;base64,${"A".repeat(600)}` }),
captureFastSelectedSource: async () => (clicked > 0
? capture({ inventoryGrid: twoTileGrid, detailDataUrl: `data:image/png;base64,${"A".repeat(600)}` })
: capture({ inventoryGrid: twoTileGrid, detailDataUrl: `data:image/png;base64,${"I".repeat(600)}` })),
parseArtifact: () => sampleParse,
persistParsedArtifact: async () => true,
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: () => undefined,
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
await runAutoScanLoop(deps, {
scanLimit: 2,
skipRows: 0,
detectedInventoryCount: null,
processInitialSelection: true,
skipInitialGridTarget: false,
});
expect(clickedTargets[0]).toEqual({ x: 179, y: 254 });
});
it("polls for the next inventory page after scroll instead of sleeping a fixed settle delay", async () => {
const started = Date.now();
let clicked = 0;
let scrolled = false;
let parsedIndex = 0;
const waits: string[] = [];
const singleTileGrid = {
centers: [{ x: 179, y: 254, row: 0, col: 0 }],
rows: 1,
cols: 1,
confidence: 86,
source: "detected" as const,
};
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async (x, y) => {
clicked += 1;
return {
ok: true,
x,
y,
cursorX: x,
cursorY: y,
clicked: true,
moved: true,
focused: true,
};
},
scrollScreen: async () => {
scrolled = true;
waits.push("scroll");
return { ok: true, notchesSent: -9 };
},
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => capture({
inventoryGrid: singleTileGrid,
inventoryFingerprint: scrolled ? "page-2" : "page-1",
detailFingerprint: clicked > 1 ? "detail-2-selected" : "detail-1-selected",
}),
captureFastSelectedSource: async () => {
const pageFingerprint = scrolled ? "page-2" : "page-1";
const detail = clicked > 1 ? "detail-2" : clicked > 0 ? "detail-1" : "detail-start";
return capture({
inventoryGrid: singleTileGrid,
inventoryFingerprint: pageFingerprint,
detailFingerprint: `${detail}:${pageFingerprint}`,
});
},
parseArtifact: () => ({
...sampleParse,
name: `Artifact ${++parsedIndex}`,
fields: {
...sampleParse.fields,
name: { value: `Artifact ${parsedIndex}`, confidence: 84, source: "ocr" as const },
},
}),
persistParsedArtifact: async () => true,
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: () => undefined,
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
const result = await runAutoScanLoop(deps, { scanLimit: 2, skipRows: 0, detectedInventoryCount: null });
expect(result.status).toBe("done");
expect(result.stats.parsed).toBe(2);
expect(waits).toEqual(["scroll"]);
expect(Date.now() - started).toBeLessThan(700);
});
it("queues store writes so the next tile can be clicked before persistence finishes", async () => {
let clicked = 0;
let selectedReads = 0;
const persistResolvers: Array<(value: boolean) => void> = [];
const twoTileGrid = {
centers: [
{ x: 179, y: 254, row: 0, col: 0 },
{ x: 325, y: 254, row: 0, col: 1 },
],
rows: 1,
cols: 2,
confidence: 86,
source: "detected" as const,
};
const deps: AutoScanLoopDependencies = {
api: {
clickScreen: async (x, y) => {
clicked += 1;
return {
ok: true,
x,
y,
cursorX: x,
cursorY: y,
clicked: true,
moved: true,
focused: true,
};
},
scrollScreen: async () => ({ ok: true, notchesSent: 0 }),
getAutomationGuard: async () => ({ ok: true, escapePressed: false, enterPressed: false, f9Pressed: false }),
},
captureSelectedSource: async () => {
selectedReads += 1;
return capture({
inventoryGrid: twoTileGrid,
detailFingerprint: `detail-read-${selectedReads}`,
});
},
captureFastSelectedSource: async () => capture({
inventoryGrid: twoTileGrid,
detailFingerprint: clicked === 0 ? "detail-start" : `detail-click-${clicked}`,
}),
parseArtifact: () => ({
...sampleParse,
name: `Artifact ${selectedReads}`,
fields: {
...sampleParse.fields,
name: { value: `Artifact ${selectedReads}`, confidence: 84, source: "ocr" as const },
},
}),
persistParsedArtifact: async () => new Promise<boolean>((resolve) => {
persistResolvers.push(resolve);
}),
saveReviewSample: async () => ({ ok: true }),
getAutoReviewReason: () => "",
shouldFlagArtifactForReview: () => false,
appendAutomationLog: () => undefined,
appendClickDiagnostics: () => undefined,
setReviewStatus: () => undefined,
setAutoScanStats: () => undefined,
shouldStop: () => false,
};
const runPromise = runAutoScanLoop(deps, { scanLimit: 2, skipRows: 0, detectedInventoryCount: null });
await new Promise((resolve) => setTimeout(resolve, 250));
expect(clicked).toBe(2);
expect(persistResolvers).toHaveLength(1);
persistResolvers[0](true);
for (let attempt = 0; attempt < 10 && persistResolvers.length < 2; attempt += 1) {
await new Promise((resolve) => setTimeout(resolve, 20));
}
expect(persistResolvers).toHaveLength(2);
persistResolvers[1](true);
const result = await runPromise;
expect(result.status).toBe("done");
expect(result.stats.parsed).toBe(2);
expect(result.stats.stored).toBe(2);
expect(result.stats.activeScanMs).toBeGreaterThan(0);
expect(result.stats.writeFlushMs).toBeGreaterThan(0);
expect(result.stats.activeScanMs).toBeLessThanOrEqual(result.stats.elapsedMs);
});
});
src/lib/autoScanLoop.ts
+310 -53
View File
@@ -1,4 +1,4 @@
import type { BooleanResult, CaptureResult, ClickResult, AutomationGuard, ScrollResult } from "../types/global";
import type { BooleanResult, CaptureOptions, CaptureResult, ClickResult, AutomationGuard, ScrollResult } from "../types/global";
import type { ParsedArtifactCandidate } from "./artifactOcrParser";
import { sessionSignature } from "./artifactStore";
import { buildGridModel, buildInventoryPagePlan, type GridTarget } from "./automationPlanner";
@@ -6,7 +6,8 @@ import { classifyAutoScanCapture, shouldAbortAfterConsecutiveMisses } from "./au
import { waitForCardReady } from "./cardReadyGate";
import { captureRejectionReason, captureSourceRejectionReason, shouldPersistParsedArtifact } from "./scannerCaptureQuality";
import type { AutoScanStats, ScanSummary } from "./scannerSession";
import { clampSkipRows, emptyAutoScanStats, resolveScanTargetCount } from "./scannerSession";
import { addCaptureTiming, addCardReadyTiming, addScrollReadyTiming, clampSkipRows, emptyAutoScanStats, resolveScanTargetCount, updateScanTiming } from "./scannerSession";
import { validateAutoScanEntryPreflight } from "./autoScanEntry";
// Simplified to match Inventory Kamera's proven approach (see docs/DECISIONS.md
// ADR-007): one click per tile, a fixed settle delay, one retry if the detail
@@ -22,8 +23,8 @@ type AutoScanApi = {
export type AutoScanLoopDependencies = {
api: AutoScanApi;
captureSelectedSource: (delayMs?: number, focusGenshin?: boolean) => Promise<CaptureResult | null>;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean) => Promise<CaptureResult | null>;
captureSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
captureFastSelectedSource: (delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult | null>;
parseArtifact: (capture: CaptureResult | null) => ParsedArtifactCandidate | null;
persistParsedArtifact: (
capture: CaptureResult | null,
@@ -49,6 +50,9 @@ export type AutoScanLoopOptions = {
scanLimit: number;
skipRows: number;
detectedInventoryCount?: number | null;
processInitialSelection?: boolean;
skipInitialGridTarget?: boolean;
ocrEngine?: CaptureOptions["ocrEngine"];
};
export type AutoScanLoopResult = {
@@ -62,9 +66,14 @@ export type AutoScanLoopResult = {
// Card-ready gating replaces a fixed settle delay: poll the detail fingerprint
// until it has changed and stabilized (or the budget is spent). See cardReadyGate.
const CARD_READY_MAX_MS = 900;
const CARD_READY_POLL_MS = 90;
const CARD_READY_MAX_MS = 420;
const CARD_READY_POLL_MS = 60;
const CARD_READY_STABLE_SAMPLES = 2;
const CARD_READY_ACCEPT_CHANGED_AFTER_MS = 200;
const SCROLL_READY_MAX_MS = 760;
const SCROLL_READY_POLL_MS = 80;
const SCROLL_READY_STABLE_SAMPLES = 2;
const SCROLL_READY_ACCEPT_CHANGED_AFTER_MS = 100;
const MISS_ABORT_THRESHOLD = 3;
const UNREADABLE_ABORT_THRESHOLD = 5;
@@ -89,28 +98,66 @@ export async function runAutoScanLoop(
} = deps;
const stats: AutoScanStats = { ...emptyAutoScanStats };
const startedAt = Date.now();
const maxTargets = resolveScanTargetCount(options.scanLimit, options.detectedInventoryCount);
const rowsToSkip = clampSkipRows(options.skipRows);
const seen = new Set<string>();
const seenDetailFingerprints = new Set<string>();
const seenPageFingerprints = new Set<string>();
let page = 0;
let blockedReason = "";
let aborted = false;
let consecutiveMisses = 0;
let rowsQueued = 0;
const primaryScreenStartWarning =
"Start-Capture ist vom Primary-Screen, kein spezifischer Genshin-Client-Marker vorhanden - Auto-Scan wird mit Vorsicht fortgesetzt.";
function updateStats() {
let writeQueue: Promise<void> = Promise.resolve();
function updateStats(preserveActiveScanMs = false) {
updateScanTiming(stats, startedAt, Date.now(), { preserveActiveScanMs });
setAutoScanStats({ ...stats });
}
async function saveAutomaticReviewSample(capture: CaptureResult | null, parsed: ParsedArtifactCandidate | null, reason: string) {
function enqueueWrite(label: string, task: () => Promise<void>) {
writeQueue = writeQueue
.catch(() => undefined)
.then(async () => {
try {
await task();
} catch (error) {
appendAutomationLog(`write failed ${label}: ${error instanceof Error ? error.message : String(error)}`);
}
});
}
async function flushWrites() {
await writeQueue.catch(() => undefined);
updateStats(true);
}
async function finish(result: AutoScanLoopResult) {
const flushStartedAt = Date.now();
stats.activeScanMs = Math.max(0, flushStartedAt - startedAt);
await flushWrites();
stats.writeFlushMs += Math.max(0, Date.now() - flushStartedAt);
updateStats(true);
return { ...result, stats: { ...stats } };
}
function saveAutomaticReviewSample(capture: CaptureResult | null, parsed: ParsedArtifactCandidate | null, reason: string) {
enqueueWrite(`review:${reason}`, async () => {
const saved = await saveReviewSample(capture, parsed, reason);
if (saved?.ok) {
stats.review++;
updateStats();
}
});
}
function persistArtifactLater(capture: CaptureResult | null, parsed: ParsedArtifactCandidate, source: string, needsReview: boolean) {
enqueueWrite(`persist:${source}:${parsed.name}`, async () => {
if (await persistParsedArtifact(capture, parsed, source, needsReview)) {
stats.stored++;
updateStats();
}
});
}
async function checkGuard() {
@@ -154,41 +201,183 @@ export async function runAutoScanLoop(
return clickResult;
}
let currentCapture = await captureSelectedSource(0, true);
const isPrimaryCapture = currentCapture?.captureTarget === "primary-screen";
const initialCaptureRejection = isPrimaryCapture ? "" : captureSourceRejectionReason(currentCapture);
function reportedClickDeliveryFailure(result: ClickResult) {
return result.moved === false || result.clicked === false;
}
function clickDeliveryFailureReason(target: GridTarget, clickResult: ClickResult) {
if (clickResult.inputBlocked) {
return "Windows blockiert die Eingabe (UIPI). Starte die App als Administrator (Scanner Diagnose > 'App als Administrator neu starten').";
}
if (clickResult.isElevated === false) {
return `Klick kam nicht an (Ziel ${target.x},${target.y}). Starte die App als Administrator und versuche es erneut.`;
}
const cursor = `${clickResult.cursorX ?? "?"},${clickResult.cursorY ?? "?"}`;
return `Cursor kam nicht am Klick-Ziel ${target.x},${target.y} an (Cursor ${cursor}). Genshin im Vordergrund lassen und erneut versuchen.`;
}
let currentCapture = await captureFastSelectedSource(0, false, {
omitFullFrame: true,
omitDetailPreview: true,
omitInventoryPreview: true,
omitCrops: true,
omitCropImages: true,
omitLockState: true,
});
const initialSurfaceRejection = validateAutoScanEntryPreflight(currentCapture);
const initialCaptureRejection = initialSurfaceRejection.ok ? captureSourceRejectionReason(currentCapture) : initialSurfaceRejection.reason;
let gridModel = buildGridModel(currentCapture?.inventoryGrid);
if (initialCaptureRejection || !gridModel || gridModel.targets.length === 0) {
const reason = initialCaptureRejection || "Kein verlaessliches Kachel-Grid erkannt. Artifact-Inventar sichtbar lassen und Smart Capture einmal ausfuehren.";
setReviewStatus(reason);
return { status: "blocked", stats, blockedReason: reason, pageCount: 0, gridLabel: reason, targetCount: maxTargets };
}
if (isPrimaryCapture) {
appendAutomationLog(primaryScreenStartWarning);
return finish({ status: "blocked", stats, blockedReason: reason, pageCount: 0, gridLabel: reason, targetCount: maxTargets });
}
let lastDetailSignature = "";
const initialParsed = parseArtifact(currentCapture);
if (initialParsed) lastDetailSignature = sessionSignature(initialParsed);
let lastDetailViewFingerprint = detailFingerprint(currentCapture);
if (lastDetailViewFingerprint) seenDetailFingerprints.add(lastDetailViewFingerprint);
const shouldSkipInitialGridTarget = Boolean(options.processInitialSelection && options.skipInitialGridTarget);
let initialProcessedOffset = 0;
let initialSelectionDuplicateSkipped = false;
if (options.processInitialSelection) {
const initialCapture = await captureSelectedSource(0, false, {
ocrMode: "artifact",
ocrProfile: "fast",
...(options.ocrEngine ? { ocrEngine: options.ocrEngine } : {}),
omitFullFrame: true,
omitInventoryPreview: true,
omitCropImages: true,
omitEquippedOcr: true,
skipOcrUnlessArtifactDetail: true,
});
const initialSurfaceRejection = validateAutoScanEntryPreflight(initialCapture);
if (!initialSurfaceRejection.ok) {
return finish({
status: "blocked",
stats,
blockedReason: initialSurfaceRejection.reason,
pageCount: 0,
gridLabel: initialSurfaceRejection.reason,
targetCount: maxTargets,
});
}
if (!initialCapture) {
return finish({
status: "blocked",
stats,
blockedReason: "Keine Capture-Daten fuer die initiale Artifact-Auswahl.",
pageCount: 0,
gridLabel: "Keine Capture-Daten fuer die initiale Artifact-Auswahl.",
targetCount: maxTargets,
});
}
const parsed = parseArtifact(initialCapture);
const rejection = captureRejectionReason(initialCapture, parsed);
if (rejection || !parsed) {
saveAutomaticReviewSample(initialCapture, parsed, `automatic:initial-selection-rejected`);
stats.verified++;
stats.misses++;
addCaptureTiming(stats, initialCapture.timings);
initialProcessedOffset = shouldSkipInitialGridTarget ? 1 : 0;
lastDetailViewFingerprint = detailFingerprint(initialCapture);
updateStats();
appendAutomationLog(`initial selection review: ${rejection || "kein Artifact lesbar"}; scan continues with next tile`);
} else {
stats.verified++;
stats.parsed++;
addCaptureTiming(stats, initialCapture.timings);
initialProcessedOffset = shouldSkipInitialGridTarget ? 1 : 0;
const signature = sessionSignature(parsed);
seen.add(signature);
lastDetailSignature = signature;
lastDetailViewFingerprint = detailFingerprint(initialCapture);
const reason = getAutoReviewReason(initialCapture, parsed);
const needsReview = reason ? true : shouldFlagArtifactForReview(parsed);
if (reason) saveAutomaticReviewSample(initialCapture, parsed, `automatic:${reason}:initial-selection`);
persistArtifactLater(initialCapture, parsed, "auto-scan-initial", needsReview);
updateStats();
appendAutomationLog(`initial selection parsed: ${parsed.name}`);
if (stats.parsed >= maxTargets) {
return finish({
status: "done",
stats,
blockedReason: "",
pageCount: 1,
gridLabel: `Initial ausgewaehltes Artifact verarbeitet, Ziel ${maxTargets} Artifacts`,
targetCount: maxTargets,
});
}
}
}
async function awaitCardReady() {
return waitForCardReady(
const startedAt = Date.now();
const result = await waitForCardReady(
{
sampleFingerprint: async () => detailFingerprint(await captureFastSelectedSource(0, true)),
sampleFingerprint: async () => detailFingerprint(await captureFastSelectedSource(0, false, {
omitFullFrame: true,
omitDetailPreview: true,
omitInventoryPreview: true,
omitCrops: true,
omitCropImages: true,
omitLockState: true,
})),
wait,
now: () => Date.now(),
checkAbort: checkGuard,
},
lastDetailViewFingerprint,
{ minStableSamples: CARD_READY_STABLE_SAMPLES, maxWaitMs: CARD_READY_MAX_MS, pollIntervalMs: CARD_READY_POLL_MS },
{
minStableSamples: CARD_READY_STABLE_SAMPLES,
maxWaitMs: CARD_READY_MAX_MS,
pollIntervalMs: CARD_READY_POLL_MS,
acceptChangedAfterMs: CARD_READY_ACCEPT_CHANGED_AFTER_MS,
},
);
addCardReadyTiming(stats, Date.now() - startedAt);
return result;
}
async function awaitInventoryPageReady(previousFingerprint: string): Promise<{
ready: Awaited<ReturnType<typeof waitForCardReady>>;
capture: CaptureResult | null;
}> {
let latestCapture: CaptureResult | null = null;
const startedAt = Date.now();
const ready = await waitForCardReady(
{
sampleFingerprint: async () => {
latestCapture = await captureFastSelectedSource(0, false, {
omitFullFrame: true,
omitDetailPreview: true,
omitInventoryPreview: true,
omitCrops: true,
omitCropImages: true,
omitLockState: true,
});
return screenFingerprint(latestCapture);
},
wait,
now: () => Date.now(),
checkAbort: checkGuard,
},
previousFingerprint,
{
minStableSamples: SCROLL_READY_STABLE_SAMPLES,
maxWaitMs: SCROLL_READY_MAX_MS,
pollIntervalMs: SCROLL_READY_POLL_MS,
acceptChangedAfterMs: SCROLL_READY_ACCEPT_CHANGED_AFTER_MS,
},
);
addScrollReadyTiming(stats, Date.now() - startedAt);
return { ready, capture: latestCapture };
}
try {
while (!blockedReason && !shouldStop() && stats.clicked < maxTargets) {
while (!blockedReason && !shouldStop() && stats.parsed < maxTargets) {
page++;
stats.pages = page;
updateStats();
@@ -207,21 +396,23 @@ export async function runAutoScanLoop(
cols: gridModel.cols,
rows: Math.max(1, gridModel.rows - pageSkipRows),
totalTargetCount: maxTargets,
processedTargets: stats.clicked,
processedTargets: stats.clicked + initialProcessedOffset,
rowsQueued,
});
const targets = pagePlan.pageTargets;
const targets = shouldSkipInitialGridTarget && page === 1
? baseTargets.slice(initialProcessedOffset)
: pagePlan.pageTargets;
if (targets.length === 0) {
blockedReason = `Keine Klick-Ziele nach dem Skippen von ${pageSkipRows} Zeile(n) auf Seite ${page}.`;
break;
}
setReviewStatus(`Automatischer Scan Seite ${page}: ${gridModel.cols} x ${gridModel.rows} Raster (${gridModel.source}, ${gridModel.confidence}%), ${targets.length} Klick-Ziele, ${stats.clicked}/${maxTargets} geklickt.`);
setReviewStatus(`Automatischer Scan Seite ${page}: ${gridModel.cols} x ${gridModel.rows} Raster (${gridModel.source}, ${gridModel.confidence}%), ${targets.length} Klick-Ziele, ${stats.parsed}/${maxTargets} gelesen.`);
let newArtifactsOnPage = 0;
for (const target of targets) {
if (shouldStop() || stats.clicked >= maxTargets) break;
if (shouldStop() || stats.parsed >= maxTargets) break;
const guardReason = await checkGuard();
if (guardReason) {
@@ -238,16 +429,13 @@ export async function runAutoScanLoop(
break;
}
if (clickResult.moved === false || clickResult.clicked === false) {
// A structural failure (cursor could not be placed, or SendInput
// was rejected outright) means clicks are not reaching Genshin at
// all - almost always an elevation mismatch. Abort immediately
// instead of clicking blindly through the rest of the inventory.
blockedReason = clickResult.inputBlocked
? "Windows blockiert die Eingabe (UIPI). Starte die App als Administrator (Scanner Diagnose > 'App als Administrator neu starten')."
: `Klick kam nicht an (Ziel ${target.x},${target.y}). Starte die App als Administrator und versuche es erneut.`;
if (clickResult.inputBlocked) {
blockedReason = clickDeliveryFailureReason(target, clickResult);
break;
}
if (reportedClickDeliveryFailure(clickResult)) {
appendAutomationLog(`warn r${target.row} c${target.col}: helper reported cursor/click miss; verifying detail change`);
}
let ready = await awaitCardReady();
if (ready.abortReason) {
@@ -258,6 +446,14 @@ export async function runAutoScanLoop(
let changedDetail = ready.changed;
if (!changedDetail) {
if (options.processInitialSelection && !initialSelectionDuplicateSkipped && !reportedClickDeliveryFailure(clickResult)) {
initialSelectionDuplicateSkipped = true;
consecutiveMisses = 0;
stats.duplicates++;
updateStats();
appendAutomationLog(`duplicate selected tile r${target.row} c${target.col}: already processed initial detail`);
continue;
}
appendAutomationLog(`retry r${target.row} c${target.col}: Detailansicht unveraendert`);
clickResult = await clickTarget(target, "retry");
stopReason = inputStopReason(clickResult);
@@ -266,6 +462,13 @@ export async function runAutoScanLoop(
aborted = true;
break;
}
if (clickResult.inputBlocked) {
blockedReason = clickDeliveryFailureReason(target, clickResult);
break;
}
if (reportedClickDeliveryFailure(clickResult)) {
appendAutomationLog(`warn r${target.row} c${target.col}: retry helper reported cursor/click miss; verifying detail change`);
}
ready = await awaitCardReady();
if (ready.abortReason) {
blockedReason = ready.abortReason;
@@ -276,6 +479,18 @@ export async function runAutoScanLoop(
}
if (!changedDetail) {
if (reportedClickDeliveryFailure(clickResult)) {
blockedReason = clickDeliveryFailureReason(target, clickResult);
break;
}
if (options.processInitialSelection && !initialSelectionDuplicateSkipped) {
initialSelectionDuplicateSkipped = true;
consecutiveMisses = 0;
stats.duplicates++;
updateStats();
appendAutomationLog(`duplicate selected tile r${target.row} c${target.col}: already processed initial detail`);
continue;
}
stats.misses++;
consecutiveMisses++;
updateStats();
@@ -289,8 +504,36 @@ export async function runAutoScanLoop(
stats.verified++;
const capture = await captureSelectedSource(0, true);
if (ready.fingerprint) {
if (seenDetailFingerprints.has(ready.fingerprint)) {
consecutiveMisses = 0;
stats.duplicates++;
lastDetailViewFingerprint = ready.fingerprint;
updateStats();
appendAutomationLog(`duplicate visual r${target.row} c${target.col}: OCR uebersprungen`);
continue;
}
seenDetailFingerprints.add(ready.fingerprint);
}
const capture = await captureSelectedSource(0, false, {
ocrMode: "artifact",
ocrProfile: "fast",
...(options.ocrEngine ? { ocrEngine: options.ocrEngine } : {}),
omitFullFrame: true,
omitInventoryPreview: true,
omitCropImages: true,
omitEquippedOcr: true,
skipOcrUnlessArtifactDetail: true,
});
const captureSurfaceRejection = validateAutoScanEntryPreflight(capture);
if (!captureSurfaceRejection.ok) {
blockedReason = captureSurfaceRejection.reason;
appendAutomationLog(`blocked r${target.row} c${target.col}: ${blockedReason}`);
break;
}
if (capture?.ocrTimedOut) {
addCaptureTiming(stats, capture.timings);
stats.misses++;
consecutiveMisses++;
lastDetailViewFingerprint = detailFingerprint(capture);
@@ -305,9 +548,10 @@ export async function runAutoScanLoop(
const parsed = parseArtifact(capture);
const rejection = captureRejectionReason(capture, parsed);
addCaptureTiming(stats, capture?.timings);
if (rejection) {
await saveAutomaticReviewSample(capture, parsed, `automatic:capture-rejected:p${page}:r${target.row}c${target.col}`);
saveAutomaticReviewSample(capture, parsed, `automatic:capture-rejected:p${page}:r${target.row}c${target.col}`);
if (parsed && shouldPersistParsedArtifact(parsed, true)) {
consecutiveMisses = 0;
const signature = sessionSignature(parsed);
@@ -316,7 +560,7 @@ export async function runAutoScanLoop(
lastDetailViewFingerprint = detailFingerprint(capture);
seen.add(signature);
newArtifactsOnPage++;
if (await persistParsedArtifact(capture, parsed, "auto-scan-review", true)) stats.stored++;
persistArtifactLater(capture, parsed, "auto-scan-review", true);
updateStats();
continue;
}
@@ -375,9 +619,9 @@ export async function runAutoScanLoop(
const reason = getAutoReviewReason(capture, parsed);
const needsReview = reason ? true : shouldFlagArtifactForReview(parsed);
if (reason) {
await saveAutomaticReviewSample(capture, parsed, `automatic:${reason}:p${page}:r${target.row}c${target.col}`);
saveAutomaticReviewSample(capture, parsed, `automatic:${reason}:p${page}:r${target.row}c${target.col}`);
}
if (await persistParsedArtifact(capture, parsed, "auto-scan", needsReview)) stats.stored++;
persistArtifactLater(capture, parsed, "auto-scan", needsReview);
updateStats();
}
@@ -386,12 +630,12 @@ export async function runAutoScanLoop(
cols: gridModel.cols,
rows: Math.max(1, gridModel.rows - pageSkipRows),
totalTargetCount: maxTargets,
processedTargets: stats.clicked,
processedTargets: stats.clicked + initialProcessedOffset,
rowsQueued,
});
rowsQueued = endOfPagePlan.rowsQueuedAfterPage;
if (aborted || stats.clicked >= maxTargets || shouldStop() || blockedReason) break;
if (aborted || stats.parsed >= maxTargets || shouldStop() || blockedReason) break;
if (newArtifactsOnPage === 0 && page > 1) {
blockedReason = `Seite ${page} hat keine neuen Artifacts geliefert; gestoppt, um nicht dieselbe Seite zu loopen.`;
@@ -425,16 +669,26 @@ export async function runAutoScanLoop(
}
}
const scrollWaitStop = await waitDuringScan(760);
if (scrollWaitStop) {
blockedReason = scrollWaitStop;
const beforeScrollFingerprint = currentPageFingerprint || screenFingerprint(currentCapture);
const scrollReady = await awaitInventoryPageReady(beforeScrollFingerprint);
if (scrollReady.ready.abortReason) {
blockedReason = scrollReady.ready.abortReason;
aborted = true;
break;
}
const beforeScrollFingerprint = currentPageFingerprint || screenFingerprint(currentCapture);
currentCapture = await captureFastSelectedSource(0, true);
const afterScrollFingerprint = screenFingerprint(currentCapture);
const scrolledCapture = scrollReady.capture;
if (!scrolledCapture) {
blockedReason = "Keine Capture-Daten nach dem Scrollen.";
break;
}
currentCapture = scrolledCapture;
const scrolledSurfaceRejection = validateAutoScanEntryPreflight(scrolledCapture);
if (!scrolledSurfaceRejection.ok) {
blockedReason = scrolledSurfaceRejection.reason;
break;
}
const afterScrollFingerprint = screenFingerprint(scrolledCapture);
if (beforeScrollFingerprint && afterScrollFingerprint && beforeScrollFingerprint === afterScrollFingerprint) {
blockedReason = "Scrollen hat die sichtbare Inventarseite nicht veraendert.";
@@ -446,7 +700,7 @@ export async function runAutoScanLoop(
break;
}
const refreshedModel = buildGridModel(currentCapture?.inventoryGrid);
const refreshedModel = buildGridModel(scrolledCapture.inventoryGrid);
if (!refreshedModel) {
blockedReason = "Kachel-Grid nach dem Scrollen verloren.";
break;
@@ -461,24 +715,27 @@ export async function runAutoScanLoop(
}
const status: ScanSummary["status"] = aborted || shouldStop() ? "stopped" : blockedReason ? "blocked" : "done";
return {
updateScanTiming(stats, startedAt);
return finish({
status,
stats,
blockedReason,
pageCount: page,
gridLabel: blockedReason || `${page} Seite(n) verarbeitet, Ziel ${maxTargets} Artifacts, ${rowsToSkip} Zeile(n) auf der ersten Seite uebersprungen`,
targetCount: maxTargets,
};
});
}
export function detailFingerprint(capture: CaptureResult | null) {
if (!capture) return "";
if (capture.detailFingerprint) return capture.detailFingerprint;
if (capture.detailDataUrl) return fingerprintDataUrl(capture.detailDataUrl);
if (capture.dataUrl) return fingerprintDataUrl(capture.dataUrl);
return "";
}
export function screenFingerprint(capture: CaptureResult | null) {
if (capture?.inventoryFingerprint) return capture.inventoryFingerprint;
if (capture?.inventoryDataUrl) return fingerprintDataUrl(capture.inventoryDataUrl);
if (capture?.dataUrl) return fingerprintDataUrl(capture.dataUrl);
return "";
src/lib/automationPlanner.test.ts
+13 -13
View File
@@ -42,7 +42,7 @@ describe("automationPlanner", () => {
});
it("plans overlapping inventory pages like Inventory Kamera for a partial final page", () => {
const targets = Array.from({ length: 40 }, (_, index) => ({
const targets = Array.from({ length: 32 }, (_, index) => ({
row: Math.floor(index / 8),
col: index % 8,
x: index * 10,
@@ -52,31 +52,31 @@ describe("automationPlanner", () => {
const firstPage = buildInventoryPagePlan({
targets,
cols: 8,
rows: 5,
totalTargetCount: 50,
rows: 4,
totalTargetCount: 45,
processedTargets: 0,
rowsQueued: 0,
});
expect(firstPage.pageTargets).toHaveLength(40);
expect(firstPage.pageTargets).toHaveLength(32);
expect(firstPage.startIndex).toBe(0);
expect(firstPage.scrollRowsAfterPage).toBe(2);
const finalPage = buildInventoryPagePlan({
targets,
cols: 8,
rows: 5,
totalTargetCount: 50,
processedTargets: 40,
rowsQueued: 5,
rows: 4,
totalTargetCount: 45,
processedTargets: 32,
rowsQueued: 4,
});
expect(finalPage.pageTargets).toHaveLength(10);
expect(finalPage.startIndex).toBe(24);
expect(finalPage.pageTargets[0]).toMatchObject({ row: 3, col: 0 });
expect(finalPage.pageTargets).toHaveLength(13);
expect(finalPage.startIndex).toBe(16);
expect(finalPage.pageTargets[0]).toMatchObject({ row: 2, col: 0 });
expect(finalPage.scrollRowsAfterPage).toBe(0);
});
it("keeps the first page top-aligned when the whole inventory fits inside one visible page", () => {
const targets = Array.from({ length: 40 }, (_, index) => ({
const targets = Array.from({ length: 32 }, (_, index) => ({
row: Math.floor(index / 8),
col: index % 8,
x: index * 10,
@@ -86,7 +86,7 @@ describe("automationPlanner", () => {
const singlePage = buildInventoryPagePlan({
targets,
cols: 8,
rows: 5,
rows: 4,
totalTargetCount: 16,
processedTargets: 0,
rowsQueued: 0,
src/lib/cardReadyGate.test.ts
+15
View File
@@ -46,6 +46,21 @@ describe("waitForCardReady", () => {
expect(result.ready).toBe(true); // budget spent, but content did change
});
it("proceeds with changed animated content after the configured minimum elapsed time", async () => {
const animated = ["a1", "a2", "a3", "a4", "a5"];
const { deps } = harness(animated, 90);
const result = await waitForCardReady(deps, "old", {
minStableSamples: 2,
maxWaitMs: 900,
pollIntervalMs: 90,
acceptChangedAfterMs: 180,
});
expect(result.ready).toBe(true);
expect(result.changed).toBe(true);
expect(result.stable).toBe(false);
expect(result.polls).toBe(3);
});
it("reports not-ready when the detail never changes within budget", async () => {
const { deps } = harness(["old", "old", "old", "old", "old", "old"], 90);
const result = await waitForCardReady(deps, "old", { minStableSamples: 2, maxWaitMs: 200, pollIntervalMs: 90 });
src/lib/cardReadyGate.ts
+8 -3
View File
@@ -16,6 +16,8 @@ export interface CardReadyOptions {
maxWaitMs?: number;
/** Delay between samples. */
pollIntervalMs?: number;
/** Proceed with changed-but-animated content after this much elapsed time. */
acceptChangedAfterMs?: number;
}
export interface CardReadyDeps {
@@ -53,6 +55,7 @@ export async function waitForCardReady(
const minStable = Math.max(1, options.minStableSamples ?? DEFAULT_MIN_STABLE);
const maxWaitMs = Math.max(0, options.maxWaitMs ?? DEFAULT_MAX_WAIT_MS);
const pollIntervalMs = Math.max(1, options.pollIntervalMs ?? DEFAULT_POLL_MS);
const acceptChangedAfterMs = Math.max(0, options.acceptChangedAfterMs ?? maxWaitMs);
const start = deps.now();
let previousSample = "";
@@ -77,11 +80,13 @@ export async function waitForCardReady(
const changed = Boolean(latest) && latest !== previousFingerprint;
const stable = stableCount >= minStable;
if (changed && stable) {
return { ready: true, changed: true, stable: true, fingerprint: latest, abortReason: "", polls };
const elapsedMs = deps.now() - start;
if (changed && (stable || elapsedMs >= acceptChangedAfterMs)) {
return { ready: true, changed: true, stable, fingerprint: latest, abortReason: "", polls };
}
if (deps.now() - start >= maxWaitMs) {
if (elapsedMs >= maxWaitMs) {
// Budget spent. Proceed if the content has at least changed, even if it is
// still animating (never fully stabilizes).
return { ready: changed, changed, stable, fingerprint: latest, abortReason: "", polls };
src/lib/genshinData.ts
+21
View File
@@ -27,6 +27,27 @@ type GenshinGameDataContract = typeof gameData & {
pieceAliases?: Record<string, string>;
characterAliases?: Record<string, string>;
};
lookup?: {
normalizedKeys?: {
sets?: Record<string, string>;
pieces?: Record<string, string>;
slots?: Record<string, string>;
stats?: Record<string, string>;
characters?: Record<string, string>;
};
goodKeys?: {
sets?: Record<string, string>;
pieces?: Record<string, string>;
stats?: Record<string, string>;
characters?: Record<string, string>;
};
setToPieces?: Record<string, string[]>;
validation?: {
valid?: boolean;
errors?: string[];
warnings?: string[];
};
};
mainStatsBySlot?: Record<string, string[]>;
mainStatValueReferences?: Record<string, Array<{ stat: string; base: number; max: number }>>;
characters?: Character[];
src/lib/genshinLookup.test.ts
+48
View File
@@ -0,0 +1,48 @@
import { describe, expect, it } from "vitest";
import { matchPiece, matchSet, matchSlot, matchStat, validateLookupPackage } from "./genshinLookup";
import { genshinGameData } from "./genshinData";
describe("genshinLookup", () => {
it("matches canonical, alias, and fuzzy artifact values", () => {
expect(matchSlot("Sands of Eon Vi")).toMatchObject({ value: "Sands of Eon", source: "alias" });
expect(matchSet("Viridescent Venere")).toMatchObject({ value: "Viridescent Venerer", source: "alias" });
expect(matchPiece("Viridescent Venerers Determination").value).toBe("Viridescent Venerer's Determination");
expect(matchStat("Crit Damage")).toMatchObject({ value: "CRIT DMG", source: "alias" });
});
it("validates the bundled lookup package", () => {
const status = validateLookupPackage();
expect(status.valid).toBe(true);
expect(status.errors).toEqual([]);
expect(status.summary.artifactPieces).toBeGreaterThan(0);
});
it("rejects missing set references", () => {
const broken = {
...genshinGameData,
artifactPieces: [{ name: "Broken Piece", setName: "Missing Set", slot: "Flower of Life", relicType: "EQUIP_BRACER" }],
};
const status = validateLookupPackage(broken as unknown as typeof genshinGameData);
expect(status.valid).toBe(false);
expect(status.errors.join("\n")).toContain("Missing Set");
});
it("rejects duplicate GOOD set keys", () => {
const broken = {
...genshinGameData,
lookup: {
...genshinGameData.lookup,
goodKeys: {
...genshinGameData.lookup?.goodKeys,
sets: {
one: "Duplicate",
two: "Duplicate",
},
},
},
};
const status = validateLookupPackage(broken as unknown as typeof genshinGameData);
expect(status.valid).toBe(false);
expect(status.errors.join("\n")).toContain("Duplicate GOOD set key");
});
});
src/lib/genshinLookup.ts
+211
View File
@@ -0,0 +1,211 @@
import { simplifyForMatch } from "./fuzzyMatch.js";
import {
artifactPieces,
characterAliases,
genshinGameData,
globalMainStats,
globalSubstats,
knownCharacters,
knownSets,
mainStatsBySlot,
pieceAliases,
setAliases,
slotAliases,
slotNames,
statAliases,
} from "./genshinData.js";
export type LookupMatchSource = "exact" | "alias" | "fuzzy" | "missing";
export interface LookupMatch {
value: string;
confidence: number;
source: LookupMatchSource;
}
export interface LookupValidationStatus {
valid: boolean;
errors: string[];
warnings: string[];
summary: {
artifactSets: number;
artifactPieces: number;
characters: number;
stats: number;
generatedAt: string;
sourceVersion: string;
};
}
type LookupData = typeof genshinGameData & {
lookup?: {
normalizedKeys?: {
sets?: Record<string, string>;
pieces?: Record<string, string>;
slots?: Record<string, string>;
stats?: Record<string, string>;
characters?: Record<string, string>;
};
goodKeys?: {
sets?: Record<string, string>;
pieces?: Record<string, string>;
stats?: Record<string, string>;
characters?: Record<string, string>;
};
setToPieces?: Record<string, string[]>;
validation?: {
valid?: boolean;
errors?: string[];
warnings?: string[];
};
};
};
const lookupData = genshinGameData as LookupData;
const normalizedSets = lookupData.lookup?.normalizedKeys?.sets ?? buildNormalizedMap(knownSets);
const normalizedPieces = lookupData.lookup?.normalizedKeys?.pieces ?? buildNormalizedMap(artifactPieces.map((piece) => piece.name));
const normalizedSlots = lookupData.lookup?.normalizedKeys?.slots ?? buildNormalizedMap(slotNames);
const normalizedStats = lookupData.lookup?.normalizedKeys?.stats ?? buildNormalizedMap([...globalMainStats, ...globalSubstats]);
const normalizedCharacters = lookupData.lookup?.normalizedKeys?.characters ?? buildNormalizedMap(knownCharacters);
export function normalizeLookupKey(value: string) {
return simplifyForMatch(value).replace(/[^a-z0-9]+/g, "");
}
export function matchPiece(raw: string, minConfidence = 0.72): LookupMatch {
return matchLookup(raw, normalizedPieces, pieceAliases, minConfidence);
}
export function matchSet(raw: string, minConfidence = 0.72): LookupMatch {
return matchLookup(raw, normalizedSets, setAliases, minConfidence);
}
export function matchSlot(raw: string, minConfidence = 0.72): LookupMatch {
return matchLookup(raw, normalizedSlots, slotAliases, minConfidence);
}
export function matchStat(raw: string, minConfidence = 0.72): LookupMatch {
return matchLookup(raw, normalizedStats, statAliases, minConfidence);
}
export function matchCharacter(raw: string, minConfidence = 0.72): LookupMatch {
return matchLookup(raw, normalizedCharacters, characterAliases, minConfidence);
}
export function validateLookupPackage(data: LookupData = lookupData): LookupValidationStatus {
const errors: string[] = [];
const warnings: string[] = [];
const pieces = data.artifactPieces ?? artifactPieces;
const sets = data.artifactSets ?? [];
const characters = data.characters ?? [];
const slots = data.slots ?? slotNames;
const stats = data.stats?.main ?? data.mainStats ?? [];
const setNames = new Set(sets.map((set) => set.name).filter(Boolean));
const slotNameSet = new Set(slots);
if (!sets.length) errors.push("No artifact sets in lookup package.");
if (!pieces.length) errors.push("No artifact pieces in lookup package.");
if (!characters.length) warnings.push("No characters in lookup package.");
for (const piece of pieces) {
if (!piece.name) errors.push("Artifact piece without name.");
if (!piece.setName || !setNames.has(piece.setName)) errors.push(`Piece "${piece.name}" references missing set "${piece.setName}".`);
if (!piece.slot || !slotNameSet.has(piece.slot)) errors.push(`Piece "${piece.name}" references missing slot "${piece.slot}".`);
}
const goodSets = Object.values(data.lookup?.goodKeys?.sets ?? {});
const duplicateGoodSet = firstDuplicate(goodSets);
if (duplicateGoodSet) errors.push(`Duplicate GOOD set key "${duplicateGoodSet}".`);
for (const [alias, target] of Object.entries(data.aliases?.stats ?? {})) {
if (!stats.includes(target) && !(data.stats?.sub ?? data.substats ?? []).includes(target)) {
errors.push(`Stat alias "${alias}" targets unknown stat "${target}".`);
}
}
const generatedAt = typeof data.generatedAt === "string" ? data.generatedAt : "";
const sourceVersion = typeof data.sourceVersion === "string" ? data.sourceVersion : "unknown";
if (!generatedAt) warnings.push("Lookup package has no generatedAt timestamp.");
return {
valid: errors.length === 0,
errors,
warnings,
summary: {
artifactSets: sets.length,
artifactPieces: pieces.length,
characters: characters.length,
stats: stats.length,
generatedAt,
sourceVersion,
},
};
}
function matchLookup(
raw: string,
normalized: Record<string, string>,
aliases: Record<string, string>,
minConfidence: number,
): LookupMatch {
const key = normalizeLookupKey(raw);
if (!key) return missingMatch();
const alias = Object.entries(aliases).find(([from]) => normalizeLookupKey(from) === key);
if (alias) return { value: alias[1], confidence: 96, source: "alias" };
if (normalized[key]) return { value: normalized[key], confidence: 100, source: "exact" };
let bestValue = "";
let bestScore = 0;
for (const [candidateKey, value] of Object.entries(normalized)) {
const score = similarity(key, candidateKey);
if (score > bestScore) {
bestScore = score;
bestValue = value;
}
}
return bestScore >= minConfidence
? { value: bestValue, confidence: Math.round(bestScore * 100), source: "fuzzy" }
: missingMatch();
}
function missingMatch(): LookupMatch {
return { value: "", confidence: 0, source: "missing" };
}
function buildNormalizedMap(values: string[]) {
return Object.fromEntries(values.filter(Boolean).map((value) => [normalizeLookupKey(value), value]));
}
function firstDuplicate(values: string[]) {
const seen = new Set<string>();
for (const value of values) {
if (seen.has(value)) return value;
seen.add(value);
}
return "";
}
function similarity(a: string, b: string) {
if (a === b) return 1;
if (!a || !b) return 0;
const distance = levenshtein(a, b);
return 1 - distance / Math.max(a.length, b.length);
}
function levenshtein(a: string, b: string) {
const previous = Array.from({ length: b.length + 1 }, (_, index) => index);
const current = new Array<number>(b.length + 1);
for (let i = 1; i <= a.length; i++) {
current[0] = i;
for (let j = 1; j <= b.length; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
current[j] = Math.min(current[j - 1] + 1, previous[j] + 1, previous[j - 1] + cost);
}
previous.splice(0, previous.length, ...current);
}
return previous[b.length];
}
src/lib/layoutProfile.test.ts
+32 -6
View File
@@ -46,13 +46,17 @@ describe("layoutProfile", () => {
expect(profileDetailRect(HD)).toEqual({ x: 1308, y: 120, width: 492, height: 838 });
});
it("produces the four artifact crops in top-to-bottom order, all clamped", () => {
it("produces the split artifact crops in top-to-bottom order, all clamped", () => {
const detail = profileDetailRect(QHD);
const crops = detailCropRects(detail, QHD);
expect(crops.map((crop) => crop.id)).toEqual([
"artifact-title",
"artifact-main-stat",
"artifact-name",
"artifact-slot",
"artifact-main-stat-label",
"artifact-main-stat-value",
"artifact-level",
"artifact-substats",
"artifact-set-effects",
"artifact-footer",
]);
let previousY = -1;
@@ -65,6 +69,28 @@ describe("layoutProfile", () => {
}
});
it("matches Inventory Kamera-like 1080p artifact field crops", () => {
const detail = profileDetailRect(HD);
const crops = Object.fromEntries(detailCropRects(detail, HD).map((crop) => [crop.id, crop.rect]));
expect(crops["artifact-slot"]).toEqual({ x: 1328, y: 185, width: 234, height: 40 });
expect(crops["artifact-main-stat-label"]).toEqual({ x: 1328, y: 264, width: 224, height: 35 });
expect(crops["artifact-level"]).toEqual({ x: 1333, y: 425, width: 70, height: 35 });
expect(crops["artifact-substats"]).toEqual({ x: 1338, y: 473, width: 408, height: 193 });
});
it("uses Inventory Kamera's tighter substat crop in the fast auto-scan profile", () => {
const detail = profileDetailRect(HD);
const crops = Object.fromEntries(detailCropRects(detail, HD, { fastProfile: true }).map((crop) => [crop.id, crop.rect]));
expect(crops["artifact-substats"]).toEqual({ x: 1338, y: 473, width: 408, height: 154 });
});
it("shifts level and substat crops for sanctified artifacts like Inventory Kamera", () => {
const detail = profileDetailRect(HD);
const crops = Object.fromEntries(detailCropRects(detail, HD, { sanctified: true }).map((crop) => [crop.id, crop.rect]));
expect(crops["artifact-level"]).toEqual({ x: 1333, y: 468, width: 70, height: 35 });
expect(crops["artifact-substats"]).toEqual({ x: 1338, y: 517, width: 408, height: 193 });
});
it("places the inventory count crop inside the inventory panel", () => {
const detail = profileDetailRect(QHD);
const inv = inventoryRect(QHD, detail);
@@ -77,9 +103,9 @@ describe("layoutProfile", () => {
const detail = profileDetailRect(QHD);
const grid = inventoryGrid(QHD, detail);
expect(grid.cols).toBe(8);
expect(grid.rows).toBe(5);
expect(grid.rows).toBe(4);
expect(grid.source).toBe("detected");
expect(grid.centers).toHaveLength(40);
expect(grid.centers).toHaveLength(32);
expect(grid.centers.every((center) => center.x < detail.x)).toBe(true);
});
@@ -88,7 +114,7 @@ describe("layoutProfile", () => {
const grid = inventoryGrid(HD, detail);
expect(grid.centers[0]).toEqual({ x: 179, y: 254, row: 0, col: 0 });
expect(grid.centers[7]).toEqual({ x: 1201, y: 254, row: 0, col: 7 });
expect(grid.centers.at(-1)).toEqual({ x: 1201, y: 958, row: 4, col: 7 });
expect(grid.centers.at(-1)).toEqual({ x: 1201, y: 782, row: 3, col: 7 });
});
it("reports a missing grid when the inventory panel is too small", () => {
src/lib/layoutProfile.ts
+67 -15
View File
@@ -90,12 +90,21 @@ export function profileDetailRect(imageSize: { width: number; height: number }):
);
}
// The four OCR crops inside the detail panel, as fractions of the detail rect.
export function detailCropRects(detailRect: LayoutRect, imageSize: { width: number; height: number }): CropTemplateRect[] {
// OCR crops inside the detail panel, as fractions of the detail rect. These are
// deliberately closer to Inventory Kamera's split-card model than the older
// broad crops: small fields get small OCR profiles, while the legacy parser
// still accepts old review samples with artifact-title/artifact-main-stat.
export function detailCropRects(
detailRect: LayoutRect,
imageSize: { width: number; height: number },
options: { sanctified?: boolean; fastProfile?: boolean } = {},
): CropTemplateRect[] {
const sanctifiedShift = options.sanctified ? 0.0520 : 0;
const substatsHeight = options.fastProfile ? 0.1841 : 0.2301;
const templates: CropTemplateRect[] = [
{
id: "artifact-title",
label: "Artifact title",
id: "artifact-name",
label: "Artifact name",
rect: {
x: Math.round(detailRect.x),
y: Math.round(detailRect.y),
@@ -104,33 +113,73 @@ export function detailCropRects(detailRect: LayoutRect, imageSize: { width: numb
},
},
{
id: "artifact-main-stat",
label: "Main stat",
id: "artifact-slot",
label: "Artifact slot",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.0405),
y: Math.round(detailRect.y + detailRect.height * 0.0772),
width: Math.round(detailRect.width * 0.4757),
height: Math.round(detailRect.height * 0.0475),
},
},
{
id: "artifact-main-stat-label",
label: "Main stat label",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.0405),
y: Math.round(detailRect.y + detailRect.height * 0.1722),
width: Math.round(detailRect.width * 0.4555),
height: Math.round(detailRect.height * 0.0416),
},
},
{
id: "artifact-main-stat-value",
label: "Main stat value",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.055),
y: Math.round(detailRect.y + detailRect.height * 0.075),
width: Math.round(detailRect.width * 0.58),
height: Math.round(detailRect.height * 0.26),
y: Math.round(detailRect.y + detailRect.height * 0.205),
width: Math.round(detailRect.width * 0.42),
height: Math.round(detailRect.height * 0.105),
},
},
{
id: "artifact-level",
label: "Artifact level",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.0506),
y: Math.round(detailRect.y + detailRect.height * (0.3634 + sanctifiedShift)),
width: Math.round(detailRect.width * 0.1417),
height: Math.round(detailRect.height * 0.0416),
},
},
{
id: "artifact-substats",
label: "Substats",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.0605),
y: Math.round(detailRect.y + detailRect.height * (0.4216 + sanctifiedShift)),
width: Math.round(detailRect.width * 0.8297),
height: Math.round(detailRect.height * substatsHeight),
},
},
{
id: "artifact-set-effects",
label: "Set effects",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.055),
y: Math.round(detailRect.y + detailRect.height * 0.34),
y: Math.round(detailRect.y + detailRect.height * 0.655),
width: Math.round(detailRect.width * 0.86),
height: Math.round(detailRect.height * 0.27),
height: Math.round(detailRect.height * 0.16),
},
},
{
id: "artifact-footer",
label: "Footer",
rect: {
x: Math.round(detailRect.x + detailRect.width * 0.055),
y: Math.round(detailRect.y + detailRect.height * 0.82),
x: Math.round(detailRect.x + detailRect.width * 0.15),
y: Math.round(detailRect.y + detailRect.height * 0.938),
width: Math.round(detailRect.width * 0.86),
height: Math.round(detailRect.height * 0.14),
height: Math.round(detailRect.height * 0.06),
},
},
];
@@ -173,7 +222,10 @@ export function inventoryGrid(imageSize: { width: number; height: number }, deta
const stepX = Math.round(imageSize.width * 0.076);
const stepY = Math.round(imageSize.height * 0.163);
const visibleRows = 5;
// Inventory Kamera treats the artifact inventory as 32 safe click targets per
// full page. The apparent fifth row sits in the bottom control band on 16:9
// captures and is not a reliable target during automated scrolling.
const visibleRows = 4;
const startX = Math.round(imageSize.width * 0.093);
const startY = Math.round(imageSize.height * 0.235);
src/lib/ocrPreprocess.test.ts
+12
View File
@@ -40,6 +40,18 @@ describe("ocrPreprocess", () => {
expect(otsuThreshold(new Array<number>(256).fill(0))).toBe(127);
});
it("can increase contrast before histogramming small numeric crops", () => {
const bitmap = bitmapFrom([
[118, 118, 118],
[138, 138, 138],
], 2, 1);
const normal = computeLuminanceHistogram(bitmap);
const contrasted = computeLuminanceHistogram(bitmap, { contrast: 80 });
const normalValues = normal.flatMap((count, value) => Array.from({ length: count }, () => value));
const contrastedValues = contrasted.flatMap((count, value) => Array.from({ length: count }, () => value));
expect(Math.max(...contrastedValues) - Math.min(...contrastedValues)).toBeGreaterThan(Math.max(...normalValues) - Math.min(...normalValues));
});
it("inverts bright foreground to black-on-white by default", () => {
// Bright text pixel + dark background pixel.
const bitmap = bitmapFrom([
src/lib/ocrPreprocess.ts
+13 -4
View File
@@ -20,6 +20,8 @@ export interface BinarizeOptions {
invertBrightForeground?: boolean;
/** Override Otsu with a fixed 0-255 luminance threshold. */
threshold?: number;
/** Optional contrast adjustment in the usual -255..255 image-processing range. */
contrast?: number;
}
const BYTES_PER_PIXEL = 4;
@@ -35,12 +37,19 @@ function luminanceAt(data: Uint8Array | Buffer, index: number): number {
return 0.299 * r + 0.587 * g + 0.114 * b;
}
export function computeLuminanceHistogram(bitmap: Bitmap): number[] {
function adjustContrast(value: number, contrast = 0) {
if (!contrast) return value;
const safeContrast = Math.max(-255, Math.min(255, contrast));
const factor = (259 * (safeContrast + 255)) / (255 * (259 - safeContrast));
return Math.max(0, Math.min(255, factor * (value - 128) + 128));
}
export function computeLuminanceHistogram(bitmap: Bitmap, options: Pick<BinarizeOptions, "contrast"> = {}): number[] {
const histogram = new Array<number>(256).fill(0);
const { data, width, height } = bitmap;
const pixels = width * height;
for (let pixel = 0; pixel < pixels; pixel++) {
const value = Math.round(luminanceAt(data, pixel * BYTES_PER_PIXEL));
const value = Math.round(adjustContrast(luminanceAt(data, pixel * BYTES_PER_PIXEL), options.contrast));
histogram[Math.max(0, Math.min(255, value))]++;
}
return histogram;
@@ -82,13 +91,13 @@ export function otsuThreshold(histogram: readonly number[]): number {
export function binarizeForOcr(bitmap: Bitmap, options: BinarizeOptions = {}): Bitmap {
const { data, width, height } = bitmap;
const invert = options.invertBrightForeground ?? true;
const threshold = options.threshold ?? otsuThreshold(computeLuminanceHistogram(bitmap));
const threshold = options.threshold ?? otsuThreshold(computeLuminanceHistogram(bitmap, options));
const output = Buffer.alloc(width * height * BYTES_PER_PIXEL);
const pixels = width * height;
for (let pixel = 0; pixel < pixels; pixel++) {
const index = pixel * BYTES_PER_PIXEL;
const isBright = luminanceAt(data, index) > threshold;
const isBright = adjustContrast(luminanceAt(data, index), options.contrast) > threshold;
// Bright foreground text -> black; dark background -> white (inverted).
const value = invert ? (isBright ? 0 : 255) : (isBright ? 255 : 0);
output[index] = value;
src/lib/scanDiagnosticsLog.ts
+132
View File
@@ -0,0 +1,132 @@
import { detailFingerprint } from "./autoScanLoop";
import type { CaptureResult, ClickResult, KeyPressResult } from "../types/global";
export type ScanDiagnosticSeverity = "info" | "ok" | "warn" | "error";
type InventoryGrid = NonNullable<CaptureResult["inventoryGrid"]>;
type InventoryCount = NonNullable<CaptureResult["inventoryCount"]>;
export interface ScanDiagnosticEvent {
id: string;
at: string;
phase: string;
severity: ScanDiagnosticSeverity;
message: string;
details?: Record<string, string | number | boolean | null | undefined>;
capture?: {
id: string;
name: string;
width: number;
height: number;
capturedAt: string;
target?: CaptureResult["captureTarget"];
fingerprint: string;
inventoryFingerprint?: string;
layoutWarning?: string;
grid?: {
rows: number;
cols: number;
confidence: number;
source: InventoryGrid["source"];
targets: number;
};
count?: {
current: number;
total: number;
confidence: number;
source: InventoryCount["source"];
text: string;
};
artifactDetail?: NonNullable<CaptureResult["artifactDetail"]>;
paimonMenu?: NonNullable<CaptureResult["paimonMenu"]>;
sanctified?: boolean;
screenshots?: {
detail?: string;
inventory?: string;
full?: string;
};
};
}
export function createScanDiagnosticEvent(input: {
phase: string;
severity?: ScanDiagnosticSeverity;
message: string;
details?: ScanDiagnosticEvent["details"];
capture?: CaptureResult | null;
includeFullScreenshot?: boolean;
}): ScanDiagnosticEvent {
return {
id: `${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`,
at: new Date().toISOString(),
phase: input.phase,
severity: input.severity ?? "info",
message: input.message,
details: input.details,
capture: input.capture ? summarizeCapture(input.capture, Boolean(input.includeFullScreenshot)) : undefined,
};
}
export function summarizeClickResult(result: ClickResult) {
return {
ok: result.ok,
moved: result.moved,
clicked: result.clicked,
inputBlocked: result.inputBlocked,
focused: result.focused,
cursor: `${result.cursorX ?? "?"},${result.cursorY ?? "?"}`,
foreground: result.foregroundProcess,
target: result.targetProcess,
};
}
export function summarizeKeyPressResult(result: KeyPressResult | null | undefined) {
return {
ok: Boolean(result?.ok),
key: result?.key,
inputBlocked: Boolean(result?.inputBlocked),
focused: Boolean(result?.focused),
eventsSent: result?.eventsSent ?? 0,
foreground: result?.foregroundProcess,
target: result?.targetProcess,
};
}
function summarizeCapture(capture: CaptureResult, includeFullScreenshot: boolean): NonNullable<ScanDiagnosticEvent["capture"]> {
return {
id: capture.id,
name: capture.name,
width: capture.width,
height: capture.height,
capturedAt: capture.capturedAt,
target: capture.captureTarget,
fingerprint: detailFingerprint(capture),
inventoryFingerprint: capture.inventoryFingerprint,
layoutWarning: capture.layout?.warning || undefined,
grid: capture.inventoryGrid
? {
rows: capture.inventoryGrid.rows,
cols: capture.inventoryGrid.cols,
confidence: capture.inventoryGrid.confidence,
source: capture.inventoryGrid.source,
targets: capture.inventoryGrid.centers.length,
}
: undefined,
count: capture.inventoryCount
? {
current: capture.inventoryCount.current,
total: capture.inventoryCount.total,
confidence: capture.inventoryCount.confidence,
source: capture.inventoryCount.source,
text: capture.inventoryCount.text,
}
: undefined,
artifactDetail: capture.artifactDetail,
paimonMenu: capture.paimonMenu,
sanctified: capture.sanctified,
screenshots: {
detail: capture.detailDataUrl,
inventory: capture.inventoryDataUrl,
full: includeFullScreenshot ? capture.dataUrl : undefined,
},
};
}
src/lib/scannerLearning.test.ts
+46
View File
@@ -30,6 +30,23 @@ describe("scannerLearning", () => {
expect(shouldSaveReviewSample({ confidence: 90, notes: ["Main stat not confidently parsed."], fields: { mainStat: { confidence: 0 } } })).toBe(true);
});
it("does not save automatic review samples only because level or equipped is missing", () => {
expect(shouldSaveReviewSample({
confidence: 77,
notes: ["Artifact level not confidently parsed.", "equipped confidence is low; review before trusting it."],
fields: {
name: { confidence: 96 },
slot: { confidence: 96 },
level: { confidence: 0 },
mainStat: { confidence: 94 },
mainValue: { confidence: 96 },
setName: { confidence: 92 },
equipped: { confidence: 45 },
substats: { confidence: 96 },
},
})).toBe(false);
});
it("derives conservative replacements from OCR review samples", () => {
const reviewCapture: CaptureResult = {
id: "review",
@@ -53,6 +70,35 @@ describe("scannerLearning", () => {
expect(learned?.textReplacements?.Moor).toBe("Moon");
});
it("derives conservative replacements from split artifact OCR fields", () => {
const reviewCapture: CaptureResult = {
id: "review",
name: "review",
width: 1920,
height: 1080,
dataUrl: "",
capturedAt: new Date(0).toISOString(),
ocr: [
{ id: "artifact-name", label: "Name", text: "Heldenepos's Unspcken Tale", confidence: 73 },
{ id: "artifact-slot", label: "Slot", text: "Goblet of Eonothen", confidence: 74 },
{ id: "artifact-main-stat-label", label: "Main stat label", text: "Pvro DMG Bonus", confidence: 72 },
{ id: "artifact-main-stat-value", label: "Main stat value", text: "46.G%", confidence: 66 },
],
};
const learned = deriveScannerLearningRules(reviewCapture, parsedArtifact({
name: "Heldenepos's Unspoken Tale",
slot: "Goblet of Eonothem",
mainStat: "Pyro DMG Bonus",
mainValue: "46.6%",
}));
expect(learned?.textReplacements?.["Heldenepos's Unspcken Tale"]).toBe("Heldenepos's Unspoken Tale");
expect(learned?.textReplacements?.Eonothen).toBe("Eonothem");
expect(learned?.textReplacements?.Pvro).toBe("Pyro");
expect(learned?.textReplacements?.["46.G%"]).toBe("46.6%");
});
it("counts learned rules", () => {
expect(countScannerLearningRules({ textReplacements: { one: "1", two: "2" } })).toBe(2);
});
src/lib/scannerLearning.ts
+18 -2
View File
@@ -43,12 +43,12 @@ export function applyScannerLearningRules(capture: CaptureResult | null, rules?:
export function shouldSaveReviewSample(parsed: { confidence: number; notes: string[]; fields: Record<string, { confidence: number }> } | null) {
if (!parsed) return true;
if (parsed.confidence < 82) return true;
const criticalFields = ["name", "slot", "mainStat", "mainValue", "setName"];
if (criticalFields.some((fieldName) => {
const field = parsed.fields[fieldName];
return field ? field.confidence < 70 : false;
})) return true;
if (reviewRelevantConfidence(parsed.fields) < 82) return true;
if (parsed.notes.some((note) => /main stat not confidently parsed|main stat value not confidently parsed|set name not confidently parsed|slot not confidently parsed|artifact name not confidently parsed/i.test(note))) {
return true;
}
@@ -60,7 +60,7 @@ export function shouldFlagArtifactForReview(
parsed: { confidence: number; notes: string[]; fields: Record<string, { confidence: number }>; substats?: string[] } | null,
) {
if (!parsed) return true;
if (parsed.confidence < 78) return true;
if (reviewRelevantConfidence(parsed.fields) < 78) return true;
const criticalFields = ["name", "slot", "mainStat", "mainValue", "setName"];
if (criticalFields.some((fieldName) => (parsed.fields[fieldName]?.confidence ?? 0) < 70)) return true;
@@ -81,6 +81,14 @@ export function shouldFlagArtifactForReview(
return false;
}
function reviewRelevantConfidence(fields: Record<string, { confidence: number }>) {
const relevant = Object.entries(fields)
.filter(([fieldName]) => fieldName !== "level" && fieldName !== "equipped")
.map(([, field]) => field.confidence);
if (relevant.length === 0) return 0;
return Math.round(relevant.reduce((sum, confidence) => sum + confidence, 0) / relevant.length);
}
export function countScannerLearningRules(rules?: Partial<ScannerLearningRules> | null) {
return Object.keys(rules?.textReplacements ?? {}).length;
}
@@ -120,8 +128,16 @@ function expectedValuesForOcrEntry(id: string, parsed: ParsedArtifactCandidate)
switch (id) {
case "artifact-title":
return [parsed.name, parsed.slot];
case "artifact-name":
return [parsed.name];
case "artifact-slot":
return [parsed.slot];
case "artifact-main-stat":
return [parsed.mainStat, parsed.mainValue];
case "artifact-main-stat-label":
return [parsed.mainStat];
case "artifact-main-stat-value":
return [parsed.mainValue];
case "artifact-substats":
return parsed.substats;
case "artifact-set-effects":
src/lib/scannerSession.test.ts
+58 -1
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { clampScanLimit, clampSkipRows, resolveScanTargetCount } from "./scannerSession";
import { addCaptureTiming, addCardReadyTiming, addScrollReadyTiming, clampScanLimit, clampSkipRows, emptyAutoScanStats, resolveScanTargetCount, updateScanTiming } from "./scannerSession";
describe("scannerSession helpers", () => {
it("clamps scan target counts into a sane range", () => {
@@ -18,4 +18,61 @@ describe("scannerSession helpers", () => {
expect(clampSkipRows(-5)).toBe(0);
expect(clampSkipRows(99)).toBe(8);
});
it("updates scan timing and projects the 100-artifact run", () => {
const stats = { ...emptyAutoScanStats, parsed: 4, verified: 2 };
addCaptureTiming(stats, { totalMs: 1200, ocrMs: 900 });
addCaptureTiming(stats, { totalMs: 800, ocrMs: 500 });
updateScanTiming(stats, 1000, 9000);
expect(stats.elapsedMs).toBe(8000);
expect(stats.activeScanMs).toBe(8000);
expect(stats.averageMsPerParsed).toBe(2000);
expect(stats.activeAverageMsPerParsed).toBe(2000);
expect(stats.artifactsPerMinute).toBe(30);
expect(stats.activeArtifactsPerMinute).toBe(30);
expect(stats.projectedMsFor100).toBe(200000);
expect(stats.activeProjectedMsFor100).toBe(200000);
expect(stats.captureMs).toBe(2000);
expect(stats.ocrMs).toBe(1400);
expect(stats.averageCaptureMs).toBe(1000);
expect(stats.averageOcrMs).toBe(700);
expect(stats.captureP50Ms).toBe(800);
expect(stats.captureP90Ms).toBe(1200);
expect(stats.ocrP50Ms).toBe(500);
expect(stats.ocrP90Ms).toBe(900);
});
it("tracks detail-card and scroll readiness timing separately from OCR", () => {
const stats = { ...emptyAutoScanStats, parsed: 2, verified: 2 };
addCardReadyTiming(stats, 180);
addCardReadyTiming(stats, 220);
addScrollReadyTiming(stats, 320);
updateScanTiming(stats, 1000, 3000);
expect(stats.cardReadyMs).toBe(400);
expect(stats.cardReadyCount).toBe(2);
expect(stats.averageCardReadyMs).toBe(200);
expect(stats.scrollReadyMs).toBe(320);
expect(stats.scrollReadyCount).toBe(1);
expect(stats.averageScrollReadyMs).toBe(320);
});
it("preserves active scan timing when final elapsed time includes queued write flush", () => {
const stats = { ...emptyAutoScanStats, parsed: 5, verified: 5, activeScanMs: 4000, writeFlushMs: 900 };
updateScanTiming(stats, 1000, 6000, { preserveActiveScanMs: true });
expect(stats.elapsedMs).toBe(5000);
expect(stats.activeScanMs).toBe(4000);
expect(stats.writeFlushMs).toBe(900);
expect(stats.averageMsPerParsed).toBe(1000);
expect(stats.activeAverageMsPerParsed).toBe(800);
expect(stats.projectedMsFor100).toBe(100000);
expect(stats.activeProjectedMsFor100).toBe(80000);
});
it("keeps active scan timing growing during the live scan phase", () => {
const stats = { ...emptyAutoScanStats, parsed: 2, verified: 2, activeScanMs: 800 };
updateScanTiming(stats, 1000, 5000);
expect(stats.elapsedMs).toBe(4000);
expect(stats.activeScanMs).toBe(4000);
expect(stats.activeAverageMsPerParsed).toBe(2000);
});
});
src/lib/scannerSession.ts
+118
View File
@@ -8,6 +8,29 @@ export type AutoScanStats = {
duplicates: number;
misses: number;
pages: number;
elapsedMs: number;
activeScanMs: number;
writeFlushMs: number;
averageMsPerParsed: number;
activeAverageMsPerParsed: number;
artifactsPerMinute: number;
activeArtifactsPerMinute: number;
projectedMsFor100: number;
activeProjectedMsFor100: number;
captureMs: number;
ocrMs: number;
averageCaptureMs: number;
averageOcrMs: number;
captureP50Ms: number;
captureP90Ms: number;
ocrP50Ms: number;
ocrP90Ms: number;
cardReadyMs: number;
cardReadyCount: number;
averageCardReadyMs: number;
scrollReadyMs: number;
scrollReadyCount: number;
averageScrollReadyMs: number;
};
export type ScanSummary = AutoScanStats & {
@@ -27,8 +50,103 @@ export const emptyAutoScanStats: AutoScanStats = {
duplicates: 0,
misses: 0,
pages: 0,
elapsedMs: 0,
activeScanMs: 0,
writeFlushMs: 0,
averageMsPerParsed: 0,
activeAverageMsPerParsed: 0,
artifactsPerMinute: 0,
activeArtifactsPerMinute: 0,
projectedMsFor100: 0,
activeProjectedMsFor100: 0,
captureMs: 0,
ocrMs: 0,
averageCaptureMs: 0,
averageOcrMs: 0,
captureP50Ms: 0,
captureP90Ms: 0,
ocrP50Ms: 0,
ocrP90Ms: 0,
cardReadyMs: 0,
cardReadyCount: 0,
averageCardReadyMs: 0,
scrollReadyMs: 0,
scrollReadyCount: 0,
averageScrollReadyMs: 0,
};
const timingSamples = new WeakMap<AutoScanStats, { captureMs: number[]; ocrMs: number[] }>();
export function updateScanTiming(
stats: AutoScanStats,
startedAt: number,
now = Date.now(),
options: { preserveActiveScanMs?: boolean } = {},
) {
const elapsedMs = Math.max(0, Math.round(now - startedAt));
stats.elapsedMs = elapsedMs;
if (options.preserveActiveScanMs) {
if (!stats.activeScanMs || stats.activeScanMs > elapsedMs) stats.activeScanMs = elapsedMs;
} else {
stats.activeScanMs = elapsedMs;
}
stats.averageMsPerParsed = stats.parsed > 0 ? Math.round(elapsedMs / stats.parsed) : 0;
stats.activeAverageMsPerParsed = stats.parsed > 0 ? Math.round(stats.activeScanMs / stats.parsed) : 0;
stats.artifactsPerMinute = elapsedMs > 0 && stats.parsed > 0
? Math.round((stats.parsed * 60000 / elapsedMs) * 10) / 10
: 0;
stats.activeArtifactsPerMinute = stats.activeScanMs > 0 && stats.parsed > 0
? Math.round((stats.parsed * 60000 / stats.activeScanMs) * 10) / 10
: 0;
stats.projectedMsFor100 = stats.averageMsPerParsed > 0 ? stats.averageMsPerParsed * 100 : 0;
stats.activeProjectedMsFor100 = stats.activeAverageMsPerParsed > 0 ? stats.activeAverageMsPerParsed * 100 : 0;
stats.averageCaptureMs = stats.verified > 0 ? Math.round(stats.captureMs / stats.verified) : 0;
stats.averageOcrMs = stats.verified > 0 ? Math.round(stats.ocrMs / stats.verified) : 0;
const samples = timingSamples.get(stats);
stats.captureP50Ms = percentile(samples?.captureMs, 50);
stats.captureP90Ms = percentile(samples?.captureMs, 90);
stats.ocrP50Ms = percentile(samples?.ocrMs, 50);
stats.ocrP90Ms = percentile(samples?.ocrMs, 90);
stats.averageCardReadyMs = stats.cardReadyCount > 0 ? Math.round(stats.cardReadyMs / stats.cardReadyCount) : 0;
stats.averageScrollReadyMs = stats.scrollReadyCount > 0 ? Math.round(stats.scrollReadyMs / stats.scrollReadyCount) : 0;
return stats;
}
export function addCaptureTiming(
stats: AutoScanStats,
timing?: { totalMs?: number; ocrMs?: number } | null,
) {
if (!timing) return stats;
const captureMs = Math.max(0, Math.round(timing.totalMs ?? 0));
const ocrMs = Math.max(0, Math.round(timing.ocrMs ?? 0));
stats.captureMs += captureMs;
stats.ocrMs += ocrMs;
const samples = timingSamples.get(stats) ?? { captureMs: [], ocrMs: [] };
samples.captureMs.push(captureMs);
samples.ocrMs.push(ocrMs);
timingSamples.set(stats, samples);
return stats;
}
export function addCardReadyTiming(stats: AutoScanStats, elapsedMs: number) {
stats.cardReadyMs += Math.max(0, Math.round(elapsedMs));
stats.cardReadyCount += 1;
return stats;
}
export function addScrollReadyTiming(stats: AutoScanStats, elapsedMs: number) {
stats.scrollReadyMs += Math.max(0, Math.round(elapsedMs));
stats.scrollReadyCount += 1;
return stats;
}
function percentile(values: readonly number[] | undefined, p: number) {
if (!values || values.length === 0) return 0;
const sorted = [...values].sort((left, right) => left - right);
const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1));
return sorted[index];
}
export function clampScanLimit(value: number) {
return Math.max(1, Math.min(1800, Math.round(value || 1)));
}
src/pages/app/AppPageLayout.tsx
+1 -1
View File
@@ -53,7 +53,7 @@ export function AppPageLayout({ controller }: AppPageLayoutProps) {
overlayIcon={<Eye size={16} />}
demoIcon={<Play size={16} />}
/>
{activeView !== "diagnose" && <AppMetrics metricCards={metricCards} />}
{activeView !== "scan" && activeView !== "diagnose" && <AppMetrics metricCards={metricCards} />}
{(activeView === "scan" || activeView === "diagnose") && (
<ScanView
mode={activeView === "diagnose" ? "diagnose" : "workspace"}
src/services/assistantBridge.ts
+3
View File
@@ -19,6 +19,7 @@ import type {
ScannerCommand,
ScannerStatusPayload,
ScannerLearningRulePayload,
KeyPressResult,
} from "../types/global";
import type { AppSnapshot } from "../types/domain";
import type { StoredArtifactRecord } from "../types/storage";
@@ -57,6 +58,7 @@ export interface AssistantBridge {
focusGenshinForScanStart: () => Promise<FocusGenshinResult>;
clickScreen: (x: number, y: number) => Promise<ClickResult>;
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => Promise<ScrollResult>;
keyPress: (key: string) => Promise<KeyPressResult>;
onScannerCommand: (callback: (command: ScannerCommand) => void) => () => void;
}
@@ -99,6 +101,7 @@ export function getAssistantBridge(): AssistantBridge | null {
focusGenshinForScanStart: () => (hasFocusGenshinForScanStart ? api.focusGenshinForScanStart() : api.focusGenshin()),
clickScreen: (x: number, y: number) => api.clickScreen(x, y),
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => api.scrollScreen(notches, anchorX, anchorY),
keyPress: (key: string) => api.keyPress(key),
showOverlay: () => api.showOverlay(),
onScannerCommand: (callback) => api.onScannerCommand(callback),
};
src/styles/global.css
+464 -68
View File
@@ -24,6 +24,14 @@
box-sizing: border-box;
}
html,
body,
#root {
width: 100%;
height: 100%;
overflow: hidden;
}
body {
margin: 0;
min-width: 1100px;
@@ -187,7 +195,8 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.app-shell {
display: grid;
grid-template-columns: 260px 1fr;
min-height: 100vh;
height: 100vh;
overflow: hidden;
background:
linear-gradient(90deg, rgba(255, 255, 255, 0.035), transparent 24%),
linear-gradient(180deg, rgba(184, 140, 255, 0.06), transparent 42%);
@@ -197,6 +206,8 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
display: flex;
flex-direction: column;
gap: 24px;
min-height: 0;
overflow: hidden;
border-right: 1px solid var(--line);
background: rgba(13, 9, 25, 0.72);
box-shadow: inset -1px 0 0 rgba(255, 255, 255, 0.04);
@@ -301,8 +312,13 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
}
.main-panel {
padding: 26px;
display: grid;
grid-template-rows: auto minmax(0, 1fr);
gap: 12px;
min-height: 0;
min-width: 0;
overflow: hidden;
padding: 18px;
}
.topbar {
@@ -314,9 +330,9 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.topbar h1 {
max-width: 780px;
margin: 4px 0 0;
margin: 2px 0 0;
color: #fbf8ff;
font-size: 30px;
font-size: 22px;
letter-spacing: 0;
text-shadow: 0 18px 48px rgba(184, 140, 255, 0.25);
}
@@ -334,6 +350,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
display: flex;
align-items: center;
gap: 10px;
min-width: 0;
}
.topbar-status {
@@ -347,8 +364,8 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.ghost-button,
.primary-button {
height: 40px;
padding: 0 14px;
height: 36px;
padding: 0 12px;
}
.ghost-button {
@@ -450,6 +467,8 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.content-grid {
display: grid;
gap: 14px;
min-height: 0;
overflow: hidden;
}
.scan-layout {
@@ -457,6 +476,8 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
}
.panel {
min-height: 0;
overflow: hidden;
padding: 18px;
}
@@ -1044,9 +1065,13 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.scanner-workbench {
display: grid;
gap: 14px;
grid-template-rows: auto auto minmax(0, 1fr);
gap: 10px;
min-height: 0;
height: 100%;
width: 100%;
justify-self: stretch;
overflow: hidden;
border: 1px solid var(--line);
border-radius: 8px;
background:
@@ -1054,7 +1079,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
rgba(18, 12, 35, 0.7);
box-shadow: var(--glass-shadow);
backdrop-filter: blur(22px);
padding: 18px;
padding: 14px;
}
.scanner-header,
@@ -1071,7 +1096,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.scanner-header h2,
.modal-header h2 {
margin: 3px 0 0;
font-size: 22px;
font-size: 20px;
}
.scanner-subcopy {
@@ -1102,7 +1127,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.source-select {
display: grid;
min-width: 0;
gap: 6px;
gap: 4px;
}
.source-select span {
@@ -1113,7 +1138,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.source-select select {
width: 100%;
min-height: 42px;
min-height: 34px;
border: 1px solid var(--line);
border-radius: 8px;
background: rgba(15, 10, 29, 0.78);
@@ -1155,7 +1180,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
color: var(--danger);
font-size: 13px;
font-weight: 700;
padding: 12px 14px;
padding: 10px 12px;
}
.dev-toggle {
@@ -1170,22 +1195,23 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.player-scan-card {
display: grid;
gap: 12px;
gap: 8px;
min-height: 0;
border: 1px solid var(--line);
border-radius: 12px;
border-radius: 8px;
background: rgba(15, 10, 29, 0.6);
padding: 16px;
padding: 10px;
}
.player-scan-row {
display: flex;
align-items: end;
flex-wrap: wrap;
gap: 10px;
gap: 8px;
}
.player-scan-row .source-select {
min-width: 260px;
min-width: 240px;
flex: 1;
}
@@ -1220,7 +1246,14 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 8px;
}
.player-scan-lower {
display: grid;
grid-template-columns: minmax(360px, auto) minmax(360px, 1fr);
gap: 10px;
align-items: center;
}
.learning-strip {
@@ -1259,10 +1292,10 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
}
.runtime-pill {
min-height: 34px;
min-height: 30px;
display: inline-flex;
align-items: center;
padding: 0 12px;
padding: 0 10px;
border-radius: 999px;
border: 1px solid rgba(188, 154, 255, 0.22);
background: rgba(15, 10, 29, 0.55);
@@ -1284,14 +1317,14 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
}
.scan-cta {
min-height: 44px;
padding: 0 22px;
min-height: 36px;
padding: 0 16px;
font-size: 14px;
}
.stop-button {
min-height: 44px;
padding: 0 20px;
min-height: 36px;
padding: 0 18px;
border: 1px solid rgba(255, 140, 157, 0.5);
border-radius: 10px;
background: rgba(255, 140, 157, 0.14);
@@ -1310,8 +1343,11 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.player-status {
margin: 0;
color: var(--text-soft);
font-size: 13px;
line-height: 1.5;
font-size: 12px;
line-height: 1.35;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.scanner-preflight {
@@ -1362,11 +1398,12 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.player-progress {
display: grid;
gap: 8px;
gap: 7px;
min-width: 0;
}
.player-progress-bar {
height: 8px;
height: 7px;
border-radius: 999px;
background: rgba(188, 154, 255, 0.12);
overflow: hidden;
@@ -1382,9 +1419,21 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.player-progress-stats {
display: flex;
flex-wrap: wrap;
gap: 14px;
gap: 6px;
color: var(--text-muted);
font-size: 12px;
font-size: 11px;
}
.player-progress-stats span {
display: inline-flex;
align-items: center;
min-height: 26px;
gap: 4px;
padding: 0 8px;
border: 1px solid rgba(188, 154, 255, 0.14);
border-radius: 999px;
background: rgba(8, 6, 18, 0.38);
white-space: nowrap;
}
.player-progress-stats strong {
@@ -1414,7 +1463,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
}
.scanner-settings-modal {
width: min(760px, 92vw);
width: min(820px, 92vw);
}
.diagnostics-actions {
@@ -1427,6 +1476,25 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
grid-template-columns: repeat(4, minmax(0, 1fr));
}
.settings-preflight {
grid-template-columns: repeat(3, minmax(0, 1fr)) minmax(160px, 1.1fr);
}
.settings-preflight p {
align-self: stretch;
display: grid;
align-content: center;
margin: 0;
border: 1px solid rgba(188, 154, 255, 0.12);
border-radius: 8px;
background: rgba(8, 6, 18, 0.34);
color: #f4efff;
font-size: 15px;
font-weight: 800;
line-height: 1.35;
padding: 10px 12px;
}
.diagnostics-status {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
@@ -1569,48 +1637,123 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
font-size: 11px;
}
.scan-config-strip {
.scan-settings-layout {
display: grid;
gap: 12px;
}
.scan-settings-controls {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 12px;
}
.settings-stepper {
display: grid;
grid-template-columns: 170px 170px minmax(0, 1fr);
gap: 10px;
align-items: end;
border: 1px solid rgba(188, 154, 255, 0.14);
border-radius: 8px;
background: rgba(10, 7, 20, 0.36);
padding: 12px;
}
.scan-config-strip label {
display: grid;
gap: 6px;
.settings-stepper-head {
min-height: 42px;
}
.scan-config-strip span {
.settings-stepper-head div {
display: grid;
gap: 4px;
}
.settings-stepper-head span,
.scan-settings-note strong {
color: var(--text-soft);
font-size: 11px;
font-weight: 800;
}
.scan-config-strip input {
min-height: 38px;
.settings-stepper-head small,
.scan-settings-note span {
color: var(--text-muted);
font-size: 12px;
line-height: 1.35;
}
.settings-stepper-row {
display: grid;
grid-template-columns: 44px minmax(0, 1fr) 44px;
gap: 8px;
}
.settings-stepper-row input {
min-width: 0;
min-height: 44px;
border: 1px solid var(--line);
border-radius: 8px;
background: rgba(15, 10, 29, 0.78);
color: #f4efff;
padding: 0 10px;
padding: 0 12px;
font-size: 18px;
font-weight: 800;
text-align: center;
outline: none;
}
.scan-config-strip input:focus {
.settings-stepper-row input:focus {
border-color: rgba(126, 231, 242, 0.52);
box-shadow: 0 0 0 3px rgba(126, 231, 242, 0.08);
}
.scan-config-strip p {
.settings-stepper-row input::-webkit-outer-spin-button,
.settings-stepper-row input::-webkit-inner-spin-button {
margin: 0;
color: var(--text-muted);
appearance: none;
}
.stepper-button,
.settings-presets button {
min-height: 44px;
border: 1px solid rgba(188, 154, 255, 0.22);
border-radius: 8px;
background: rgba(188, 154, 255, 0.08);
color: #f4efff;
font-weight: 900;
cursor: pointer;
}
.stepper-button {
font-size: 22px;
line-height: 1;
}
.stepper-button:hover,
.stepper-button:focus-visible,
.settings-presets button:hover,
.settings-presets button:focus-visible {
border-color: rgba(126, 231, 242, 0.45);
background: rgba(126, 231, 242, 0.12);
}
.settings-presets {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 8px;
}
.settings-presets button {
min-height: 34px;
color: var(--text-soft);
font-size: 12px;
line-height: 1.45;
}
.scan-settings-note {
display: grid;
gap: 4px;
border: 1px solid rgba(126, 231, 242, 0.16);
border-radius: 8px;
background: rgba(126, 231, 242, 0.06);
padding: 11px 12px;
}
.grid-detection-strip {
@@ -1682,23 +1825,142 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
overflow-wrap: anywhere;
}
.scan-evidence-timeline {
display: grid;
gap: 10px;
max-height: 520px;
overflow-y: auto;
padding-right: 4px;
}
.scan-evidence-event {
display: grid;
gap: 8px;
border: 1px solid rgba(255, 255, 255, 0.1);
border-left: 3px solid rgba(255, 255, 255, 0.24);
border-radius: 8px;
background: rgba(255, 255, 255, 0.035);
padding: 10px;
}
.scan-evidence-event.ok {
border-left-color: var(--mint);
}
.scan-evidence-event.warn {
border-left-color: var(--amber);
}
.scan-evidence-event.error {
border-left-color: var(--danger);
}
.scan-evidence-header {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: 8px;
min-width: 0;
}
.scan-evidence-header span,
.scan-evidence-header em {
color: var(--text-muted);
font-size: 11px;
font-style: normal;
font-weight: 800;
text-transform: uppercase;
}
.scan-evidence-header strong {
color: var(--text);
font-size: 13px;
}
.scan-evidence-event p {
margin: 0;
color: var(--text-soft);
font-size: 12px;
line-height: 1.35;
}
.scan-evidence-details {
display: flex;
flex-wrap: wrap;
gap: 6px;
}
.scan-evidence-details span {
border: 1px solid rgba(255, 255, 255, 0.09);
border-radius: 999px;
background: rgba(0, 0, 0, 0.16);
color: var(--text-muted);
font-size: 11px;
padding: 4px 7px;
}
.scan-evidence-capture {
display: grid;
grid-template-columns: minmax(190px, 0.8fr) minmax(220px, 1.2fr);
gap: 10px;
align-items: start;
}
.scan-evidence-capture > div:first-child {
display: grid;
gap: 4px;
min-width: 0;
}
.scan-evidence-capture strong {
color: var(--text);
font-size: 12px;
}
.scan-evidence-capture span {
color: var(--text-muted);
font-size: 11px;
overflow-wrap: anywhere;
}
.scan-evidence-capture .evidence-warning {
color: var(--amber);
}
.scan-evidence-images {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 8px;
}
.scan-evidence-images img {
width: 100%;
max-height: 160px;
object-fit: contain;
border: 1px solid rgba(255, 255, 255, 0.1);
border-radius: 6px;
background: rgba(0, 0, 0, 0.3);
}
.scanner-main-grid {
display: grid;
grid-template-columns: minmax(0, 1fr) minmax(280px, 360px);
gap: 14px;
align-items: stretch;
grid-template-columns: minmax(190px, 210px) minmax(520px, 1fr);
gap: 12px;
align-items: start;
}
.capture-stage-shell {
display: grid;
gap: 10px;
gap: 8px;
min-width: 0;
}
.capture-stage {
display: grid;
min-height: clamp(440px, 58vh, 700px);
max-height: 720px;
width: 100%;
aspect-ratio: 41 / 80;
min-height: 0;
max-height: 410px;
place-items: center;
overflow: hidden;
border: 1px solid rgba(188, 154, 255, 0.16);
@@ -1706,19 +1968,31 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
background: rgba(5, 4, 12, 0.72);
}
.capture-stage-shell.is-empty .capture-stage {
aspect-ratio: 41 / 80;
max-height: 410px;
}
.capture-stage-shell.has-capture .capture-stage {
background:
radial-gradient(circle at 50% 34%, rgba(126, 231, 242, 0.08), transparent 44%),
rgba(5, 4, 12, 0.78);
}
.capture-stage img {
display: block;
width: auto;
height: auto;
max-width: min(100%, 980px);
max-height: min(100%, 680px);
max-width: 100%;
max-height: 100%;
object-fit: contain;
object-position: center top;
}
.capture-stage-meta {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 10px;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 6px;
}
.capture-stage-meta div {
@@ -1727,7 +2001,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
border: 1px solid rgba(188, 154, 255, 0.14);
border-radius: 8px;
background: rgba(8, 6, 18, 0.34);
padding: 10px 12px;
padding: 7px 8px;
}
.capture-stage-meta span {
@@ -1738,36 +2012,45 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.capture-stage-meta strong {
color: #f4efff;
font-size: 12px;
font-size: 11px;
overflow-wrap: anywhere;
}
.empty-stage {
display: grid;
place-items: center;
align-content: center;
gap: 8px;
padding: 18px;
color: var(--text-soft);
font-size: 12px;
line-height: 1.35;
text-align: center;
}
.empty-stage strong {
color: #f4efff;
font-size: 14px;
}
.empty-stage span {
max-width: 22ch;
}
.scanner-result-panel {
display: grid;
align-content: start;
max-width: 360px;
gap: 12px;
max-width: none;
gap: 10px;
border: 1px solid rgba(126, 231, 242, 0.18);
border-radius: 8px;
background: rgba(126, 231, 242, 0.07);
padding: 14px;
padding: 12px;
}
.result-heading h3 {
margin: 3px 0 0;
font-size: 18px;
font-size: 17px;
}
.result-score {
@@ -1798,8 +2081,8 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.scanner-result-brief span {
display: inline-flex;
align-items: center;
min-height: 30px;
padding: 0 10px;
min-height: 28px;
padding: 0 9px;
border: 1px solid rgba(188, 154, 255, 0.14);
border-radius: 999px;
background: rgba(8, 6, 18, 0.38);
@@ -1811,7 +2094,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
.scanner-result-actions {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 10px;
gap: 8px;
}
.scanner-result-caption {
@@ -2045,6 +2328,10 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
grid-template-columns: 1fr;
}
.player-scan-lower {
grid-template-columns: 1fr;
}
.scanner-toolbar {
align-items: stretch;
flex-direction: column;
@@ -2054,12 +2341,12 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
justify-content: flex-start;
}
.scan-config-strip {
grid-template-columns: 1fr 1fr;
.scan-settings-controls {
grid-template-columns: 1fr;
}
.scan-config-strip p {
grid-column: 1 / -1;
.settings-preflight {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
.capture-stage-meta {
@@ -2068,7 +2355,7 @@ button[data-clickable="true"]:focus-visible:not(:disabled):not([aria-disabled="t
}
.capture-stage img {
max-height: 640px;
max-height: 100%;
}
button:disabled {
@@ -2186,6 +2473,11 @@ button:disabled {
.diagnose-view {
display: grid;
gap: 14px;
min-height: 0;
height: 100%;
overflow-y: auto;
overscroll-behavior: contain;
padding-right: 6px;
width: 100%;
}
@@ -2230,9 +2522,113 @@ button:disabled {
gap: 12px;
}
.diagnose-card-heading h3 {
margin: 2px 0 0;
font-size: 17px;
}
.app-diagnosis-card {
gap: 12px;
}
.diagnosis-source {
display: inline-flex;
align-items: center;
gap: 6px;
border: 1px solid rgba(255, 255, 255, 0.11);
border-radius: 999px;
padding: 7px 10px;
color: var(--text-soft);
font-size: 12px;
white-space: nowrap;
}
.app-diagnosis-grid {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 10px;
}
.app-diagnosis-section {
display: grid;
gap: 8px;
min-width: 0;
border-left: 2px solid rgba(255, 255, 255, 0.2);
border-radius: 8px;
background: rgba(255, 255, 255, 0.035);
padding: 11px 12px;
}
.app-diagnosis-section.ok {
border-left-color: var(--mint);
}
.app-diagnosis-section.warn {
border-left-color: var(--amber);
}
.app-diagnosis-section.risk {
border-left-color: #ff6b8a;
}
.app-diagnosis-section.next {
border-left-color: var(--cyan);
}
.app-diagnosis-title {
display: flex;
align-items: center;
gap: 7px;
min-width: 0;
color: var(--text);
}
.app-diagnosis-title strong {
overflow: hidden;
font-size: 13px;
text-overflow: ellipsis;
white-space: nowrap;
}
.app-diagnosis-section ul {
display: grid;
gap: 7px;
margin: 0;
padding-left: 16px;
color: var(--text-soft);
font-size: 12px;
line-height: 1.35;
}
.app-diagnosis-section li::marker {
color: rgba(255, 255, 255, 0.45);
}
@media (max-width: 1200px) {
.diagnose-grid {
grid-template-columns: 1fr;
}
.app-diagnosis-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
}
@media (max-width: 720px) {
.scan-evidence-capture {
grid-template-columns: 1fr;
}
.scan-evidence-images {
grid-template-columns: 1fr;
}
.app-diagnosis-grid {
grid-template-columns: 1fr;
}
.diagnosis-source {
width: 100%;
justify-content: center;
}
}
src/types/global.d.ts
Vendored
+76 -1
View File
@@ -3,6 +3,18 @@ import type { StoredArtifactRecord } from "./storage";
export interface CaptureOptions {
skipOcr?: boolean;
skipOcrUnlessArtifactDetail?: boolean;
ocrMode?: "full" | "artifact";
ocrProfile?: "full" | "fast";
ocrEngine?: "current" | "native-tesseract" | "benchmark" | "ik-traineddata";
scanEntryMode?: "visible-inventory" | "direct-inventory" | "paimon-menu" | "auto-entry";
omitFullFrame?: boolean;
omitDetailPreview?: boolean;
omitInventoryPreview?: boolean;
omitCrops?: boolean;
omitCropImages?: boolean;
omitEquippedOcr?: boolean;
omitLockState?: boolean;
}
export interface CaptureSourceInfo {
@@ -16,7 +28,7 @@ export interface CaptureCrop {
id: string;
label: string;
rect: { x: number; y: number; width: number; height: number };
dataUrl: string;
dataUrl?: string;
}
export interface OcrResult {
@@ -24,6 +36,7 @@ export interface OcrResult {
label: string;
text: string;
confidence: number;
elapsedMs?: number;
}
export type CaptureTarget = "genshin-client" | "primary-screen" | "desktop-source";
@@ -38,6 +51,8 @@ export interface CaptureResult {
captureTarget?: CaptureTarget;
detailDataUrl?: string;
inventoryDataUrl?: string;
detailFingerprint?: string;
inventoryFingerprint?: string;
ocrSkipped?: boolean;
ocrTimedOut?: boolean;
crops?: CaptureCrop[];
@@ -49,6 +64,23 @@ export interface CaptureResult {
confidence: number;
source: "detected" | "fallback" | "missing";
};
artifactDetail?: {
present: boolean;
confidence: number;
orangeHits: number;
greenHits: number;
textHits: number;
titleOrangeHits?: number;
upperTextHits?: number;
lowerGreenHits?: number;
};
paimonMenu?: {
present: boolean;
confidence: number;
profileLightPct: number;
profileCreamPct: number;
menuTileDarkPct: number;
};
inventoryCount?: {
current: number;
total: number;
@@ -57,11 +89,23 @@ export interface CaptureResult {
text: string;
};
locked?: boolean;
sanctified?: boolean;
layout?: {
aspect: string;
isSixteenNine: boolean;
warning: string;
};
timings?: {
totalMs: number;
prepareMs: number;
ocrMs: number;
cropCount: number;
ocrEngine: CaptureOptions["ocrEngine"];
ocrWorkerPoolSize?: number;
ocrProfile?: CaptureOptions["ocrProfile"];
ocrFieldMs?: Record<string, number>;
ocrSkipped: boolean;
};
}
export interface WindowBounds {
@@ -97,6 +141,7 @@ export interface RuntimeInfo {
ok: boolean;
isElevated: boolean;
platform: string;
appBuild?: AppRuntimeInfo;
hotkeys?: Record<string, boolean>;
genshinFound?: boolean;
genshinHwnd?: number;
@@ -106,6 +151,15 @@ export interface RuntimeInfo {
helperPid?: number;
}
export interface AppRuntimeInfo {
signature: string;
pid: number;
startedAt: string;
cwd: string;
isDev: boolean;
expectedOcrWorkerPoolSize?: number;
}
export interface FocusGenshinResult {
focused: boolean;
alreadyForeground: boolean;
@@ -132,6 +186,8 @@ export type ScannerCommand =
| {
type: "start-auto";
scanLimit?: number;
scanEntryMode?: CaptureOptions["scanEntryMode"];
ocrEngine?: CaptureOptions["ocrEngine"];
};
export interface ScannerLearningRulePayload {
@@ -184,6 +240,17 @@ export interface ScrollResult {
isElevated?: boolean;
}
export interface KeyPressResult {
ok: boolean;
key: string;
focused?: boolean;
foregroundProcess?: string;
targetProcess?: string;
isElevated?: boolean;
inputBlocked?: boolean;
eventsSent?: number;
}
export type ArtifactSaveResult = ArtifactStoreSaveResult;
export interface ArtifactStoreLoadResult {
@@ -267,9 +334,16 @@ export interface ScannerStatusPayload {
snapshotBuilds: number;
grid: CaptureResult["inventoryGrid"] | null;
automationLog: string[];
diagnosticEvents?: unknown[];
runtimeInfo: RuntimeInfo | null;
appBuild?: AppRuntimeInfo;
storedTotal: number | null;
learningRuleCount: number;
lookupStatus?: unknown;
ocrEngine?: CaptureOptions["ocrEngine"];
ocrWarmup?: unknown;
entryMode?: CaptureOptions["scanEntryMode"];
benchmarkSummary?: unknown;
updatedAt: string | null;
[key: string]: unknown;
}
@@ -309,6 +383,7 @@ declare global {
captureSource: (sourceId: string, delayMs?: number, focusGenshin?: boolean, options?: CaptureOptions) => Promise<CaptureResult>;
clickScreen: (x: number, y: number) => Promise<ClickResult>;
scrollScreen: (notches: number, anchorX?: number, anchorY?: number) => Promise<ScrollResult>;
keyPress: (key: string) => Promise<KeyPressResult>;
getAutomationGuard: () => Promise<AutomationGuard>;
focusMainWindow: () => Promise<BooleanResult>;
focusGenshin: () => Promise<FocusGenshinResult>;
vite.config.ts
+3
View File
@@ -6,6 +6,9 @@ export default defineConfig({
server: {
port: 5173,
strictPort: true,
watch: {
ignored: ["**/outputs/**"],
},
},
build: {
outDir: "dist",