bao/genshin-assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare scanner branch for merge

Browse Source
This commit is contained in:
AzuTear
2026-07-09 08:44:50 +02:00
parent f791d1464c
commit 8b73c01e46
69 changed files with 6700 additions and 3773 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/scanner-ik-progress-report.md
+114 -37
View File
@@ -16,17 +16,22 @@ Inventory-Kamera-style artifact scanner:
missing-detail states block before OCR/store/review work.
- OCR uses a fast artifact profile that skips low-value fields and derives
slot, set, and main-stat value through lookup constraints when safe.
- The OCR worker pool, field crop split, page model, scroll model, and readiness
waits now mirror the relevant IK design choices more closely.
- The OCR worker pool, field crop split, page model, scroll model, and direct
detail-change verification now mirror the relevant IK design choices more
closely.
- Diagnostics now preserve state evidence, timings, screenshots where useful,
entry events, focus/input events, preflight failures, and scan-loop reasons.
- A live soak runner now measures throughput and quality, compares current vs.
IK-traineddata engines, and refuses to run against stale Electron builds.
The requested final goal is not proven complete yet. The current live dev port is
still owned by an older elevated Electron instance, so the latest code cannot be
truthfully benchmarked against IK until the app is restarted with UAC approval
and `npm run scan:goal:compare` completes a qualified 100-artifact run.
The current-vs-IK-traineddata comparison proof is now captured. On 2026-07-08,
`npm run scan:goal:compare:validated` passed with
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`.
At `limit=100`, `current` won the qualified comparison with `100/100` parsed,
`0` review, `0` misses, `378 ms/artifact` active average, and `37800 ms`
projected time for 100 artifacts. `ik-traineddata` was not qualified at
`limit=100` because it parsed `97/100`, had `5` review and `3` misses. The
separate 3 artifacts/second target is still not proven.
## What Changed
@@ -51,13 +56,19 @@ against a canonical package before it is accepted.
name, slot, main-stat label, main-stat value, level, substats, set effects,
equipped/footer, lock, and rarity.
- Fast auto-scan profile skips lower-value OCR work:
set effects, slot crop, main-stat value crop, equipped footer, crop images,
full-frame payloads, and inventory preview payloads.
set effects, main-stat value crop, crop images, full-frame payloads, and
inventory preview payloads. The equipped footer remains in real artifact-read
captures when its marker is visible, because ownership now matters for phase 1
validation. Preflight and poll captures still skip OCR/crops/lock-state work.
The slot crop remains in the fast path because it materially improved
real-read quality.
- Slot, set, and main-stat value are derived when lookup, slot rules, and level
constraints make that safe.
- Field-specific Tesseract PSM/whitelist cleanup and preprocessing are used.
- OCR crops are passed as PNG buffers internally instead of Base64 DataURLs.
- Exact visual duplicates are skipped before OCR.
- Exact visual duplicate skipping is disabled in the hottest path; duplicate
handling now primarily uses parsed artifact signatures so OCR is not skipped
solely from a crop fingerprint collision.
Why this matters:
@@ -76,8 +87,16 @@ manual-debug capture that OCRs every visible thing.
- `npm run scan:goal:current`
- `npm run scan:goal:ik`
- `npm run scan:goal:compare`
- `npm run scan:goal:compare:validated`
- `npm run scan:goal:compare:validated:wait`
- `npm run scan:iterate:compare:validated`
- `npm run scan:iterate:compare:validated:wait`
- `npm run scan:live:preflight`
- `npm run scan:live:preflight:wait`
- `scan-performance-assessment.json` ranks runs by quality first and speed
second.
second, and records whether the 100-artifact result is a complete
current-vs-IK comparison through `goal100Decision` and
`goal100.comparisonComplete`.
Important rule:
@@ -113,12 +132,15 @@ outside the artifact inventory/detail state.
clicking the risky lower band.
- Last/partial page planning bottom-aligns like IK, avoiding unnecessary
duplicate reads after scroll.
- Card readiness uses detail fingerprint polling:
max 420 ms, 60 ms polls, changed cards may proceed after 200 ms.
- Detail-change verification now uses the artifact OCR capture itself instead
of a separate card-ready capture before OCR.
- Scroll readiness uses inventory fingerprint polling:
max 760 ms, 80 ms polls, changed pages may proceed after 100 ms.
- Store/review writes are queued so the next tile can be clicked before disk
writes finish. The queue is still flushed before final summary.
- Store/review writes are held out of the click/capture/OCR hot path. The
latest source batches artifact store writes before the final summary instead
of issuing one save/reload cycle per artifact.
- Auto-scan artifact captures skip Electron source enumeration in the hot path
and call the GDI capture helper directly.
- Focus is done once at scan start; hot-loop captures do not refocus every tile.
Why this matters:
@@ -175,21 +197,67 @@ Latest repo validation after the recent changes:
| `npm run build` | Passed |
| `git diff --check` | Passed |
Live evidence already collected earlier on 2026-07-07:
Live evidence already collected:
- Probe click changed artifact detail successfully.
- Limit 2 live auto-scan completed with 2/2 parsed and 0 misses.
- Limit 20 live soak completed on the first visible page.
- Limit 45 live soak crossed into a scrolled page.
- On 2026-07-08, `/scanner/start?entry=visible-inventory&limit=50&engine=current`
completed `50/50` parsed and stored, `0` review, `0` duplicates, `0` misses,
`2` pages, `61765 ms` elapsed, `1235 ms/artifact`, `averageCaptureMs: 186`,
and `averageOcrMs: 162`.
- A deferred single-write flush experiment also completed `50/50`, but regressed
to `63616 ms` because `writeFlushMs` was `8163`; the source now uses batch
persist instead, pending a fresh elevated live measurement.
- After direct GDI hot-path optimization, a 20-artifact run completed
`20/20` parsed, `19` stored, `0` review, `1` duplicate, `0` misses,
`7966 ms` elapsed, `398 ms/artifact`, `averageCaptureMs: 193`, and
`averageOcrMs: 167`.
- A 45-artifact direct-GDI run completed `45/45` parsed, `42` stored,
`0` review, `3` duplicates, `0` misses, `2` pages, `18625 ms` elapsed,
`414 ms/artifact`, `averageCaptureMs: 187`, and `averageOcrMs: 162`.
- A 100-artifact direct-GDI run on signature
`2026-07-08-direct-gdi-hotpath` completed `100/100` parsed, `97` stored,
`0` review, `3` duplicates, `0` misses, `4` pages, `42064 ms` elapsed,
`421 ms/artifact`, `averageCaptureMs: 179`, and `averageOcrMs: 154`.
- `npm run eval` passed after the speed work with `23/23` exact-match cases,
`100%` field accuracy, and `100%` critical fields.
- The 3 artifacts/second target is now prepared in code but not live-proven:
artifact hot-path captures omit detail preview payloads, and stats expose
capture roundtrip/overhead timing. A qualifying 20-artifact run must finish in
`<= 6667 ms` with 0 misses and no silent OCR quality regression.
- Follow-up 3/s attempts on 2026-07-08 fixed the false review trigger caused by
omitted detail previews. The best clean `limit=20` run reached `7285 ms`
(`364 ms/artifact`, about `2.75 artifacts/second`) with `20/20` parsed,
`0` review, and `0` misses. The final stable run on
`2026-07-08-direct-gdi-reviewfix` completed `20/20` with `0` review,
`0` misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail-region capture,
5 OCR workers, DataURL buffer decode, and substat `PSM.SINGLE_COLUMN` were
tested and rejected as slower than the direct-GDI baseline.
- Final current-vs-IK-traineddata comparison on 2026-07-08:
`npm run scan:goal:compare:validated` passed. Evidence file:
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`,
`createdAt: 2026-07-08T18:41:11.6120957+02:00`. `goal100Decision` was
`qualified-comparison: winner=current`; `goal100.comparisonComplete` was true.
`current` completed `100/100` parsed, `97` stored, `0` review, `0` misses,
`4` pages, `378 ms/artifact` active average. `ik-traineddata` completed
`97/100` parsed, `92` stored, `5` review, `3` misses and was rejected for
parsing fewer artifacts than requested.
- The review queue now has a bounded corpus-growth workflow:
`npm run eval:review-candidates` writes a deduplicated, Git-ignored worklist
to `outputs/review-eval-candidates/`. This separates complete modern OCR
samples from stale captures and prevents the parser's own guess from being
promoted to ground truth without human confirmation. After manual checking,
`npm run eval:prepare-confirmed` turns one candidate plus explicit expected
labels into a paste-ready confirmed corpus snippet.
Current live limitation:
- `/health` still reports an older elevated build:
`2026-07-07-ocr-pool4-hotloop-no-refocus`.
- Current source expects:
`2026-07-07-ik32-fastsubstats-active-timing`.
- The live soak runner correctly refuses to benchmark the stale runtime.
- A UAC restart attempt was canceled, so the latest code is not yet live.
- The fast path and current-vs-IK-traineddata 100-artifact comparison are proven
in the current live environment. 3 artifacts/second is not proven; remaining
speed work needs a larger OCR or capture-pipeline change, not more click
tuning.
## Inventory Kamera Comparison
@@ -199,13 +267,14 @@ Current live limitation:
| Entry | ESC/B inventory navigation and tab click | Direct `B` path plus IK-style fallback, with preflight guards |
| Page model | 32 artifact items per page | 32 safe targets (`8 x 4`) implemented |
| Last page | Bottom-aligned partial page after scroll | Implemented in page planner |
| Item wait | About 200 ms fixed wait | Fingerprint polling, accepts changed card after 200 ms |
| Item wait | About 200 ms fixed wait | No separate wait capture; OCR capture verifies changed detail |
| Scroll wait | About 100 ms fast wait after scroll | Fingerprint polling, accepts changed page after 100 ms |
| OCR model | Native Tesseract worker queue and custom traineddata | Tesseract.js pool with current and IK-traineddata comparison path |
| Capture hot path | Direct window/screen capture without source-list scan per item | Direct GDI capture in auto-scan artifact loop |
| Field parsing | OCR plus game-data lookup | OCR plus generated lookup, GOOD keys, aliases, slot/stat constraints |
| Quality gate | Mature behavior by design and user history | Explicit benchmark/soak quality gates added |
| Diagnostics | Logs/screenshots in IK flow | Diagnostics timeline plus JSON evidence bundle |
| 100-artifact proof | Reference target | Not yet proven on latest app build |
| 100-artifact proof | Reference target | Qualified current-vs-IK comparison captured; `current` won with `100/100`, 0 review, 0 misses, 37.8s projected |
What is theoretically better than before:
@@ -220,19 +289,29 @@ What is theoretically better than before:
What is not yet proven better than IK:
- Native Tesseract speed is not integrated as the default.
- The latest code has not completed the 100-artifact live run.
- Review rate and miss rate on the user's real inventory still need the new
live report.
- 3 artifacts/second is not proven.
- Native Inventory Kamera outside this app was not re-run in the same session;
the completed comparison is against the bundled `ik-traineddata` scan engine.
## Theoretical Runtime Flow
For short iteration while tuning:
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Run `npm run scan:iterate:compare:validated:wait` from a visible artifact inventory
when starting directly after UAC, or `npm run scan:iterate:compare:validated`
if preflight already passes.
3. Inspect `scan-performance-assessment.json`, review samples, and timings if the
20-artifact comparison fails quality gates.
For the intended 100-artifact comparison:
1. Start current elevated app with `npm run dev:admin` and confirm UAC.
2. Verify `/health.appBuild.signature` matches `electron/main.ts`.
3. Warm current and IK-traineddata OCR workers.
4. Run a small bounded probe from the artifact inventory.
5. Run `npm run scan:goal:compare`.
5. Run `npm run scan:goal:compare:validated:wait` directly after UAC, or
`npm run scan:goal:compare:validated` if preflight already passes.
6. For each engine and limit (`2, 5, 20, 45, 100`):
- focus Genshin once,
- verify lookup and layout,
@@ -274,17 +353,15 @@ Expected bottleneck sequence:
## Risks and Remaining Work
1. Restart with UAC and run the latest build live.
2. Run `npm run scan:goal:compare` from a visible artifact inventory.
3. If the 100-artifact winner is not qualified, inspect:
`scan-performance-assessment.json`, review samples, diagnostic timeline, and
field timings.
4. If `ik-traineddata` wins but Tesseract.js is still slow, evaluate native
Tesseract integration.
5. Grow the eval corpus with confirmed real review samples before tightening
1. Keep `current` as the default OCR engine for now; it won the qualified
current-vs-IK-traineddata live comparison.
2. If pursuing 3 artifacts/second, focus on capture/OCR pipeline changes rather
than click timing.
3. Grow the eval corpus with confirmed real review samples before tightening
parser thresholds further.
6. Validate a positive locked-artifact sample.
7. Keep recommendations secondary until scanner quality is proven.
4. Validate a positive locked-artifact sample.
5. Keep recommendations secondary until scanner quality remains stable across
repeated live sessions.
## Definition of Done for the IK Target