feat(scanner): add native artifact pipeline
Add native IK-style capture processing, Artifact Inventory, explicit promotion and single-result review. Confirm the three live OCR corrections in the eval corpus and preserve extraction/value separation.
This commit is contained in:
AzuTear
+155
-282
@@ -1,300 +1,173 @@
|
||||
# Scanner rework status
|
||||
# Scanner Rework Status
|
||||
|
||||
Progress on the approved scanner/OCR rework. See ADR-007/008/009/010 in
|
||||
[DECISIONS.md](DECISIONS.md) for the decisions behind these. For the current
|
||||
live automation runbook, see
|
||||
[AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
|
||||
For the next result/inventory product phase, see
|
||||
[scanner-results-inventory-roadmap.md](scanner-results-inventory-roadmap.md).
|
||||
Updated: 2026-07-09
|
||||
|
||||
## Current Scanner Status
|
||||
Short current-state entry point: [CURRENT_STATUS.md](CURRENT_STATUS.md).
|
||||
|
||||
See [scanner-ik-progress-report.md](scanner-ik-progress-report.md) for the full
|
||||
report.
|
||||
## Was Es Kann
|
||||
|
||||
Current status after the 2026-07-09 merge to `main`:
|
||||
- Native IK-Erfassung ist als neuer schneller Pfad verdrahtet: Electron/React
|
||||
starten und zeigen Status, der C# Helper prueft IK-Listen, fokussiert Genshin,
|
||||
klickt das 16:9 8x4 Grid, scrollt und schreibt Detailkarten-Crops.
|
||||
- IK `inventorylists` aus Inventory Kamera 1.4.4 sind 1:1 unter
|
||||
`data/ik-inventorylists` uebernommen und werden in Builds als Resource
|
||||
mitgeliefert. Der Helper meldet Version `6.7.0` mit `61` Artifact-Sets,
|
||||
`289` Artifact-Pieces, `247` Waffen, `119` Charakteren und `715` Materialien.
|
||||
- Jeder native Run legt `manifest.json`, `capture-jobs.jsonl` und `status.json`
|
||||
im Run-Ordner an. Damit kann OCR/Parsing als nachgelagerte Queue laufen,
|
||||
ohne den Capture-Loop wieder in React/Electron zu ziehen.
|
||||
- Nach einem nativen Run kann die App `capture-jobs.jsonl` nachgelagert
|
||||
verarbeiten und `scan-results.json` plus `processing-report.json` schreiben.
|
||||
`scan-results.json` enthaelt dauerhafte Ergebnis-Eintraege mit
|
||||
Extraction-Status, Review-Zustand und bewusst deferierter Value-Auswertung.
|
||||
Diese Stufe OCRt/parst nach der Erfassung, matched native Artifact-Ergebnisse
|
||||
gegen IK `inventorylists`, zwingt IK-Konflikte in Review, verarbeitet Crops
|
||||
ueber eine bounded Queue, schreibt Parser-Feldconfidence in die Resultate und
|
||||
persistiert standardmaessig noch nicht in die DB.
|
||||
- Smart Capture reads the current artifact detail view through focused 16:9
|
||||
crops, OCR preprocessing, deterministic parser matching, and local lookup data.
|
||||
- Visible-inventory auto-scan is the production baseline: preflight, grid
|
||||
detection, focus, click, detail verification, OCR, parse, store/review, scroll,
|
||||
and summary.
|
||||
- The current live path has completed a 100-artifact run with `100/100` verified
|
||||
and parsed, `98` stored, `2` duplicates, `0` review samples, `0` misses, and
|
||||
`393 ms/artifact`.
|
||||
- Native IK live smoke passed on 2026-07-09 with runtime signature
|
||||
`2026-07-09-native-ik-visual-probe`: visual preflight ready, guarded probe
|
||||
changed the detail panel, `2/2` native artifact card crops captured,
|
||||
post-capture processing parsed `2/2`, `0` review, `0` errors, and
|
||||
`queueConcurrency: 2` without persisting to the artifact store.
|
||||
- Native 20/50/100 dry validation also passed in the same live session. The
|
||||
100-item run captured and parsed `100/100` across 4 pages with `0` errors,
|
||||
`3` correctly gated Review results, and `0` store writes. See
|
||||
`docs/NATIVE_SCANNER_VALIDATION_2026-07-09.md`.
|
||||
- Stored artifacts, review samples, local text replacements, GOOD-compatible
|
||||
import/export, scanner diagnostics, and OCR eval are implemented.
|
||||
- The scan page now has a compact `Letzte Ergebnisse` rail. It uses the latest
|
||||
native `scan-results.json` entries after post-processing and falls back to
|
||||
newest stored artifacts when no native run result is loaded. Rail rows open
|
||||
the Inventory surface for detail inspection.
|
||||
- The app now has an `Inventory` navigation view for browsing native scan
|
||||
result entries, stored artifacts, and snapshot fallback rows with filters,
|
||||
sorting, a detail panel, and native crop previews loaded from the scan run
|
||||
directory. The same view is now scoped to Artifact scanning: it shows IK
|
||||
Artifact set/piece coverage, a Native Artifact pipeline strip for capture,
|
||||
post-processing, review, promotion, and evidence, plus a per-result
|
||||
`Naechster Schritt` panel. Weapons, materials, and character details remain
|
||||
loaded data only and are intentionally hidden from the active feature UI while
|
||||
they are not scanned.
|
||||
- The live-soak runner writes JSON/CSV evidence bundles and validates quality via
|
||||
`scan-performance-assessment.json`.
|
||||
|
||||
- The scanner architecture now follows the relevant Inventory Kamera model:
|
||||
32 artifact targets per page, lookup-derived fields, fast artifact OCR profile,
|
||||
direct detail-fingerprint verification from the OCR capture, page-overlap
|
||||
planning, and batched store work.
|
||||
- The live runner can compare `current` and `ik-traineddata` engines and rejects
|
||||
runs that are fast but fail miss/review quality thresholds.
|
||||
- The final current-vs-IK-traineddata 100-artifact comparison is now proven for
|
||||
the current live environment. On 2026-07-08,
|
||||
`npm run scan:goal:compare:validated` passed with evidence at
|
||||
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`.
|
||||
`current` won with `100/100` parsed, `0` review, `0` misses, and
|
||||
`378 ms/artifact` active average. `ik-traineddata` was rejected at 100 because
|
||||
it parsed `97/100`, had `5` review and `3` misses. The next optional speed
|
||||
target remains `3 artifacts/second`, which means `333 ms/artifact` or faster
|
||||
on clean 20-artifact iterations.
|
||||
- The merge-ready default is the visible-inventory path. The app blocks normal
|
||||
guided Auto-Scan unless Artifact inventory and a visible detail card are
|
||||
detected. `auto-entry`, `direct-inventory`, and `paimon-menu` remain explicit
|
||||
Dev-Control experiments.
|
||||
- Ownership and lock-state proof is no longer theoretical: live captures parsed
|
||||
equipped characters (`Citlali`, `Linnea`), unlocked artifacts reported
|
||||
`locked: false`, a visibly locked artifact reported `locked: true`, and a
|
||||
bounded auto-scan persisted the locked/equipped state.
|
||||
- Current speed is acceptable for the next product phase. The next work should
|
||||
prioritize correct artifact content extraction, a minimal live result rail,
|
||||
scanned artifact inventory, detail evaluation, and review-safe value scoring
|
||||
before another broad speed pass.
|
||||
## Was Zuletzt Gemacht Wurde
|
||||
|
||||
## Done (implemented, unit-tested, build green)
|
||||
- Der aktive UI-Scope wurde auf Artifact scanning begrenzt.
|
||||
- Waffen, Materialien und Charakterdetails bleiben als vendored IK-Daten
|
||||
vorhanden, werden aber nicht mehr als aktive Scanner-Features in der
|
||||
Inventory UI dargestellt.
|
||||
- Die Inventory UI zeigt jetzt eine Native Artifact Pipeline:
|
||||
Scope, Native Capture, OCR Queue, Review Gate, Promotion und Evidenz.
|
||||
- Native Ergebnislabels wurden geschaerft:
|
||||
`Geparst` bedeutet extrahiert, `Stored` bedeutet persistiert, `Review`
|
||||
bedeutet unsicher und `Wert offen` bedeutet bewusst noch nicht bewertet.
|
||||
- Native Artifact Details zeigen jetzt einen `Naechster Schritt`:
|
||||
Promotion bereit, Review zuerst, bereits im Store, blockiert oder Wert spaeter.
|
||||
- Inventory Filter wurden um native Ergebnisse und speicherbare Ergebnisse
|
||||
erweitert.
|
||||
- Ein ausgewaehltes, sauberes natives Ergebnis kann jetzt nach einer zweiten
|
||||
UI-Bestaetigung promotet werden. Der Main-Prozess validiert die autoritative
|
||||
Run-Datei erneut, prueft Store-Duplikate, schreibt den Store, aktualisiert
|
||||
`scan-results.json` und protokolliert in `promotion-log.jsonl`.
|
||||
- Native Review-Ergebnisse koennen direkt im Inventory anhand des Crops editiert,
|
||||
gegen IK, kanonische Main-Werte und legale Substat-Rolls validiert sowie
|
||||
freigegeben oder abgelehnt werden. Freigaben aktualisieren `scan-results.json`,
|
||||
schreiben `review-log.jsonl` und erzeugen einen normalen Eval-Review-Sample.
|
||||
- Projekt-, Roadmap-, Architektur-, Runbook- und Checklist-Doku wurden auf
|
||||
Artifact-only Scope nachgezogen.
|
||||
|
||||
- **OCR eval harness** — `src/eval/`, `npm run eval`, gate in `npm test`. See
|
||||
[ocr-eval.md](ocr-eval.md).
|
||||
- **C# input/capture sidecar** — `native/input-helper/`, `npm run helper:build`.
|
||||
Replaces the PowerShell helper on the same JSON protocol; PowerShell remains a
|
||||
fallback. Verified end-to-end (spawn, runtime, base64 capture).
|
||||
- **Layout profiles + OCR preprocessing** — `src/lib/layoutProfile.ts` (pure
|
||||
geometry, 16:9 detection), `src/lib/ocrPreprocess.ts` (grayscale + Otsu
|
||||
binarize). main.ts now uses calibrated 16:9 detail/count/grid coordinates
|
||||
first and OCRs an upscaled + binarized copy.
|
||||
- **Card-ready gating** — `src/lib/cardReadyGate.ts` replaces the fixed 280 ms
|
||||
settle with change+stability polling; robust to animation.
|
||||
- **GOOD interop** — `src/lib/goodInterop.ts` (export + best-effort import for
|
||||
scanned records), Electron file-picker import/export, and store merge.
|
||||
- **Rescan-merge** — `src/lib/artifactMerge.ts` collapses leveled re-scan
|
||||
duplicates by a level-independent identity.
|
||||
- **Data staleness warning** — `src/lib/dataPackageStatus.ts`, surfaced in the
|
||||
Scanner Diagnose data-package line.
|
||||
- **Lock detection (experimental)** — `src/lib/lockDetection.ts`, wired into
|
||||
live capture as a read-only `locked` flag and persisted with scanned records.
|
||||
- **Elevated live automation path** — `npm run dev:admin` now starts through
|
||||
`scripts/dev-admin.ps1` and logs to `outputs/admin-start/admin-dev.log`.
|
||||
Live status confirmed `isElevated: true`, `genshinFound: true`, and
|
||||
`targetProcess: "GenshinImpact"`.
|
||||
- **Read-only click probe** — `/automation/probe-click?index=1` verified that
|
||||
the app can focus Genshin, move to a visible inventory tile, click it, and
|
||||
observe a changed detail panel fingerprint (`clicked: true`,
|
||||
`inputBlocked: false`, `changed: true`).
|
||||
- **Bounded auto-scan validation** — `/scanner/start?limit=2` completed live
|
||||
with 2 clicks, 2 verified detail views, 2 parsed artifacts, 2 stored records,
|
||||
2 review samples, and 0 misses.
|
||||
- **Auto-scan OCR performance pass** - auto-scan captures now use an artifact
|
||||
OCR mode that skips inventory-count OCR on each tile, keeps equipped-character
|
||||
OCR on the real artifact-read captures, raises the substat crop to catch
|
||||
artifact level, stores automatic review samples without full-screen/inventory screenshots, reads only the tail of large
|
||||
JSONL files, avoids review noise when only level/equipped is missing, starts
|
||||
the scan with an OCR-free preflight capture, prevents repeated startup review
|
||||
reprocessing, omits full-frame and inventory-preview Base64 payloads from tile
|
||||
captures, and applies crop-specific Tesseract page-segmentation/whitelist
|
||||
parameters.
|
||||
- **Visible-page live soak helper** - `scripts/live-soak.ps1` now drives the
|
||||
dev-control health/status, smart-capture, probe-click, bounded scan, and
|
||||
review-tail endpoints and writes evidence to `outputs/live-soak/`. On
|
||||
2026-07-07 it completed probes at indices 1 and 3 plus scan limits 2, 5, 10,
|
||||
and 20 against the elevated running app. The limit 20 run finished `done` with
|
||||
20 attempted, 20 verified, 18 parsed, 18 stored, 1 review, 1 duplicate, 1
|
||||
miss, and 1 page.
|
||||
- **Scroll/page-transition live soak** - after the helper and loop fixes,
|
||||
`scripts/live-soak.ps1 -Limits 45 -ProbeIndices 1 -SkipSmartCapture`
|
||||
completed `done` on 2026-07-07 with 45 attempted, 45 verified, 35 parsed, 35
|
||||
stored, 9 review, 1 duplicate, 9 misses, and 2 pages. This validates that the
|
||||
scanner can cross from the first visible page into a scrolled page in the live
|
||||
1920x1080 setup.
|
||||
- **Lookup package layer** - `scripts/generate-genshin-data.cjs` now emits
|
||||
normalized lookup keys, GOOD keys, piece/set/slot links, aliases, source
|
||||
version metadata, generated time, and validation summary. `src/lib/genshinLookup.ts`
|
||||
provides pure matching and validation APIs, and the scanner status/dev-control
|
||||
path exposes lookup validity. Auto-scan preflight blocks when the lookup package
|
||||
is invalid.
|
||||
- **Inventory-Kamera-style field split** - artifact detail crops now separate
|
||||
name, slot, main-stat label, main-stat value, level, substats, set effects, and
|
||||
footer. OCR uses field-specific PSM/whitelist cleanup, and the parser derives
|
||||
slot/set/main-stat through lookup constraints before falling back to review.
|
||||
- **Paimon-menu auto-entry scaffold** - auto-scan supports
|
||||
`scanEntryMode: "paimon-menu"` and `/scanner/start?entry=paimon-menu&limit=N`.
|
||||
The entry sends only read-only navigation (`ESC`, `B`, artifact-tab click),
|
||||
then requires a valid lookup, supported layout, and detected artifact grid
|
||||
before the scan loop starts. The existing visible-inventory start remains the
|
||||
fallback/debug path.
|
||||
- **OCR benchmark endpoint scaffold** - `/scanner/benchmark-ocr?limit=N` captures
|
||||
identical artifact crops with the current engine and returns timing/field counts.
|
||||
`/scanner/benchmark-ocr?engine=compare` can also compare the local
|
||||
Inventory-Kamera-traineddata Tesseract.js path when
|
||||
`genshin_fast_09_04_21.traineddata` is present in `data/tessdata`, `work/`, or
|
||||
`IK_TESSDATA_DIR`. The OCR worker pool defaults to four workers and can be
|
||||
tuned with `GAA_OCR_WORKERS=1..8`. Native Tesseract is still not the default
|
||||
and should only replace `tesseract.js` after the benchmark proves it faster
|
||||
and more accurate on the same crops.
|
||||
- **Quality-gated live comparison** - `scripts/live-soak.ps1` now supports
|
||||
goal runs for `current`, `ik-traineddata`, and `compare`, writes CSV/JSON
|
||||
summaries, groups results by limit, identifies timing bottlenecks, and rejects
|
||||
winners that miss the requested count, exceed 2% misses, or exceed 15% review.
|
||||
`npm run scan:assessment:test` verifies this ranking logic without Genshin.
|
||||
The assessment also reports `goal100Decision` and
|
||||
`goal100.comparisonComplete`, so a single-engine 100-artifact run cannot be
|
||||
misread as the final IK comparison. Use
|
||||
`npm run scan:iterate:compare:validated:wait` for the 20-artifact live
|
||||
iteration and `npm run scan:goal:compare:validated:wait` for the final proof
|
||||
when starting directly after UAC. The validator `--summary` output includes
|
||||
the assessment path and timestamp for reporting.
|
||||
- **State-polled guided entry** - the guided auto-entry waits for Inventory,
|
||||
artifact grid, and first detail card evidence instead of sleeping the full
|
||||
fixed delay every time. OCR/review/store work still starts only after artifact
|
||||
detail preflight passes.
|
||||
- **Hot-loop speed pass (2026-07-08)** - the scan loop no longer performs a
|
||||
separate card-ready capture before OCR; the artifact OCR capture itself
|
||||
verifies detail-fingerprint change. Routine click diagnostics and scan stat
|
||||
publishes are throttled. Auto-scan artifact captures no longer update the
|
||||
full preview/topbar UI on every tile. Store writes can be batched so the scan
|
||||
path avoids per-artifact save/reload churn. Auto-scan artifact captures now
|
||||
use a direct GDI hot path and skip Electron `desktopCapturer.getSources()` in
|
||||
the per-artifact loop.
|
||||
- **3/s instrumentation pass (2026-07-08)** - artifact hot-path captures omit
|
||||
the detail-preview payload, and scan stats now split inner capture time from
|
||||
end-to-end capture roundtrip time. Use `averageCaptureRoundTripMs` and
|
||||
`averageCaptureRoundTripOverheadMs` in the next `limit=20` live iteration to
|
||||
decide whether the next cut belongs in native capture transport or OCR.
|
||||
- **Repeatability and capture-overhead guardrails (2026-07-09)** -
|
||||
`scripts/live-soak.ps1` now writes capture roundtrip and roundtrip-overhead
|
||||
timing into `scan-performance-assessment.json`. The assessment validator can
|
||||
enforce optional speed budgets with `--max-active-average-ms` and
|
||||
`--max-capture-roundtrip-overhead-ms`, and `npm run scan:repeatability:wait`
|
||||
runs 20/45/100 current-engine passes as repeatability evidence without
|
||||
presenting them as an IK comparison. `-RepeatabilityRun` now sets those limits
|
||||
inside PowerShell, and the script refuses unsafe limits above 1800 so npm/cmd
|
||||
argument parsing cannot accidentally turn `20,45,100` into one oversized run.
|
||||
- **Distinctive partial piece recovery (2026-07-09)** - a live repeatability run
|
||||
exposed four identical OCR misses where the piece name was read as
|
||||
`Wontiroms Creation pan`. The parser now derives a piece only when a long OCR
|
||||
fragment uniquely matches exactly one known artifact piece. This recovered the
|
||||
local case as `Sharpness That Ceased Upon Wondrous Creation` /
|
||||
`Disenchantment in Deep Shadow` without adding a broad fuzzy exception.
|
||||
- **Repeatability live pass after parser fix (2026-07-09)** -
|
||||
`outputs/live-soak/2026-07-09T09-29-11/scan-performance-assessment.json`
|
||||
captured a clean current-engine 20-artifact run: `20/20` parsed, `0` review,
|
||||
`0` misses, `336 ms/artifact` active average, `318 ms` average capture
|
||||
roundtrip, and `138 ms` average roundtrip overhead. The strict 3 artifacts per
|
||||
second budget still failed by 3 ms (`336 ms` vs `333 ms`).
|
||||
- **3/s follow-up experiments (2026-07-09)** - tested and rejected several
|
||||
shortcut-style optimizations because live runs got slower or added risk:
|
||||
skipping Paimon-menu analysis, skipping lock-state as a production shortcut,
|
||||
reducing the artifact-level crop scale, and raising the OCR worker pool to 6.
|
||||
The kept low-risk changes are fast-profile OCR crop priority and avoiding a
|
||||
duplicate DataURL string when the native helper already returns Base64. A
|
||||
follow-up clean 20-artifact run after payload cleanup reached `351 ms/artifact`,
|
||||
`331 ms` capture roundtrip, and `146 ms` roundtrip overhead, so the next
|
||||
credible 3/s work is native capture transport/roundtrip reduction, not UI
|
||||
recommendation work.
|
||||
- **3/s live attempt (2026-07-08)** - the missing-detail-preview review trigger
|
||||
was fixed and tested. The best clean 20-artifact run reached `7285 ms`
|
||||
(`364 ms/artifact`, about `2.75 artifacts/second`) with 0 review and 0 misses.
|
||||
The final stable run on `2026-07-08-direct-gdi-reviewfix` completed `20/20`
|
||||
with 0 review, 0 misses, and `7973 ms` elapsed (`399 ms/artifact`). Detail
|
||||
region capture, 5 OCR workers, DataURL buffer decode, and substat
|
||||
`PSM.SINGLE_COLUMN` were tested and rejected as slower.
|
||||
- **Review-to-eval loop (2026-07-08)** - `npm run eval:review-candidates`
|
||||
exports the local review queue into `outputs/review-eval-candidates/` as a
|
||||
human-labeling worklist. The exporter deduplicates samples, surfaces complete
|
||||
fast-field captures first, marks stale captures, and now surfaces equipped
|
||||
footer OCR plus `locked=true/false` payload counts for the next ownership/lock
|
||||
validation pass. Its output is deliberately
|
||||
ignored by Git and must not be treated as ground truth until fields are
|
||||
confirmed against the real artifact. Confirmed review labels now have a
|
||||
dedicated corpus file, `src/eval/corpus/confirmedReviewCorpus.ts`, with tests
|
||||
that reject duplicate ids, empty labels, and unconfirmed entries. The helper
|
||||
`npm run eval:prepare-confirmed` generates a paste-ready confirmed-case
|
||||
snippet only when explicit expected labels are provided.
|
||||
- **Prepared ownership/learning loop (2026-07-08)** - fast auto-scan no longer
|
||||
drops the artifact footer by profile alone; it omits footer OCR only when the
|
||||
capture option explicitly requests that or when the footer marker is absent.
|
||||
Parser tests cover noisy equipped names, split `Equipped:`/name footers, and
|
||||
one-letter OCR fragments that must stay `Not detected`. Scanner learning now
|
||||
persists text replacements, field aliases, constrained fixes, crop adjustment
|
||||
proposals, and UI-profile adjustment proposals instead of truncating everything
|
||||
back to text replacements.
|
||||
- **Visible-inventory merge guard (2026-07-09)** - the normal guided Auto-Scan
|
||||
start no longer falls back into `auto-entry` when the artifact detail card is
|
||||
missing. It now blocks and asks the operator to open the Artifact inventory
|
||||
with a visible detail card. The explicit `auto-entry`, `direct-inventory`, and
|
||||
`paimon-menu` Dev-Control modes remain available for targeted experiments, but
|
||||
they are not the merge-ready default path.
|
||||
- **Ownership live smoke (2026-07-09)** - live artifact detail capture parsed
|
||||
and stored an equipped footer as `equipped: "Citlali"` and the grey lock state
|
||||
as `locked: false`. A same-session visible-inventory run with
|
||||
`/scanner/start?entry=visible-inventory&limit=20&engine=current` completed
|
||||
`20/20` verified and parsed, `19` stored, `1` duplicate, `0` review, and
|
||||
`0` misses in `8047 ms` elapsed (`402 ms/artifact`).
|
||||
- **Locked artifact live proof (2026-07-09)** - a visibly locked artifact was
|
||||
selected through a read-only inventory tile click. Smart Capture reported
|
||||
`locked: true` with `lockSignal.ratio: 0.14797913950456323` over threshold
|
||||
`0.06`, and `/scanner/start?entry=visible-inventory&limit=1&engine=current`
|
||||
persisted the same artifact with `equipped: "Citlali"` and `locked: true`.
|
||||
Lock detection now decodes the lock crop PNG before measuring active lock
|
||||
pixels because Electron's native bitmap channel order was ambiguous in live
|
||||
captures.
|
||||
## Was Noch Nicht
|
||||
|
||||
## Next product phase - result rail and inventory
|
||||
- Native IK-Erfassung captured aktuell Karten-Crops; OCR/Parser laufen erst
|
||||
nachgelagert ueber den Processor. GOOD-Speichern und Evaluierung laufen noch
|
||||
nicht automatisch im nativen Pipeline-Nachgang.
|
||||
- Native IK-Erfassung ist aktuell auf das sichtbare 16:9 Artifact-Inventar
|
||||
begrenzt.
|
||||
- The result rail is still a foundation: it can show native `scan-results.json`
|
||||
entries after post-processing, but does not yet calculate or show real value
|
||||
scores.
|
||||
- The inventory browser is still a foundation: value scoring and richer
|
||||
review/edit flows are not complete.
|
||||
- Upgrade projection and build recommendations should wait until inventory and
|
||||
detail evaluation have trustworthy stored artifacts.
|
||||
- Repeatability across later sessions, more accounts, more locked/equipped
|
||||
combinations, and more confirmed OCR corpus cases still needs growth.
|
||||
- The strict `333 ms/artifact` budget for 3 artifacts/second is not proven.
|
||||
- Weapons, materials, and character details are intentionally out of active
|
||||
scope until their values are actually scanned.
|
||||
|
||||
The next implementation pass is planned in
|
||||
[scanner-results-inventory-roadmap.md](scanner-results-inventory-roadmap.md).
|
||||
Summary:
|
||||
## Wo Es Noch Probleme Macht
|
||||
|
||||
1. Add result data contracts that preserve extraction confidence separately from
|
||||
artifact value score.
|
||||
2. Rework the scan view into screenshot/preview plus a compact right-side live
|
||||
result rail.
|
||||
3. Add a scanned artifact inventory menu with minimal list/grid rows and
|
||||
score/review pills.
|
||||
4. Add click-through artifact detail with screenshot/crops, parsed fields, OCR
|
||||
confidence, value reasons, and review state.
|
||||
5. Add upgrade projection later as a detail-only feature with worst/middle/best
|
||||
cases and explicit uncertainty.
|
||||
6. Defer the full screenshot queue/worker pipeline until the result/inventory
|
||||
contracts are stable or timing evidence shows the current loop is the blocker.
|
||||
- Die native Pipeline hat jetzt 20/50/100-Skalenevidenz aus einer Live-Session.
|
||||
Spaetere Session-Repeatability und Packaged-App-Evidenz stehen noch aus.
|
||||
- Capture roundtrip and OCR remain the main timing costs in the 100-artifact
|
||||
path.
|
||||
- Auto-scan still requires a visible artifact inventory detail card for the
|
||||
production path; broader auto-entry modes remain dev experiments.
|
||||
- OCR/parser quality is good on the confirmed corpus, but review samples must be
|
||||
manually labeled before they can become permanent eval ground truth.
|
||||
- Non-16:9 or unusual game layouts are intentionally higher risk and should
|
||||
block or fall into review instead of silently scanning.
|
||||
- The UI now labels these risks in the Artifact Inventory pipeline. Same-session
|
||||
scale is proven, but external-tool parity and later-session repeatability are
|
||||
intentionally not claimed.
|
||||
|
||||
## Remaining - needs the live environment or a UI pass
|
||||
## Was Noch Verbesserungsfaehig Ist
|
||||
|
||||
These cannot be finished/validated without Genshin running at the user's
|
||||
resolution or without UI work best tested live:
|
||||
- Repeat the bounded 20/50/100 native runs in a later session and in a packaged
|
||||
app to verify that the current post-capture queue remains repeatable.
|
||||
- Keep native persistence explicit: clean selected results can be promoted
|
||||
after confirmation, while uncertain results must pass single-item review.
|
||||
- Finish value scoring and value-reason contracts in the existing detail view from
|
||||
`docs/scanner-results-inventory-roadmap.md`.
|
||||
- Keep diagnostics available but out of the primary scan surface.
|
||||
- Continue expanding confirmed OCR eval cases from real review samples. The
|
||||
first three native Review cases are corrected, approved, and permanent
|
||||
regression cases.
|
||||
- Add artifact value/detail evaluation without collapsing extraction confidence
|
||||
and artifact value into one ambiguous status.
|
||||
- Continue performance work only when it reduces capture/OCR overhead without
|
||||
weakening review, miss, duplicate, or safety gates.
|
||||
|
||||
1. **Validate/tune OCR preprocessing** on more real captures — confirm invert +
|
||||
threshold + upscale factor help (not hurt) actual Tesseract reads. The
|
||||
text-level eval harness cannot measure image preprocessing.
|
||||
2. **Wire and benchmark native IK-traineddata OCR** against the same crop set.
|
||||
The current benchmark can use IK-traineddata through Tesseract.js; native
|
||||
Tesseract integration remains the next implementation step before any engine
|
||||
default changes.
|
||||
3. **Validate explicit entry modes separately** from world, direct inventory,
|
||||
and Paimon/menu states with low limits only. These are now Dev-Control
|
||||
experiments, not the normal merge path; the normal Auto-Scan button blocks
|
||||
unless the visible artifact detail card is already present.
|
||||
4. **Repeat locked=true on another page/session** if lock behavior changes.
|
||||
The first positive live proof passed on 2026-07-09, including store
|
||||
persistence. Further repeats are useful for confidence but no longer block
|
||||
the merge.
|
||||
## Was Als Naechstes Ansteht
|
||||
|
||||
5. **3 artifacts/second iteration** - not reached yet. The next credible path is
|
||||
either native Tesseract/IK-traineddata integration that materially reduces
|
||||
substat OCR time, or a larger capture pipeline change that avoids full-frame
|
||||
PNG/Base64 transport without hurting safety checks. The target remains
|
||||
`<= 6667 ms` elapsed for 20 parsed artifacts with 0 misses and no silent OCR
|
||||
review regression. The latest clean 20-artifact repeatability run reached
|
||||
`336 ms/artifact`, so 3/s remains close but unproven.
|
||||
1. Artifact Value Evaluation ergaenzen, aber erst nach sauberer Extraktion und
|
||||
weiterhin getrennt von OCR/Parser-Confidence.
|
||||
2. Detail-Ansicht um Value-Gruende und optionale Upgrade-Projektion erweitern.
|
||||
3. Packaged-App-Verhalten fuer Helper, IK-Listen, Preload-Bridge,
|
||||
Crop-Preview und Smoke-Kommandos pruefen.
|
||||
4. Empfehlungen und Build-UX erst danach wieder nach vorne ziehen.
|
||||
|
||||
6. **Broader scan soak test** — direct-GDI current-engine runs now passed at
|
||||
`20/20`, `45/45`, and `100/100` with 0 misses. Continue with
|
||||
`npm run scan:repeatability:wait` in later sessions to check duplicate rate,
|
||||
scroll behavior, and capture roundtrip timing without changing defaults.
|
||||
7. **Repeatability pass** — repeat the qualified current-vs-IK-traineddata run
|
||||
in a later live session before making major OCR-engine defaults or speed
|
||||
claims beyond this environment. Current-engine-only repeatability is useful
|
||||
evidence, but it is not an IK parity claim.
|
||||
## Current Validation Commands
|
||||
|
||||
Visible-page limits up to 20, scroll/page-transition limit 45, the final
|
||||
100-artifact current-vs-IK-traineddata comparison, equipped footer live smokes,
|
||||
and one positive locked-artifact persistence proof have passed for the current
|
||||
environment. Remaining soak work is repeatability, OCR corpus growth, additional
|
||||
equipped/locked repeats, and optional 3 artifacts/second speed work.
|
||||
```powershell
|
||||
npm run scan:live:preflight
|
||||
npm run scan:iterate:validated
|
||||
npm run scan:goal:validated
|
||||
npm run scan:repeatability:wait
|
||||
npm run scan:assessment:validate -- --latest --summary
|
||||
npm run eval
|
||||
npm test
|
||||
npm run build
|
||||
```
|
||||
|
||||
## Grow the eval corpus
|
||||
Latest static validation after the Artifact-only Inventory UI update:
|
||||
|
||||
Every low-confidence review sample already stores its crops + OCR. Confirm/correct
|
||||
those via `reviewSampleToEvalCase` and commit them into `src/eval/corpus/` so the
|
||||
harness keeps measuring real-world accuracy across patches. See
|
||||
[ocr-eval.md](ocr-eval.md).
|
||||
```powershell
|
||||
npm run lint # passed
|
||||
npm test # passed, 252 tests
|
||||
npm run build # passed
|
||||
git diff --check
|
||||
```
|
||||
|
||||
`git diff --check` passed with existing CRLF warnings for
|
||||
`electron/services/inputHelperPowerShellFallback.ts` and `src/styles/base.css`.
|
||||
|
||||
Reference in New Issue
Block a user