feat(scanner): add native artifact pipeline

Add native IK-style capture processing, Artifact Inventory, explicit promotion and single-result review. Confirm the three live OCR corrections in the eval corpus and preserve extraction/value separation.
2026-07-09 23:30:42 +02:00
parent 28d60eb915
commit 639b0b7f59
92 changed files with 13606 additions and 1703 deletions
@@ -6,18 +6,27 @@ This document describes the structure, boundaries, flows, and technical rules of

 Genshin Artifact Assistant is a local desktop application. Electron owns OS integration, screen capture, IPC, and overlay windows. React owns the interactive UI. Domain logic for OCR parsing, scoring, scanner state, and recommendations lives in TypeScript modules under `src/lib`.

+The scanner performance direction is now native-first: the C# input helper owns
+the fast capture/click/scroll loop and uses vendored Inventory Kamera
+`inventorylists` as the scanner dictionary source. Electron is the process,
+IPC, packaging, hotkey, and dev-control shell. React is only the visual control
+and status surface for this path.
+
 ```mermaid
 flowchart LR
    User["User"]
    Genshin["Genshin Impact Window"]
    Electron["Electron Main Process"]
+    Native["C# Input Helper / Native Scanner"]
    React["React Renderer"]
    Parser["OCR Parser and Scoring"]
    LocalData["Local Snapshot / Future SQLite"]

    User --> React
    React --> Electron
-    Electron --> Genshin
+    Electron --> Native
+    Native --> Genshin
+    Native --> Electron
    Electron --> React
    React --> Parser
    Parser --> React
@@ -53,16 +62,24 @@ flowchart LR
 | `electron/appWindowManager.ts` | Main window and overlay window lifecycle, menu-bar removal, dashboard focus behavior, and renderer window command delivery |
 | `electron/services/inputHelper.ts` | Stable JSON protocol client for the compiled C# input/capture sidecar plus PowerShell fallback startup |
 | `electron/services/inputHelperPowerShellFallback.ts` | PowerShell fallback script body for environments where the compiled helper is unavailable |
+| `electron/services/nativeScannerProcessingService.ts` | Post-capture processor for native IK runs; reads crop jobs, OCRs/parses card crops, matches IK metadata, writes processing outputs, and safely loads native crop previews |
+| `electron/services/nativeScannerResultWorkflowService.ts` | Authoritative promotion and review workflow for native results; reloads run state, validates edits, updates durable results/logs, and only writes the artifact store after explicit confirmation |
 | `electron/services/goodFileService.ts` | Local GOOD export file writing and GOOD import file dialog/read handling |
 | `electron/preload.cjs` | Safe renderer bridge exposed as `window.assistantApi` |
+| `native/input-helper/IkInventoryLists.cs` | Loads and validates the vendored IK `inventorylists` feature catalog |
+| `native/input-helper/NativeScannerFiles.cs` | Writes native scanner manifest, status, and JSONL crop-job files |
 | `src/lib/artifactStore.ts` | Pure signature/id/record helpers for the persistent artifact store |
+| `src/lib/ikArtifactMatcher.ts` | Pure IK inventorylist matcher for native artifact set/piece/slot validation and GOOD key metadata |
+| `src/lib/ikCatalogMatcher.ts` | Pure IK inventorylist matcher for simple weapon, character, and material names plus compact GOOD-key samples |
+| `src/lib/ikScanCapabilities.ts` | Pure capability summary that separates IK catalog coverage from implemented native capture support |
+| `src/lib/scanResultEntry.ts` | Pure durable scan-result entry/status helpers that keep extraction confidence separate from deferred artifact value evaluation |
 | `src/App.tsx` | Thin React entry that renders the app page |
 | `src/pages/AppPage.tsx` and `src/pages/app/*` | App page composition and high-level layout routing |
 | `src/features/scan/hooks/useScanViewController.ts` | Scan feature state composition and view-controller assembly |
 | `src/features/scan/hooks/scanViewScanActions.ts` | Manual scan and visible-grid scan orchestration |
-| `src/features/scan/hooks/scanViewEntryActions.ts` | Guided auto-entry choreography for visible inventory, direct inventory, and IK-style fallback paths |
+| `src/features/scan/hooks/scanViewEntryActions.ts` | Guided auto-entry choreography for visible inventory, direct inventory, and ESC/B fallback paths |
 | `src/features/scan/hooks/useScanGoodInterop.ts` | Scan-page GOOD import/export actions against renderer repository ports |
-| `src/features/inventory/*` | Planned scanned-artifact inventory, compact result list/grid, filters, and artifact detail views |
+| `src/features/inventory/*` | Scanned-artifact inventory browser for native result entries, stored artifacts, filters, sorting, compact detail state, native crop preview display, and IK catalog status |
 | `src/lib/artifactOcrParser.ts` | Converts OCR output into a parsed artifact candidate with confidence and notes |
 | `src/lib/fuzzyMatch.ts` | Generic fuzzy string matching for OCR text against known game data |
 | `src/lib/genshinLookup.ts` | Pure lookup and validation API for generated Genshin data |
@@ -72,6 +89,7 @@ flowchart LR
 | `src/lib/upgradeProjection.ts` | Planned best/middle/worst upgrade projection for under-leveled artifacts |
 | `src/lib/scoring.ts` | Recommendation and build scoring logic |
 | `src/lib/demoData.ts` | Temporary local demo snapshot |
+| `data/ik-inventorylists/*` | 1:1 vendored Inventory Kamera inventory lists used by the native scanner data preflight and future matching |
 | `src/data/genshinGameData.json` | Generated local dictionary of characters, artifact sets, slots, and stats |
 | `scripts/generate-genshin-data.cjs` | Regenerates the local Genshin dictionary from `genshin-db` |
 | `src/types/*` | Shared app, capture, and domain contracts |
@@ -117,9 +135,14 @@ sequenceDiagram
 | Capture sources | Electron desktopCapturer | Electron main | Scan UI |
 | Screenshot | Windows GDI / desktopCapturer | Electron main | Cropper, OCR, UI preview |
 | OCR crops | Electron main | Electron main | Details modal, parser |
+| Native card crops | C# input helper | Native scanner run directory under app userData | Electron status, inventory preview, later OCR/parse queue |
+| IK inventorylists | `data/ik-inventorylists` copied from Inventory Kamera 1.4.4 | Source data package | Native scanner data preflight and future matcher |
+| IK simple catalog match | IK weapon/character/material inventorylist catalog | `IkCatalogItemMatch` | Future category scanners, inventory catalog evidence |
 | Game dictionary | `genshin-db` generated JSON | `src/data/genshinGameData.json` | OCR parser |
-| Parsed artifact candidate | OCR parser | Renderer domain logic | Result panel, future local DB |
-| Scan result entry | Scan loop and parser/evaluator | Renderer domain logic | Live scan rail, artifact inventory, summary |
+| Parsed artifact candidate | OCR parser | Renderer domain logic | Result panel, post-capture report, future local DB |
+| IK artifact match | IK artifact inventorylist catalog | `ScanResultIkMatch` | Native post-capture review gate, scan result entry, inventory detail |
+| Parser field confidence | Native post-capture parser | `ScanResultFieldConfidence` | Scan result entry, inventory detail review signal |
+| Scan result entry | Scan loop or native post-capture processor | `StoredScanResultEntry` / native run `scan-results.json` | Live scan rail, artifact inventory, summary |
 | Artifact evaluation | Deterministic evaluator | `src/lib` | Result pills, inventory sort/filter, detail reasons |
 | Upgrade projection | Projection helper | `src/lib` | Artifact detail view only |
 | Review samples | User action in Scan UI | Electron userData `review-samples.jsonl` | Future regression tests and OCR training |
@@ -147,15 +170,15 @@ sequenceDiagram
 - SendInput's return value is checked: zero injected events (UIPI, e.g. elevated Genshin vs. non-elevated app) aborts with an explicit hint instead of silently clicking into nothing.
 - Dev-only probes under `http://127.0.0.1:17317` are used for live validation:
  `/automation/probe-click?index=N` tests one read-only tile selection, and
-  `/scanner/start?entry=visible-inventory&limit=N` starts an auto-scan with a
-  temporary limit payload from an already visible artifact detail view.
+  `/scanner/start?limit=N` starts the native capture scan with a temporary
+  limit payload from an already visible artifact detail view.
  The live known-good result on 2026-07-07 is documented in
  [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).
 - Click verification: after each click the parsed detail-panel signature should change. An unchanged signature is a soft miss (it can also mean two OCR-identical neighbor pieces, common among +0 artifacts), so it is retried once with a small offset, logged with the stuck artifact name, and then skipped - never fatal on its own. The scan aborts only when the first ~6 clicks of page 1 produce nothing new (diagnosis hint: elevated Genshin blocks SendInput via UIPI, or grid coordinates are wrong) or a later page yields zero new artifacts.
 - Scan stats separate clicked (click attempts), parsed (readable captures), stored (persisted), review (review samples), duplicates, and misses, so "scanned" cannot be mistaken for "successfully read".
- Artifact grid automation uses Inventory Kamera's 32-target full-page model
-  (`8 x 4` safe click targets). The apparent lower fifth row sits in the
-  bottom control band on 16:9 captures and is not clicked automatically.
+- Artifact grid automation uses 32 safe click targets per full page
+  (`8 x 4`). The apparent lower fifth row sits in the bottom control band on
+  16:9 captures and is not clicked automatically.
 - Guided auto-entry is state gated. The normal scan button first performs a
  lightweight no-OCR preflight; OCR/store/review work starts only after the
  artifact inventory grid and artifact detail card are visually confirmed.
@@ -167,12 +190,54 @@ sequenceDiagram
 - Item verification uses the artifact OCR capture's own detail fingerprint, so
  the loop no longer performs a separate card-ready capture before OCR. Page
  waits remain fingerprint based and can proceed as soon as the inventory pane
-  changes and stabilizes, while still accepting IK-like 100 ms scroll readiness
-  points.
+  changes and stabilizes.
 - Scrolling sends one wheel notch per grid row with the cursor anchored over the inventory (assumption: roughly one row per notch; overlap is absorbed by dedupe, and a page without new artifacts stops the scan).
 - The scan never deletes, enhances, feeds, locks, or spends anything; it only selects tiles to read them.

-Parsed artifacts from both modes are persisted into `artifact-store.json` keyed by a content signature that excludes the equipped character, so re-equipping updates a record instead of duplicating it. Leveling an artifact currently creates a new record (documented limitation until rescan-merge exists).
+**Native IK capture scan** is the new high-speed path. It runs inside the C#
+sidecar, computes a fixed 16:9 8x4 visible-inventory grid, clicks and scrolls
+natively, captures the artifact detail card as PNG crops, and reports progress
+through IPC/dev-control. The renderer only starts, stops, and displays status.
+OCR, parsing, GOOD persistence, and artifact value evaluation are deliberately
+separate follow-up stages so capture speed is not blocked by UI work.
+
+Each native run writes a self-contained run directory under app userData:
+
+- `manifest.json`: run schema, IK data version/categories, Genshin bounds,
+  grid, detail crop rectangle, explicit scan category, and downstream queue
+  contract.
+- `capture-jobs.jsonl`: one job per captured card crop with page/row/column,
+  client/screen coordinates, click event count, image path, and downstream
+  `ocr-parse-store` marker. Jobs include the scan category; today only
+  `artifacts` produces jobs. Native preflight and post-capture processing also
+  guard this category boundary so catalog-only weapon, character, and material
+  entries cannot be accidentally parsed as artifacts.
+- `status.json`: latest scanner status snapshot for recovery and dev tooling.
+  Its `supportedCategories` block separates IK catalog availability from native
+  capture support: artifacts are the only native-capture category today, while
+  weapons, characters, and materials are catalog-only until their own capture
+  flows have evidence.
+- `scan-results.json`: durable per-artifact result entries with capture
+  metadata, parsed artifact identity when available, extraction status, and
+  value status. Native capture currently writes `deferred` value status for
+  clean extraction and `review` for uncertain extraction.
+- `processing-report.json`: optional post-capture OCR/parse report generated
+  from `capture-jobs.jsonl`. It records `queueConcurrency` for the bounded
+  post-capture OCR/parse worker and is non-persisting by default until native
+  crop OCR has live validation evidence.
+
+Parsed artifacts from manual and renderer auto-scan modes are persisted into
+`artifact-store.json` keyed by a content signature that excludes the equipped
+character, so re-equipping updates a record instead of duplicating it. Leveling
+an artifact currently creates a new record (documented limitation until
+rescan-merge exists). Native IK capture currently writes card crops first; a
+downstream OCR/parse processor can report parsed artifacts from those crops
+without slowing capture. Store promotion remains opt-in until native crop OCR is
+validated.
+The Inventory surface derives a dry-run promotion summary from native
+`scan-results.json` entries and the local artifact store. It can show which
+native artifacts are ready for explicit promotion, already stored, review-only,
+or blocked, but it does not write store records by itself.
 During auto-scan, artifact store writes can be batched and flushed after the
 click/capture/OCR loop to avoid per-artifact save/reload churn in the hot path.
 Auto-scan artifact captures also bypass Electron source-list enumeration and use
@@ -192,10 +257,19 @@ Architecture rules:
  OCR/debug stats stay in diagnostics or detail.
 - A scan result row preserves extraction status and artifact value status as
  separate data even when the UI shows one compact pill.
- The artifact inventory view owns browsing, filtering, sorting, and opening
-  detail.
+- The artifact inventory view owns browsing, filtering, sorting, opening detail,
+  and showing the current artifact-only native pipeline state.
+- Weapons, materials, and character details stay out of the active Inventory UI
+  until their values are actually scanned; loaded IK catalog data alone is not a
+  user-facing scanner capability.
 - The artifact detail view owns screenshot/crop inspection, OCR confidence,
  parser notes, value score reasons, and upgrade projection.
+- Native crop previews are served through the Electron bridge only for PNG paths
+  inside the selected native scanner run directory.
+- Native artifact post-processing uses IK artifact set/piece/slot matching as a
+  review gate. A conflict is extraction uncertainty, not a weak artifact value.
+- Native artifact post-processing stores per-field parser confidence so detail
+  review can explain which OCR/parser fields are trustworthy.
 - Upgrade projection is a local deterministic/probabilistic helper, never a
  claim that an artifact will roll a specific way.

@@ -228,8 +302,9 @@ Future queue refactor:
 - OCR/parse/evaluation may process bounded queued screenshot/crop jobs.
 - Queueing must preserve stop behavior, duplicate handling, review decisions,
  and the existing read-only safety boundary.
- The queue refactor is secondary to content extraction and result/inventory
-  contracts while current scan speed remains acceptable.
+- The native post-capture processor already consumes crop jobs with bounded
+  parallelism while preserving report order. Further queue work is secondary to
+  live throughput evidence and result quality.

 ## Security And Safety

@@ -251,10 +326,8 @@ Future queue refactor:

 ## Performance

-Current OCR is still measured against the IK target rather than assumed good.
-The app keeps a Tesseract.js worker pool, can use the Inventory-Kamera
-`genshin_fast_09_04_21.traineddata` path for comparison, and reports capture,
-OCR, card-ready, scroll-ready, active-scan, and projected-100 timings. A default
-engine change requires a same-capture benchmark and a qualified live soak result.
+Current OCR is measured through the eval harness and live scan assessments
+rather than assumed good. The app keeps a Tesseract.js worker pool and reports
+capture, OCR, card-ready, scroll-ready, active-scan, and projected-100 timings.
 For the next product phase, performance work should not displace extraction
 quality, result clarity, or review safety unless evidence shows a regression.