genshin-assistant/docs/PROJECT.md

# Project

This document is the source of truth for project intent, scope, runtime facts, and operational expectations.

For implementation structure, see [ARCHITECTURE.md](ARCHITECTURE.md). For engineering standards, see [CONVENTIONS.md](CONVENTIONS.md).
For the latest Inventory-Kamera comparison work, see
[scanner-ik-progress-report.md](scanner-ik-progress-report.md).
For the 2026-07-09 scanner merge evidence, see [MERGE_READINESS.md](MERGE_READINESS.md).

## Project Identity

| Field | Value |
| --- | --- |
| Project name | Genshin Artifact Assistant |
| Status | Scanner baseline merged to `main`; repeatability, corpus growth, and recommendation work are next |
| Platform | Windows desktop |
| Target users | Genshin Impact players who want artifact decisions without complex optimizer setup |
| Runtime | Electron app with React UI and TypeScript |

## Vision

Genshin Artifact Assistant should make artifact management feel like a no-brainer. The user opens Genshin, runs a scan, and receives simple decisions: what is good, who can use it, what can probably be marked as trash, and which builds are currently available.

The app is not intended to replace deep min-max tools. It prioritizes time savings, confidence, and understandable recommendations over perfect theorycrafting.

## Goals

- Make artifact scanning stable enough that a normal user can trust it without babysitting every click.
- Build one local canonical Genshin data package for artifact sets, pieces, slots, stats, and characters.
- Parse artifact name, slot, main stat, substats, set, equipped state, and confidence deterministically against that package.
- Save weak or failed reads automatically as review samples and turn corrections into reusable local fixes.
- Keep the app offline-first and usable without Genshin Optimizer, Inventory Kamera, Enka, or HoYoLAB.
- Re-introduce recommendations only after the scanner base is trustworthy.

## Non-Goals

- No memory reads, process hooks, game modification, packet inspection, or anti-cheat bypassing.
- No automatic deleting, feeding, enhancing, or spending resources.
- No advanced formula editor or full power-user optimizer in the MVP.
- No cloud sync by default.

## Functional Requirements

| ID | Requirement | Priority | Status |
| --- | --- | --- | --- |
| FR-001 | List capture sources and automatically prefer the detected Genshin window when available. | Must | Implemented |
| FR-002 | Read one currently opened artifact reliably from the local screen and show its parsed result. | Must | Implemented |
| FR-003 | Generate and maintain a local canonical Genshin data package for sets, pieces, slots, stats, characters, aliases, and UI profiles. | Must | Implemented baseline |
| FR-004 | Parse artifact fields only through deterministic matching, validation, and derivation against the canonical package. | Must | Implemented baseline |
| FR-005 | Run a stable automatic inventory scan: detect grid, click tile, verify detail change, parse, store, continue, scroll, resume. | Must | Implemented for visible-inventory baseline |
| FR-006 | Save low-confidence, failed, conflicting, or stale scans automatically as review samples with reason codes. | Must | Implemented baseline |
| FR-007 | Apply local learned fixes from review corrections before every new parse. | Must | Implemented baseline |
| FR-008 | Keep the scan UI operator-friendly: main preview first, debug in modals or drawers, completion summary after scan. | Must | Implemented baseline |
| FR-009 | Provide account-level artifact triage after scanner trust is acceptable. | Should | Pending |
| FR-010 | Provide 1-3 simple build suggestions per character from owned artifacts after scanner trust is acceptable. | Should | Pending |
| FR-011 | Farming overlay for reward scans. | Later | Prototype shell |

## Non-Functional Requirements

| Category | Requirement | Measurement |
| --- | --- | --- |
| Safety | Never perform irreversible in-game actions. | Code review and manual test |
| Performance | Single artifact read should feel interactive and batch scan should not stall on false progress. | Capture latency monitored manually; auto-scan stops on blocked verification |
| IK target | First 100 artifacts should scan with accuracy at least as good as Inventory Kamera and equal or better speed. | `npm run scan:goal:compare:validated` or `npm run scan:goal:compare:validated:wait` quality-gated report |
| Privacy | Captures and parsed data stay local by default. | No remote upload in scanner path |
| Reliability | Uncertain OCR must be visible to the user. | Confidence and details view |
| Learning loop | Scanner mistakes should become reusable local review samples. | `review-samples.jsonl` |
| Maintainability | Scanner heuristics must be isolated and documented. | Parser tests, scan-loop tests, data generator, review sample pipeline |

## Tech Stack

| Area | Choice | Notes |
| --- | --- | --- |
| Desktop shell | Electron | Windows local app and overlay windows |
| Frontend | React + TypeScript + Vite | UI and client state |
| Styling | CSS with dark purple glassmorphism system | Premium fintech-inspired visual direction |
| OCR | Tesseract.js prototype plus deterministic normalization/derivation | OCR alone is not trusted as the decision source |
| Capture | Electron desktopCapturer plus Windows GDI Smart Capture | GDI path is used for Genshin Smart Capture reliability |
| Input automation | C# sidecar with elevated dev runner when needed | Live-validated for read-only inventory selection clicks; see `docs/AUTOMATION_LIVE_SCAN.md` |
| Tests | Vitest + TypeScript checks | Current validation baseline; regression samples must expand |
| Packaging | electron-builder | Configured in `package.json` |

## Runtime

| Environment | Entry Point | Notes |
| --- | --- | --- |
| Local dev | `npm run dev` | Starts Vite and Electron |
| Local dev with automation | `npm run dev:admin` | Required when `GenshinImpact.exe` is elevated; Windows blocks lower-integrity cursor/click input |
| Production build | `npm run build` | Builds React and Electron main process |
| Preview | `npm run preview` | Browser preview only; capture bridge is unavailable |

## Current State Review

### What already works

- The app can enumerate capture sources and often identify the Genshin window automatically.
- Single-artifact capture is no longer blind full-screen OCR; it produces detail crops, OCR blocks, parsed fields, confidence, and notes.
- A local canonical data package already exists in `src/data/genshinGameData.json`, generated from `genshin-db`.
- The parser already uses known sets, pieces, slots, stat aliases, set aliases, character aliases, and derived slot/set mapping.
- Review samples, learned replacements, parser notes, and stored artifacts already persist locally.
- The auto-scan loop is no longer a naive click spammer: it has preflight, verification, miss handling, page fingerprinting, and stop conditions.
- The scanner now has an Inventory-Kamera comparison path: 32 safe artifact
  targets per page, lookup-derived fields, fast OCR crop profile, current vs.
  IK-traineddata benchmark endpoint, and a quality-gated live soak runner.
- Elevated live automation is validated in the current dev environment:
  `/automation/probe-click?index=1` changed the selected artifact and
  `/scanner/start?limit=2` completed with 2/2 verified reads and 0 misses.
- A 2026-07-08 visible-inventory 50-artifact run completed cleanly with
  50/50 parsed and stored, 0 review, 0 duplicates, and 0 misses. It is stable
  but still too slow for the 2-3 artifacts/second target.
- Later 2026-07-08 direct-GDI hot-path runs completed 20/20 parsed with
  0 misses and 0 review. The best clean 20-artifact iteration reached
  7285 ms, or roughly 2.75 artifacts/second; the final stable
  `2026-07-08-direct-gdi-reviewfix` run completed in 7973 ms. The
  3 artifacts/second target remains unproven.
- The same direct-GDI path completed a 100-artifact run with 100/100 parsed,
  0 review, 0 misses, and 42064 ms elapsed across 4 pages.
- Review samples can now be exported with `npm run eval:review-candidates` into
  a Git-ignored human-labeling worklist. This is the next quality phase before
  adding more OCR corpus cases or trusting review queue data as labels.

### What is still structurally weak

- The scan experience is still partly orchestrated from `src/App.tsx`, which makes behavior changes harder than they should be.
- Broader scan soak testing has reached clean 20-, 45-, and 100-artifact runs
  with 0 misses on the current engine. The current-vs-IK-traineddata comparison
  is now captured; `current` won the qualified 100-artifact comparison on
  2026-07-08.
- OCR quality is still inconsistent enough that some fields are recovered by fallback and derivation more often than they should be.
- Learned fixes currently focus on text replacements; they do not yet update crop offsets, UI profile variants, or scanner targeting rules in a structured way.
- The scan page is cleaner than before, but it still exposes too much operator/debug state in the main flow.
- Recommendations and build logic exist, but the scanner is not yet reliable enough to make them the core focus.
- The latest source has completed the final current-vs-IK-traineddata live
  comparison for this environment. Repeatability and 3 artifacts/second are
  still open.

### Current product conclusion

The app has crossed from OCR-demo/prototype into a validated scanner baseline on
`main`. The current merge-ready path is the visible-inventory scan flow: the
operator opens Artifact inventory with a visible detail card, the app verifies
the state, scans read-only, persists parsed artifacts, and keeps uncertain data
reviewable. The next product phase is not another broad scanner rewrite; it is
repeatability, corpus growth, UI polish, and then recommendations on top of the
trusted artifact store.

## Product Direction

- Artifact scanning is the first-class feature.
- Character optimization returns only after scan quality is trustworthy.
- Team building stays out of the critical path until artifact ingestion is stable.
- Inventory Kamera remains a reference for scan choreography and page movement, not a runtime dependency.
- Self-learning stays deterministic and local first: review samples, aliases, crop offsets, and UI profile tuning before any ML retraining discussion.

## Execution Plan

### Phase 0 - Stabilize the operator surface

Outcome:
- Scan page reduced to source, main preview, result panel, primary scan actions, and compact status.
- Diagnostics, logs, crops, confidence breakdown, review queue, and learning internals moved behind modal or drawer entry points.
- Scan completion popup summarizes scanned, stored, duplicates, review samples, blocked reason, and elapsed time.

Status:
- Mostly done for the scanner baseline. The Diagnose/dev surface is separated
  and scrollable, scan summaries are compact, and the normal Auto-Scan path is
  guarded. Further UI polish remains useful but no longer blocks scanner merge.

### Phase 1 - Canonical game data package

Outcome:
- `scripts/generate-genshin-data.cjs` emits one stricter package contract for:
  - artifact sets
  - artifact pieces
  - slot-by-piece mapping
  - stats and allowed mains by slot
  - characters
  - aliases
  - UI profiles
  - source version metadata
- Parser regression tests run against saved review samples and known bad cases.
- Parser stops "free guessing" outside the canonical package.

Status:
- Implemented as a generated lookup package in `src/data/genshinGameData.json`
  with validation and parser integration. Continue regenerating and expanding
  aliases deliberately when Genshin data or OCR samples require it.

### Phase 2 - Deterministic parser hardening

Outcome:
- Name, slot, set, main stat, and equipped fields are parsed through layered validation:
  1. direct OCR cleanup
  2. alias normalization
  3. exact package match
  4. constrained fuzzy match
  5. safe derivation from piece/slot/value references
- Main stat/value inference is tightened with slot constraints and reference tables.
- Bad parses automatically generate structured review reasons.
- Equipped-character parsing is canonical-data constrained: noisy known names can
  match through aliases/fuzzy lookup, but unknown footer fragments stay
  `Not detected` instead of being stored as invented character names.

Status:
- Implemented for the merge baseline. Parser tests cover canonical set/slot/stat
  matching, equipped-character footer noise, known aliases, and unsafe one-letter
  fragments. Continue growing the confirmed review corpus before tightening
  thresholds further.

### Phase 3 - Scanner core rebuild

Outcome:
- Auto-scan becomes a dedicated engine with explicit states:
  - preflight
  - grid detection
  - click target
  - wait stable
  - detail verify
  - parse
  - store or review
  - next tile
  - row scroll
  - resume or stop
- Progress counts only when a new verified artifact or duplicate signature is confirmed.
- Repeated pages, unchanged detail cards, blocked cursor movement, and scroll failures stop the scan with diagnosis instead of producing fake progress.
- Fast artifact-read captures include the equipped footer when an equipped
  marker is visible; preflight and polling captures still skip expensive OCR.

Status:
- Implemented and merged for the visible-inventory path. Live validation on
  2026-07-09 covered `20/20` verified/parsed with `0` review and `0` misses,
  equipped-character persistence, unlocked lock state, positive locked state,
  and locked persistence. Explicit entry-mode experiments remain separate from
  the normal merge-ready path.

### Phase 4 - Input automation replacement

Outcome:
- Replace the production automation path with a persistent Windows sidecar dedicated to:
  - focus
  - move
  - click
  - scroll
  - capture
  - probe
- Session probe decides which input mode works before auto-scan is unlocked.
- Auto-scan never starts on a session that cannot prove one successful detail-card change.

Status:
- Read-only C# helper path and elevated dev startup are validated for the
  visible-inventory scanner baseline. Direct inventory, Paimon-menu, and
  auto-entry modes remain Dev-Control experiments and should be tested with low
  limits before being promoted.

### Phase 5 - Learning loop that actually compounds

Outcome:
- Weak scans save themselves as review samples automatically.
- User corrections update local:
  - text replacements
  - alias maps
  - crop offsets
  - UI profile adjustments
  - constrained set/piece/slot fixes
- "Apply learned fixes" runs before every parse.
- Review samples become both parser regression fixtures and learning inputs.

Status:
- Prepared in code for text replacements, field aliases, constrained fixes,
  crop adjustment proposals, and UI-profile adjustment proposals. Crop/profile
  changes still require live review before being auto-applied.

### Phase 6 - Recommendations come back on top of a trusted scanner

Outcome:
- Account snapshot and build suggestions are only promoted once scan quality is high enough to trust owned artifacts.
- Recommendations explain uncertainty and surface conflicts instead of pretending perfect certainty.

Status:
- Next major product area after repeatability and corpus work. Do not promote
  recommendation UX until stored artifact quality is backed by more confirmed
  review samples and repeat live scan runs.

## Immediate Next Implementation Order

1. Keep the visible-inventory scanner path as the production baseline and avoid
   promoting `auto-entry`, `direct-inventory`, or `paimon-menu` until they pass
   their own low-limit live validations.
2. Grow the confirmed OCR corpus from review samples exported by
   `npm run eval:review-candidates` and prepared through
   `npm run eval:prepare-confirmed`.
3. Repeat live scanner runs in later sessions to prove repeatability across
   pages, locked/unlocked artifacts, equipped footers, and duplicate handling.
4. Continue the optional `3 artifacts/second` work only if the next change can
   reduce OCR/capture transport time without weakening quality gates.
5. Start recommendation/product UX work only after repeat scan quality and
   confirmed corpus coverage are strong enough to trust stored artifacts.

## Open Questions

| Question | Status |
| --- | --- |
| Is the current C# helper sufficient for production packaging, or does a later Rust/C++ sidecar still materially reduce latency or packaging risk? | Open |
| When should UI-profile learning be allowed to change crop geometry automatically versus requiring review approval? | Open |
| What scan-quality threshold is high enough before recommendations should be considered user-facing again? | Open |
| Which Genshin UI languages should be supported after English once the scanner contract is stable? | Open |