genshin-assistant/docs/PROJECT.md

Project

This document is the source of truth for project intent, scope, runtime facts, and operational expectations.

For implementation structure, see ARCHITECTURE.md. For engineering standards, see CONVENTIONS.md. For the latest Inventory-Kamera comparison work, see scanner-ik-progress-report.md. For the 2026-07-09 scanner merge evidence, see MERGE_READINESS.md.

Project Identity

Field	Value
Project name	Genshin Artifact Assistant
Status	Scanner baseline merged to main; repeatability, corpus growth, and recommendation work are next
Platform	Windows desktop
Target users	Genshin Impact players who want artifact decisions without complex optimizer setup
Runtime	Electron app with React UI and TypeScript

Vision

Genshin Artifact Assistant should make artifact management feel like a no-brainer. The user opens Genshin, runs a scan, and receives simple decisions: what is good, who can use it, what can probably be marked as trash, and which builds are currently available.

The app is not intended to replace deep min-max tools. It prioritizes time savings, confidence, and understandable recommendations over perfect theorycrafting.

Goals

• Make artifact scanning stable enough that a normal user can trust it without babysitting every click.
• Build one local canonical Genshin data package for artifact sets, pieces, slots, stats, and characters.
• Parse artifact name, slot, main stat, substats, set, equipped state, and confidence deterministically against that package.
• Save weak or failed reads automatically as review samples and turn corrections into reusable local fixes.
• Keep the app offline-first and usable without Genshin Optimizer, Inventory Kamera, Enka, or HoYoLAB.
• Re-introduce recommendations only after the scanner base is trustworthy.

Non-Goals

• No memory reads, process hooks, game modification, packet inspection, or anti-cheat bypassing.
• No automatic deleting, feeding, enhancing, or spending resources.
• No advanced formula editor or full power-user optimizer in the MVP.
• No cloud sync by default.

Functional Requirements

ID	Requirement	Priority	Status
FR-001	List capture sources and automatically prefer the detected Genshin window when available.	Must	Implemented
FR-002	Read one currently opened artifact reliably from the local screen and show its parsed result.	Must	Implemented
FR-003	Generate and maintain a local canonical Genshin data package for sets, pieces, slots, stats, characters, aliases, and UI profiles.	Must	Implemented baseline
FR-004	Parse artifact fields only through deterministic matching, validation, and derivation against the canonical package.	Must	Implemented baseline
FR-005	Run a stable automatic inventory scan: detect grid, click tile, verify detail change, parse, store, continue, scroll, resume.	Must	Implemented for visible-inventory baseline
FR-006	Save low-confidence, failed, conflicting, or stale scans automatically as review samples with reason codes.	Must	Implemented baseline
FR-007	Apply local learned fixes from review corrections before every new parse.	Must	Implemented baseline
FR-008	Keep the scan UI operator-friendly: main preview first, debug in modals or drawers, completion summary after scan.	Must	Implemented baseline
FR-009	Provide account-level artifact triage after scanner trust is acceptable.	Should	Pending
FR-010	Provide 1-3 simple build suggestions per character from owned artifacts after scanner trust is acceptable.	Should	Pending
FR-011	Farming overlay for reward scans.	Later	Prototype shell

Non-Functional Requirements

Category	Requirement	Measurement
Safety	Never perform irreversible in-game actions.	Code review and manual test
Performance	Single artifact read should feel interactive and batch scan should not stall on false progress.	Capture latency monitored manually; auto-scan stops on blocked verification
IK target	First 100 artifacts should scan with accuracy at least as good as Inventory Kamera and equal or better speed.	npm run scan:goal:compare:validated or npm run scan:goal:compare:validated:wait quality-gated report
Privacy	Captures and parsed data stay local by default.	No remote upload in scanner path
Reliability	Uncertain OCR must be visible to the user.	Confidence and details view
Learning loop	Scanner mistakes should become reusable local review samples.	review-samples.jsonl
Maintainability	Scanner heuristics must be isolated and documented.	Parser tests, scan-loop tests, data generator, review sample pipeline

Tech Stack

Area	Choice	Notes
Desktop shell	Electron	Windows local app and overlay windows
Frontend	React + TypeScript + Vite	UI and client state
Styling	CSS with dark purple glassmorphism system	Premium fintech-inspired visual direction
OCR	Tesseract.js prototype plus deterministic normalization/derivation	OCR alone is not trusted as the decision source
Capture	Electron desktopCapturer plus Windows GDI Smart Capture	GDI path is used for Genshin Smart Capture reliability
Input automation	C# sidecar with elevated dev runner when needed	Live-validated for read-only inventory selection clicks; see docs/AUTOMATION_LIVE_SCAN.md
Tests	Vitest + TypeScript checks	Current validation baseline; regression samples must expand
Packaging	electron-builder	Configured in package.json

Runtime

Environment	Entry Point	Notes
Local dev	npm run dev	Starts Vite and Electron
Local dev with automation	npm run dev:admin	Required when GenshinImpact.exe is elevated; Windows blocks lower-integrity cursor/click input
Production build	npm run build	Builds React and Electron main process
Preview	npm run preview	Browser preview only; capture bridge is unavailable

Current State Review

What already works

• The app can enumerate capture sources and often identify the Genshin window automatically.
• Single-artifact capture is no longer blind full-screen OCR; it produces detail crops, OCR blocks, parsed fields, confidence, and notes.
• A local canonical data package already exists in src/data/genshinGameData.json, generated from genshin-db.
• The parser already uses known sets, pieces, slots, stat aliases, set aliases, character aliases, and derived slot/set mapping.
• Review samples, learned replacements, parser notes, and stored artifacts already persist locally.
• The auto-scan loop is no longer a naive click spammer: it has preflight, verification, miss handling, page fingerprinting, and stop conditions.
• The scanner now has an Inventory-Kamera comparison path: 32 safe artifact targets per page, lookup-derived fields, fast OCR crop profile, current vs. IK-traineddata benchmark endpoint, and a quality-gated live soak runner.
• Elevated live automation is validated in the current dev environment: /automation/probe-click?index=1 changed the selected artifact and /scanner/start?limit=2 completed with 2/2 verified reads and 0 misses.
• A 2026-07-08 visible-inventory 50-artifact run completed cleanly with 50/50 parsed and stored, 0 review, 0 duplicates, and 0 misses. It is stable but still too slow for the 2-3 artifacts/second target.
• Later 2026-07-08 direct-GDI hot-path runs completed 20/20 parsed with 0 misses and 0 review. The best clean 20-artifact iteration reached 7285 ms, or roughly 2.75 artifacts/second; the final stable 2026-07-08-direct-gdi-reviewfix run completed in 7973 ms. The 3 artifacts/second target remains unproven.
• The same direct-GDI path completed a 100-artifact run with 100/100 parsed, 0 review, 0 misses, and 42064 ms elapsed across 4 pages.
• Review samples can now be exported with npm run eval:review-candidates into a Git-ignored human-labeling worklist. This is the next quality phase before adding more OCR corpus cases or trusting review queue data as labels.

What is still structurally weak

• The scan experience is still partly orchestrated from src/App.tsx, which makes behavior changes harder than they should be.
• Broader scan soak testing has reached clean 20-, 45-, and 100-artifact runs with 0 misses on the current engine. The current-vs-IK-traineddata comparison is now captured; current won the qualified 100-artifact comparison on 2026-07-08.
• OCR quality is still inconsistent enough that some fields are recovered by fallback and derivation more often than they should be.
• Learned fixes currently focus on text replacements; they do not yet update crop offsets, UI profile variants, or scanner targeting rules in a structured way.
• The scan page is cleaner than before, but it still exposes too much operator/debug state in the main flow.
• Recommendations and build logic exist, but the scanner is not yet reliable enough to make them the core focus.
• The latest source has completed the final current-vs-IK-traineddata live comparison for this environment. Repeatability and 3 artifacts/second are still open.

Current product conclusion

The app has crossed from OCR-demo/prototype into a validated scanner baseline on main. The current merge-ready path is the visible-inventory scan flow: the operator opens Artifact inventory with a visible detail card, the app verifies the state, scans read-only, persists parsed artifacts, and keeps uncertain data reviewable. The next product phase is not another broad scanner rewrite; it is repeatability, corpus growth, UI polish, and then recommendations on top of the trusted artifact store.

Product Direction

• Artifact scanning is the first-class feature.
• Character optimization returns only after scan quality is trustworthy.
• Team building stays out of the critical path until artifact ingestion is stable.
• Inventory Kamera remains a reference for scan choreography and page movement, not a runtime dependency.
• Self-learning stays deterministic and local first: review samples, aliases, crop offsets, and UI profile tuning before any ML retraining discussion.

Execution Plan

Phase 0 - Stabilize the operator surface

Outcome:

• Scan page reduced to source, main preview, result panel, primary scan actions, and compact status.
• Diagnostics, logs, crops, confidence breakdown, review queue, and learning internals moved behind modal or drawer entry points.
• Scan completion popup summarizes scanned, stored, duplicates, review samples, blocked reason, and elapsed time.

Status:

• Mostly done for the scanner baseline. The Diagnose/dev surface is separated and scrollable, scan summaries are compact, and the normal Auto-Scan path is guarded. Further UI polish remains useful but no longer blocks scanner merge.

Phase 1 - Canonical game data package

Outcome:

• scripts/generate-genshin-data.cjs emits one stricter package contract for:
  • artifact sets
  • artifact pieces
  • slot-by-piece mapping
  • stats and allowed mains by slot
  • characters
  • aliases
  • UI profiles
  • source version metadata
• Parser regression tests run against saved review samples and known bad cases.
• Parser stops "free guessing" outside the canonical package.

Status:

• Implemented as a generated lookup package in src/data/genshinGameData.json with validation and parser integration. Continue regenerating and expanding aliases deliberately when Genshin data or OCR samples require it.

Phase 2 - Deterministic parser hardening

Outcome:

• Name, slot, set, main stat, and equipped fields are parsed through layered validation:
  1. direct OCR cleanup
  2. alias normalization
  3. exact package match
  4. constrained fuzzy match
  5. safe derivation from piece/slot/value references
• Main stat/value inference is tightened with slot constraints and reference tables.
• Bad parses automatically generate structured review reasons.
• Equipped-character parsing is canonical-data constrained: noisy known names can match through aliases/fuzzy lookup, but unknown footer fragments stay Not detected instead of being stored as invented character names.

Status:

• Implemented for the merge baseline. Parser tests cover canonical set/slot/stat matching, equipped-character footer noise, known aliases, and unsafe one-letter fragments. Continue growing the confirmed review corpus before tightening thresholds further.

Phase 3 - Scanner core rebuild

Outcome:

• Auto-scan becomes a dedicated engine with explicit states:
  • preflight
  • grid detection
  • click target
  • wait stable
  • detail verify
  • parse
  • store or review
  • next tile
  • row scroll
  • resume or stop
• Progress counts only when a new verified artifact or duplicate signature is confirmed.
• Repeated pages, unchanged detail cards, blocked cursor movement, and scroll failures stop the scan with diagnosis instead of producing fake progress.
• Fast artifact-read captures include the equipped footer when an equipped marker is visible; preflight and polling captures still skip expensive OCR.

Status:

• Implemented and merged for the visible-inventory path. Live validation on 2026-07-09 covered 20/20 verified/parsed with 0 review and 0 misses, equipped-character persistence, unlocked lock state, positive locked state, and locked persistence. Explicit entry-mode experiments remain separate from the normal merge-ready path.

Phase 4 - Input automation replacement

Outcome:

• Replace the production automation path with a persistent Windows sidecar dedicated to:
  • focus
  • move
  • click
  • scroll
  • capture
  • probe
• Session probe decides which input mode works before auto-scan is unlocked.
• Auto-scan never starts on a session that cannot prove one successful detail-card change.

Status:

• Read-only C# helper path and elevated dev startup are validated for the visible-inventory scanner baseline. Direct inventory, Paimon-menu, and auto-entry modes remain Dev-Control experiments and should be tested with low limits before being promoted.

Phase 5 - Learning loop that actually compounds

Outcome:

• Weak scans save themselves as review samples automatically.
• User corrections update local:
  • text replacements
  • alias maps
  • crop offsets
  • UI profile adjustments
  • constrained set/piece/slot fixes
• "Apply learned fixes" runs before every parse.
• Review samples become both parser regression fixtures and learning inputs.

Status:

• Prepared in code for text replacements, field aliases, constrained fixes, crop adjustment proposals, and UI-profile adjustment proposals. Crop/profile changes still require live review before being auto-applied.

Phase 6 - Recommendations come back on top of a trusted scanner

Outcome:

• Account snapshot and build suggestions are only promoted once scan quality is high enough to trust owned artifacts.
• Recommendations explain uncertainty and surface conflicts instead of pretending perfect certainty.

Status:

• Next major product area after repeatability and corpus work. Do not promote recommendation UX until stored artifact quality is backed by more confirmed review samples and repeat live scan runs.

Immediate Next Implementation Order

1. Keep the visible-inventory scanner path as the production baseline and avoid promoting auto-entry, direct-inventory, or paimon-menu until they pass their own low-limit live validations.
2. Grow the confirmed OCR corpus from review samples exported by npm run eval:review-candidates and prepared through npm run eval:prepare-confirmed.
3. Repeat live scanner runs in later sessions to prove repeatability across pages, locked/unlocked artifacts, equipped footers, and duplicate handling.
4. Continue the optional 3 artifacts/second work only if the next change can reduce OCR/capture transport time without weakening quality gates.
5. Start recommendation/product UX work only after repeat scan quality and confirmed corpus coverage are strong enough to trust stored artifacts.

Open Questions

Question	Status
Is the current C# helper sufficient for production packaging, or does a later Rust/C++ sidecar still materially reduce latency or packaging risk?	Open
When should UI-profile learning be allowed to change crop geometry automatically versus requiring review approval?	Open
What scan-quality threshold is high enough before recommendations should be considered user-facing again?	Open
Which Genshin UI languages should be supported after English once the scanner contract is stable?	Open