genshin-assistant/docs/DECISIONS.md

Decisions

This document contains Architecture Decision Records.

ADR Index

ID	Title	Status	Date
ADR-001	Build a local Electron app first	Accepted	2026-07-04
ADR-002	Use screen capture as the primary scanner source	Accepted	2026-07-04
ADR-003	Keep APIs and GOOD compatibility optional	Accepted	2026-07-04
ADR-004	Treat in-game marking as a later opt-in feature	Accepted	2026-07-04
ADR-005	Use a generated Genshin data package for OCR matching	Accepted	2026-07-04
ADR-006	Persistent input helper and JSON artifact store before SQLite	Accepted	2026-07-04
ADR-007	Measure OCR accuracy with a labeled eval harness before reworking the scanner	Accepted	2026-07-05
ADR-008	Replace the PowerShell input/capture helper with a C# sidecar	Accepted	2026-07-05
ADR-009	Resolution-anchored layout profiles and OCR preprocessing over color detection	Accepted	2026-07-05
ADR-010	Elevated dev runner and bounded live automation probes	Accepted	2026-07-07

ADR-001: Build A Local Electron App First

Status

Accepted

Context

The product needs a Windows desktop UI, local screen capture, possible overlay windows, and future optional input automation.

Decision

Use Electron with React and TypeScript for the MVP.

Consequences

• Fast UI iteration and easy local packaging.
• Electron main-process code must be treated as a separate boundary from renderer code.
• Native or Rust sidecars can be added later for high-performance capture/OCR work.

ADR-002: Use Screen Capture As The Primary Scanner Source

Status

Accepted

Context

The user wants an app that works without Inventory Kamera, Genshin Optimizer, Enka, or HoYoLAB as core dependencies.

Decision

Use local screen capture as the primary source. Current Smart Capture focuses Genshin, hides the app, captures the primary screen through Windows GDI, detects the artifact detail panel, then OCRs focused crops.

Consequences

• The app remains offline-first.
• OCR and crop reliability are core product risks.
• UI language, resolution, HDR, and game layout changes need explicit test coverage.

ADR-003: Keep APIs And GOOD Compatibility Optional

Status

Accepted

Context

External APIs and existing optimizer formats can speed up setup, but should not define the main user workflow.

Decision

Keep Enka, HoYoLAB, Akasha, Genshin Optimizer, and GOOD import/export as optional future compatibility layers.

Consequences

• The app can work without external accounts or cookies.
• Data package and scanner quality become more important.
• Compatibility can be added when it helps testing, migration, or export.

ADR-004: Treat In-Game Marking As A Later Opt-In Feature

Status

Accepted

Context

Locking or marking artifacts in game may save time, but input automation increases ToS and misclick risk.

Decision

Do not ship in-game marking in the scanner MVP. If implemented later, it must be off by default, reversible, whitelisted, previewed before execution, and stoppable with ESC or user mouse movement.

Consequences

• Early scanner work stays lower risk.
• App-internal triage remains the first decision layer.
• No delete, feed, enhance, or resource-spending automation is allowed.

ADR-005: Use A Generated Genshin Data Package For OCR Matching

Status

Accepted

Context

Hardcoded arrays for characters and artifact sets caused repeated scanner failures whenever the user tested a newer character, set, or artifact name.

Decision

Generate src/data/genshinGameData.json from genshin-db and use it as the local matching dictionary for artifact sets, artifact piece names, characters, slots, main stats, and substats.

Consequences

• The scanner can recognize new characters and sets as soon as the local data package is regenerated from an updated genshin-db.
• Parser logic stays generic and testable instead of growing one-off fixes.
• OCR still needs good crops and text quality; the data package improves recognition but cannot solve unreadable screenshots by itself.

ADR-006: Persistent Input Helper And JSON Artifact Store Before SQLite

Status

Accepted

Context

Per-action PowerShell scripts recompiled the Win32 interop for every click, scroll, and capture (1-2s each) and one Marshal::SizeOf call was broken in Windows PowerShell 5.1, so SendInput clicks silently never executed. Scan results were also not persisted anywhere; only review samples reached disk. Adding better-sqlite3 (native module) was considered too heavy for this step.

Decision

Run one persistent PowerShell helper process (compiled once, JSON protocol over stdin/stdout) for focus, cursor/ESC state, click, scroll, and GDI capture. Persist parsed artifacts into artifact-store.json in userData, deduplicated by a content signature that excludes the equipped character. Keep SQLite as the planned future store; the JSON store is the migration source.

Consequences

• Batch scans become fast enough to be testable and the failsafe (ESC or user mouse movement aborts) can poll cheaply between actions.
• Automated clicks are verified by checking that the parsed detail signature changed; repeated failures abort with a diagnosis hint instead of clicking blindly.
• Leveling an artifact changes its signature and creates a new record; rescan-merge is an open follow-up.
• If the helper process dies it is respawned on the next request; pending requests fail loudly instead of hanging.

ADR-007: Measure OCR Accuracy With A Labeled Eval Harness Before Reworking The Scanner

Status

Accepted

Context

OCR and crop reliability are the core product risk (ADR-002), but there was no way to measure field-level accuracy. Every OCR, crop, or parser change was a blind change - regressions could only be caught by a human re-testing against the live game, and there was no baseline to compare a new engine or preprocessing step against.

Decision

Add a field-level eval harness (src/eval/) that runs the real parseArtifactCandidate over a labeled corpus and reports per-field and per-case accuracy. Seed the corpus from the existing parser test cases (verified labels), and grow it from human-confirmed review samples via reviewSampleToEvalCase. Run it as a gate under npm test (must stay 100% on the verified seed) and as a full report via npm run eval.

Consequences

• OCR/layout/preprocessing changes are measured, not guessed at; a new engine or a preprocessing step has to beat a recorded baseline.
• The review queue does double duty: it flags artifacts for human correction and feeds the eval corpus. Review-sample parsed blocks are label candidates, never ground truth, to avoid the parser grading itself.
• The corpus label is the source of truth. If a code change intentionally alters a correct output, the label is updated in the same commit.

ADR-008: Replace The PowerShell Input/Capture Helper With A C# Sidecar

Status

Accepted

Context

The persistent PowerShell helper (ADR-006) still carries Windows PowerShell 5.1 quirks (the Marshal::SizeOf interop bug), compiles Win32 interop at startup, and captures each frame by writing a PNG to the temp directory and reading it back. Inventory Kamera - the proven reference for automated Genshin scanning - uses a C#/.NET stack with InputSimulator (SendInput) and direct GDI/BitBlt capture.

Decision

Replace the PowerShell helper with a self-contained .NET (C#) sidecar that speaks the same JSON-over-stdin/stdout protocol, so the Electron-side InputHelperService interface stays stable. The sidecar does per-monitor DPI-aware SendInput click/scroll, BitBlt client-rect capture returning bytes without a temp file, and elevation detection.

Consequences

• No PS 5.1 marshalling bugs, no per-call interop compile, no temp-PNG churn; lower latency makes batch scans and the ESC/mouse failsafe polling cheaper.
• Adds a .NET build/publish step and ships a compiled exe with the app.
• The migration is behind the existing service interface, so the renderer and scan loop do not change.

ADR-009: Resolution-Anchored Layout Profiles And OCR Preprocessing Over Color Detection

Status

Accepted

Context

The current pipeline finds the artifact detail panel with hardcoded orange/green color thresholds (inferDetailRect) and then crops fixed percentages of that guessed rectangle. This is brittle against HDR, color profiles, UI scale, aspect ratio, and game UI updates. Inventory Kamera instead requires borderless 16:9 and scales fixed crop coordinates from a reference resolution, then feeds Tesseract preprocessed (grayscale, upscaled, thresholded) crops - which is why general Tesseract is accurate enough for them.

Decision

Adopt the same approach: require borderless 16:9, drive crops from resolution-anchored layout profiles scaled from a reference resolution (color detection only as a fallback), and add a per-region preprocessing pass (grayscale, upscale, threshold, invert) plus a digit-whitelist mode for numeric fields. The set name stays derived from the static piece-to-set data package (no dedicated set-effect crop). English-only OCR is accepted. Every change is validated against the ADR-007 eval harness.

Consequences

• Crop positions become deterministic per resolution instead of per-frame guesses.
• Preprocessing is expected to lift accuracy enough that a custom OCR engine is only pursued if the eval harness shows Tesseract-plus-preprocessing is insufficient.
• Non-16:9 or non-borderless setups are explicitly unsupported for the auto scanner; the app should detect and warn rather than silently misread.

ADR-010: Elevated Dev Runner And Bounded Live Automation Probes

Status

Accepted

Context

Automatic grid scanning needs read-only mouse movement, click, and wheel input to reach the focused Genshin window. A lower-integrity app can fail to deliver input to an elevated or protected target because of Windows UIPI/integrity boundaries. During live testing, npm run dev:admin originally printed that a new Administrator window was started, but the elevated PowerShell received no arguments, so the intended dev process did not reliably start.

The project also needed a smaller live validation path than a full inventory scan. A full scan is too risky as the first proof of input delivery because it can click many tiles before a bad coordinate, focus issue, or blocked input is understood.

Decision

Keep automatic scan input automation read-only and require an elevated runtime when Windows reports that automation would otherwise be blocked. Replace the old dev-admin.cmd entry with scripts/dev-admin.ps1, quote the elevated PowerShell arguments explicitly, and log elevated startup to outputs/admin-start/admin-dev.log.

Add dev-only HTTP checks:

• /automation/probe-click?index=N or ?row=R&col=C performs one safe inventory selection click and verifies whether the detail panel changed.
• /scanner/start?limit=N sends a temporary scan-limit payload to the renderer, so live auto-scan validation can start with two items instead of the UI default.

Document the workflow in AUTOMATION_LIVE_SCAN.md.

Consequences

• The user still has to approve Windows UAC manually; the app must not try to click the Secure Desktop prompt.
• We can distinguish input delivery from OCR/parser quality with a one-click probe before running any broader scan.
• Live validation now has a low-risk path: check elevation and Genshin detection, run a single probe click, then run a bounded limit=2 scan.
• The implementation remains inside the allowed safety boundary: no memory reads, hooks, injection, game-file modification, deleting, feeding, enhancing, locking/unlocking, or spending resources.