genshin-assistant/docs/DECISIONS.md

# Decisions

This document contains Architecture Decision Records.

## ADR Index

| ID | Title | Status | Date |
| --- | --- | --- | --- |
| ADR-001 | Build a local Electron app first | Accepted | 2026-07-04 |
| ADR-002 | Use screen capture as the primary scanner source | Accepted | 2026-07-04 |
| ADR-003 | Keep APIs and GOOD compatibility optional | Accepted | 2026-07-04 |
| ADR-004 | Treat in-game marking as a later opt-in feature | Accepted | 2026-07-04 |
| ADR-005 | Use a generated Genshin data package for OCR matching | Accepted | 2026-07-04 |
| ADR-006 | Persistent input helper and JSON artifact store before SQLite | Accepted | 2026-07-04 |
| ADR-007 | Measure OCR accuracy with a labeled eval harness before reworking the scanner | Accepted | 2026-07-05 |
| ADR-008 | Replace the PowerShell input/capture helper with a C# sidecar | Accepted | 2026-07-05 |
| ADR-009 | Resolution-anchored layout profiles and OCR preprocessing over color detection | Accepted | 2026-07-05 |
| ADR-010 | Elevated dev runner and bounded live automation probes | Accepted | 2026-07-07 |

## ADR-001: Build A Local Electron App First

### Status

Accepted

### Context

The product needs a Windows desktop UI, local screen capture, possible overlay windows, and future optional input automation.

### Decision

Use Electron with React and TypeScript for the MVP.

### Consequences

- Fast UI iteration and easy local packaging.
- Electron main-process code must be treated as a separate boundary from renderer code.
- Native or Rust sidecars can be added later for high-performance capture/OCR work.

## ADR-002: Use Screen Capture As The Primary Scanner Source

### Status

Accepted

### Context

The user wants an app that works without Inventory Kamera, Genshin Optimizer, Enka, or HoYoLAB as core dependencies.

### Decision

Use local screen capture as the primary source. Current Smart Capture focuses Genshin, hides the app, captures the primary screen through Windows GDI, detects the artifact detail panel, then OCRs focused crops.

### Consequences

- The app remains offline-first.
- OCR and crop reliability are core product risks.
- UI language, resolution, HDR, and game layout changes need explicit test coverage.

## ADR-003: Keep APIs And GOOD Compatibility Optional

### Status

Accepted

### Context

External APIs and existing optimizer formats can speed up setup, but should not define the main user workflow.

### Decision

Keep Enka, HoYoLAB, Akasha, Genshin Optimizer, and GOOD import/export as optional future compatibility layers.

### Consequences

- The app can work without external accounts or cookies.
- Data package and scanner quality become more important.
- Compatibility can be added when it helps testing, migration, or export.

## ADR-004: Treat In-Game Marking As A Later Opt-In Feature

### Status

Accepted

### Context

Locking or marking artifacts in game may save time, but input automation increases ToS and misclick risk.

### Decision

Do not ship in-game marking in the scanner MVP. If implemented later, it must be off by default, reversible, whitelisted, previewed before execution, and stoppable with ESC or user mouse movement.

### Consequences

- Early scanner work stays lower risk.
- App-internal triage remains the first decision layer.
- No delete, feed, enhance, or resource-spending automation is allowed.

## ADR-005: Use A Generated Genshin Data Package For OCR Matching

### Status

Accepted

### Context

Hardcoded arrays for characters and artifact sets caused repeated scanner failures whenever the user tested a newer character, set, or artifact name.

### Decision

Generate `src/data/genshinGameData.json` from `genshin-db` and use it as the local matching dictionary for artifact sets, artifact piece names, characters, slots, main stats, and substats.

### Consequences

- The scanner can recognize new characters and sets as soon as the local data package is regenerated from an updated `genshin-db`.
- Parser logic stays generic and testable instead of growing one-off fixes.
- OCR still needs good crops and text quality; the data package improves recognition but cannot solve unreadable screenshots by itself.

## ADR-006: Persistent Input Helper And JSON Artifact Store Before SQLite

### Status

Accepted

### Context

Per-action PowerShell scripts recompiled the Win32 interop for every click, scroll, and capture (1-2s each) and one `Marshal::SizeOf` call was broken in Windows PowerShell 5.1, so SendInput clicks silently never executed. Scan results were also not persisted anywhere; only review samples reached disk. Adding `better-sqlite3` (native module) was considered too heavy for this step.

### Decision

Run one persistent PowerShell helper process (compiled once, JSON protocol over stdin/stdout) for focus, cursor/ESC state, click, scroll, and GDI capture. Persist parsed artifacts into `artifact-store.json` in userData, deduplicated by a content signature that excludes the equipped character. Keep SQLite as the planned future store; the JSON store is the migration source.

### Consequences

- Batch scans become fast enough to be testable and the failsafe (ESC or user mouse movement aborts) can poll cheaply between actions.
- Automated clicks are verified by checking that the parsed detail signature changed; repeated failures abort with a diagnosis hint instead of clicking blindly.
- Leveling an artifact changes its signature and creates a new record; rescan-merge is an open follow-up.
- If the helper process dies it is respawned on the next request; pending requests fail loudly instead of hanging.

## ADR-007: Measure OCR Accuracy With A Labeled Eval Harness Before Reworking The Scanner

### Status

Accepted

### Context

OCR and crop reliability are the core product risk (ADR-002), but there was no
way to measure field-level accuracy. Every OCR, crop, or parser change was a
blind change - regressions could only be caught by a human re-testing against the
live game, and there was no baseline to compare a new engine or preprocessing
step against.

### Decision

Add a field-level eval harness (`src/eval/`) that runs the real
`parseArtifactCandidate` over a labeled corpus and reports per-field and
per-case accuracy. Seed the corpus from the existing parser test cases (verified
labels), and grow it from human-confirmed review samples via
`reviewSampleToEvalCase`. Run it as a gate under `npm test` (must stay 100% on
the verified seed) and as a full report via `npm run eval`.

### Consequences

- OCR/layout/preprocessing changes are measured, not guessed at; a new engine or
  a preprocessing step has to beat a recorded baseline.
- The review queue does double duty: it flags artifacts for human correction and
  feeds the eval corpus. Review-sample `parsed` blocks are label *candidates*,
  never ground truth, to avoid the parser grading itself.
- The corpus label is the source of truth. If a code change intentionally alters
  a correct output, the label is updated in the same commit.

## ADR-008: Replace The PowerShell Input/Capture Helper With A C# Sidecar

### Status

Accepted

### Context

The persistent PowerShell helper (ADR-006) still carries Windows PowerShell 5.1
quirks (the `Marshal::SizeOf` interop bug), compiles Win32 interop at startup,
and captures each frame by writing a PNG to the temp directory and reading it
back. Inventory Kamera - the proven reference for automated Genshin scanning -
uses a C#/.NET stack with InputSimulator (SendInput) and direct GDI/BitBlt
capture.

### Decision

Replace the PowerShell helper with a self-contained .NET (C#) sidecar that speaks
the same JSON-over-stdin/stdout protocol, so the Electron-side `InputHelperService`
interface stays stable. The sidecar does per-monitor DPI-aware SendInput
click/scroll, BitBlt client-rect capture returning bytes without a temp file, and
elevation detection.

### Consequences

- No PS 5.1 marshalling bugs, no per-call interop compile, no temp-PNG churn;
  lower latency makes batch scans and the ESC/mouse failsafe polling cheaper.
- Adds a .NET build/publish step and ships a compiled exe with the app.
- The migration is behind the existing service interface, so the renderer and
  scan loop do not change.

## ADR-009: Resolution-Anchored Layout Profiles And OCR Preprocessing Over Color Detection

### Status

Accepted

### Context

The current pipeline finds the artifact detail panel with hardcoded orange/green
color thresholds (`inferDetailRect`) and then crops fixed percentages of that
guessed rectangle. This is brittle against HDR, color profiles, UI scale, aspect
ratio, and game UI updates. Inventory Kamera instead requires borderless 16:9 and
scales fixed crop coordinates from a reference resolution, then feeds Tesseract
preprocessed (grayscale, upscaled, thresholded) crops - which is why general
Tesseract is accurate enough for them.

### Decision

Adopt the same approach: require borderless 16:9, drive crops from
resolution-anchored layout profiles scaled from a reference resolution (color
detection only as a fallback), and add a per-region preprocessing pass
(grayscale, upscale, threshold, invert) plus a digit-whitelist mode for numeric
fields. The set name stays derived from the static piece-to-set data package (no
dedicated set-effect crop). English-only OCR is accepted. Every change is
validated against the ADR-007 eval harness.

### Consequences

- Crop positions become deterministic per resolution instead of per-frame guesses.
- Preprocessing is expected to lift accuracy enough that a custom OCR engine is
  only pursued if the eval harness shows Tesseract-plus-preprocessing is
  insufficient.
- Non-16:9 or non-borderless setups are explicitly unsupported for the auto
  scanner; the app should detect and warn rather than silently misread.

## ADR-010: Elevated Dev Runner And Bounded Live Automation Probes

### Status

Accepted

### Context

Automatic grid scanning needs read-only mouse movement, click, and wheel input
to reach the focused Genshin window. A lower-integrity app can fail to deliver
input to an elevated or protected target because of Windows UIPI/integrity
boundaries. During live testing, `npm run dev:admin` originally printed that a
new Administrator window was started, but the elevated PowerShell received no
arguments, so the intended dev process did not reliably start.

The project also needed a smaller live validation path than a full inventory
scan. A full scan is too risky as the first proof of input delivery because it
can click many tiles before a bad coordinate, focus issue, or blocked input is
understood.

### Decision

Keep automatic scan input automation read-only and require an elevated runtime
when Windows reports that automation would otherwise be blocked. Replace the
old `dev-admin.cmd` entry with `scripts/dev-admin.ps1`, quote the elevated
PowerShell arguments explicitly, and log elevated startup to
`outputs/admin-start/admin-dev.log`.

Add dev-only HTTP checks:

- `/automation/probe-click?index=N` or `?row=R&col=C` performs one safe
  inventory selection click and verifies whether the detail panel changed.
- `/scanner/start?limit=N` sends a temporary scan-limit payload to the renderer,
  so live auto-scan validation can start with two items instead of the UI
  default.

Document the workflow in [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).

### Consequences

- The user still has to approve Windows UAC manually; the app must not try to
  click the Secure Desktop prompt.
- We can distinguish input delivery from OCR/parser quality with a one-click
  probe before running any broader scan.
- Live validation now has a low-risk path: check elevation and Genshin
  detection, run a single probe click, then run a bounded `limit=2` scan.
- The implementation remains inside the allowed safety boundary: no memory
  reads, hooks, injection, game-file modification, deleting, feeding, enhancing,
  locking/unlocking, or spending resources.