genshin-assistant/docs/DECISIONS.md

# Decisions

This document contains Architecture Decision Records.

## ADR Index

| ID | Title | Status | Date |
| --- | --- | --- | --- |
| ADR-001 | Build a local Electron app first | Accepted | 2026-07-04 |
| ADR-002 | Use screen capture as the primary scanner source | Accepted | 2026-07-04 |
| ADR-003 | Keep APIs and GOOD compatibility optional | Accepted | 2026-07-04 |
| ADR-004 | Treat in-game marking as a later opt-in feature | Accepted | 2026-07-04 |
| ADR-005 | Use a generated Genshin data package for OCR matching | Accepted | 2026-07-04 |
| ADR-006 | Persistent input helper and JSON artifact store before SQLite | Accepted | 2026-07-04 |
| ADR-007 | Measure OCR accuracy with a labeled eval harness before reworking the scanner | Accepted | 2026-07-05 |
| ADR-008 | Replace the PowerShell input/capture helper with a C# sidecar | Accepted | 2026-07-05 |
| ADR-009 | Resolution-anchored layout profiles and OCR preprocessing over color detection | Accepted | 2026-07-05 |
| ADR-010 | Elevated dev runner and bounded live automation probes | Accepted | 2026-07-07 |
| ADR-011 | Retire alternate OCR comparison paths after current scanner baseline | Superseded | 2026-07-09 |
| ADR-012 | Separate live scan results, artifact inventory, and value evaluation | Accepted | 2026-07-09 |
| ADR-013 | Move high-speed scanning into the native helper and vendor IK inventorylists | Accepted | 2026-07-09 |

## ADR-001: Build A Local Electron App First

### Status

Accepted

### Context

The product needs a Windows desktop UI, local screen capture, possible overlay windows, and future optional input automation.

### Decision

Use Electron with React and TypeScript for the MVP.

### Consequences

- Fast UI iteration and easy local packaging.
- Electron main-process code must be treated as a separate boundary from renderer code.
- Native or Rust sidecars can be added later for high-performance capture/OCR work.

## ADR-002: Use Screen Capture As The Primary Scanner Source

### Status

Accepted

### Context

The user wants an app that works without optimizer imports, external scanner
tools, Enka, or HoYoLAB as core dependencies.

### Decision

Use local screen capture as the primary source. Current Smart Capture focuses Genshin, hides the app, captures the primary screen through Windows GDI, detects the artifact detail panel, then OCRs focused crops.

### Consequences

- The app remains offline-first.
- OCR and crop reliability are core product risks.
- UI language, resolution, HDR, and game layout changes need explicit test coverage.

## ADR-003: Keep APIs And GOOD Compatibility Optional

### Status

Accepted

### Context

External APIs and existing optimizer formats can speed up setup, but should not define the main user workflow.

### Decision

Keep Enka, HoYoLAB, Akasha, Genshin Optimizer, and GOOD import/export as optional future compatibility layers.

### Consequences

- The app can work without external accounts or cookies.
- Data package and scanner quality become more important.
- Compatibility can be added when it helps testing, migration, or export.

## ADR-004: Treat In-Game Marking As A Later Opt-In Feature

### Status

Accepted

### Context

Locking or marking artifacts in game may save time, but input automation increases ToS and misclick risk.

### Decision

Do not ship in-game marking in the scanner MVP. If implemented later, it must be off by default, reversible, whitelisted, previewed before execution, and stoppable with ESC or user mouse movement.

### Consequences

- Early scanner work stays lower risk.
- App-internal triage remains the first decision layer.
- No delete, feed, enhance, or resource-spending automation is allowed.

## ADR-005: Use A Generated Genshin Data Package For OCR Matching

### Status

Accepted

### Context

Hardcoded arrays for characters and artifact sets caused repeated scanner failures whenever the user tested a newer character, set, or artifact name.

### Decision

Generate `src/data/genshinGameData.json` from `genshin-db` and use it as the local matching dictionary for artifact sets, artifact piece names, characters, slots, main stats, and substats.

### Consequences

- The scanner can recognize new characters and sets as soon as the local data package is regenerated from an updated `genshin-db`.
- Parser logic stays generic and testable instead of growing one-off fixes.
- OCR still needs good crops and text quality; the data package improves recognition but cannot solve unreadable screenshots by itself.

## ADR-006: Persistent Input Helper And JSON Artifact Store Before SQLite

### Status

Accepted

### Context

Per-action PowerShell scripts recompiled the Win32 interop for every click, scroll, and capture (1-2s each) and one `Marshal::SizeOf` call was broken in Windows PowerShell 5.1, so SendInput clicks silently never executed. Scan results were also not persisted anywhere; only review samples reached disk. Adding `better-sqlite3` (native module) was considered too heavy for this step.

### Decision

Run one persistent PowerShell helper process (compiled once, JSON protocol over stdin/stdout) for focus, cursor/ESC state, click, scroll, and GDI capture. Persist parsed artifacts into `artifact-store.json` in userData, deduplicated by a content signature that excludes the equipped character. Keep SQLite as the planned future store; the JSON store is the migration source.

### Consequences

- Batch scans become fast enough to be testable and the failsafe (ESC or user mouse movement aborts) can poll cheaply between actions.
- Automated clicks are verified by checking that the parsed detail signature changed; repeated failures abort with a diagnosis hint instead of clicking blindly.
- Leveling an artifact changes its signature and creates a new record; rescan-merge is an open follow-up.
- If the helper process dies it is respawned on the next request; pending requests fail loudly instead of hanging.

## ADR-007: Measure OCR Accuracy With A Labeled Eval Harness Before Reworking The Scanner

### Status

Accepted

### Context

OCR and crop reliability are the core product risk (ADR-002), but there was no
way to measure field-level accuracy. Every OCR, crop, or parser change was a
blind change - regressions could only be caught by a human re-testing against the
live game, and there was no baseline to compare a new engine or preprocessing
step against.

### Decision

Add a field-level eval harness (`src/eval/`) that runs the real
`parseArtifactCandidate` over a labeled corpus and reports per-field and
per-case accuracy. Seed the corpus from the existing parser test cases (verified
labels), and grow it from human-confirmed review samples via
`reviewSampleToEvalCase`. Run it as a gate under `npm test` (must stay 100% on
the verified seed) and as a full report via `npm run eval`.

### Consequences

- OCR/layout/preprocessing changes are measured, not guessed at; a new engine or
  a preprocessing step has to beat a recorded baseline.
- The review queue does double duty: it flags artifacts for human correction and
  feeds the eval corpus. Review-sample `parsed` blocks are label *candidates*,
  never ground truth, to avoid the parser grading itself.
- The corpus label is the source of truth. If a code change intentionally alters
  a correct output, the label is updated in the same commit.

## ADR-008: Replace The PowerShell Input/Capture Helper With A C# Sidecar

### Status

Accepted

### Context

The persistent PowerShell helper (ADR-006) still carries Windows PowerShell 5.1
quirks (the `Marshal::SizeOf` interop bug), compiles Win32 interop at startup,
and captures each frame by writing a PNG to the temp directory and reading it
back. A C#/.NET sidecar with SendInput and direct GDI/BitBlt capture is a
cleaner fit for the validated Windows automation path.

### Decision

Replace the PowerShell helper with a self-contained .NET (C#) sidecar that speaks
the same JSON-over-stdin/stdout protocol, so the Electron-side `InputHelperService`
interface stays stable. The sidecar does per-monitor DPI-aware SendInput
click/scroll, BitBlt client-rect capture returning bytes without a temp file, and
elevation detection.

### Consequences

- No PS 5.1 marshalling bugs, no per-call interop compile, no temp-PNG churn;
  lower latency makes batch scans and the ESC/mouse failsafe polling cheaper.
- Adds a .NET build/publish step and ships a compiled exe with the app.
- The migration is behind the existing service interface, so the renderer and
  scan loop do not change.

## ADR-009: Resolution-Anchored Layout Profiles And OCR Preprocessing Over Color Detection

### Status

Accepted

### Context

The current pipeline finds the artifact detail panel with hardcoded orange/green
color thresholds (`inferDetailRect`) and then crops fixed percentages of that
guessed rectangle. This is brittle against HDR, color profiles, UI scale, aspect
ratio, and game UI updates. A stricter borderless 16:9 profile with fixed crop
coordinates from a reference resolution is easier to validate and reproduce than
per-frame color hunting.

### Decision

Adopt the same approach: require borderless 16:9, drive crops from
resolution-anchored layout profiles scaled from a reference resolution (color
detection only as a fallback), and add a per-region preprocessing pass
(grayscale, upscale, threshold, invert) plus a digit-whitelist mode for numeric
fields. The set name stays derived from the static piece-to-set data package (no
dedicated set-effect crop). English-only OCR is accepted. Every change is
validated against the ADR-007 eval harness.

### Consequences

- Crop positions become deterministic per resolution instead of per-frame guesses.
- Preprocessing is expected to lift accuracy enough that a custom OCR engine is
  only pursued if the eval harness shows Tesseract-plus-preprocessing is
  insufficient.
- Non-16:9 or non-borderless setups are explicitly unsupported for the auto
  scanner; the app should detect and warn rather than silently misread.

## ADR-010: Elevated Dev Runner And Bounded Live Automation Probes

### Status

Accepted

### Context

Automatic grid scanning needs read-only mouse movement, click, and wheel input
to reach the focused Genshin window. A lower-integrity app can fail to deliver
input to an elevated or protected target because of Windows UIPI/integrity
boundaries. During live testing, `npm run dev:admin` originally printed that a
new Administrator window was started, but the elevated PowerShell received no
arguments, so the intended dev process did not reliably start.

The project also needed a smaller live validation path than a full inventory
scan. A full scan is too risky as the first proof of input delivery because it
can click many tiles before a bad coordinate, focus issue, or blocked input is
understood.

### Decision

Keep automatic scan input automation read-only and require an elevated runtime
when Windows reports that automation would otherwise be blocked. Replace the
old `dev-admin.cmd` entry with `scripts/dev-admin.ps1`, quote the elevated
PowerShell arguments explicitly, and log elevated startup to
`outputs/admin-start/admin-dev.log`.

Add dev-only HTTP checks:

- `/automation/probe-click?index=N` or `?row=R&col=C` performs one safe
  inventory selection click and verifies whether the detail panel changed.
- `/scanner/start?limit=N` sends a temporary scan-limit payload to the renderer,
  so live auto-scan validation can start with two items instead of the UI
  default.

Document the workflow in [AUTOMATION_LIVE_SCAN.md](AUTOMATION_LIVE_SCAN.md).

### Consequences

- The user still has to approve Windows UAC manually; the app must not try to
  click the Secure Desktop prompt.
- We can distinguish input delivery from OCR/parser quality with a one-click
  probe before running any broader scan.
- Live validation now has a low-risk path: check elevation and Genshin
  detection, run a single probe click, then run a bounded `limit=2` scan.
- The implementation remains inside the allowed safety boundary: no memory
  reads, hooks, injection, game-file modification, deleting, feeding, enhancing,
  locking/unlocking, or spending resources.

## ADR-011: Retire Alternate OCR Comparison Paths After Current Scanner Baseline

### Status

Superseded

### Context

The project previously carried alternate OCR-engine and comparison paths to
decide whether the current scanner should change OCR defaults. The current
visible-inventory path has now produced clean 100-artifact evidence, and the
next product phase is result clarity, artifact inventory, detail evaluation, and
corpus growth. Keeping an alternate engine path increases app size and
maintenance surface without serving the current user workflow.

### Decision

Retire the alternate OCR-engine and current-vs-reference comparison paths. Keep a
single current OCR/capture path and preserve the quality-gated live soak
assessment. A qualified scan result must:

- finish cleanly,
- parse at least the requested count,
- keep miss rate at or below 2%,
- keep review rate at or below 15%,
- report active scan timing and bottlenecks,
- come from a runtime whose `/health.appBuild.signature` matches the current
  source.

`scripts/live-soak.ps1` writes the evidence bundle and
`scan-performance-assessment.json`. `npm run scan:assessment:test` verifies that
the ranking logic rejects fast but low-quality synthetic runs without needing
Genshin. `npm run scan:assessment:validate -- --summary` prints the assessment
path and `createdAt` timestamp so reports can cite the exact evidence file.

### Consequences

- Speed claims cannot be based on click count or elapsed time alone.
- A new OCR engine is out of scope until scanner UX, inventory, detail
  evaluation, and corpus growth justify reopening that work.
- Stale elevated Electron instances are treated as invalid evidence, not as a
  harmless warning.
- The validated `:wait` scan scripts are acceptable for manual post-UAC startup;
  non-waiting scripts remain useful when automation should fail fast.

## ADR-012: Separate Live Scan Results, Artifact Inventory, And Value Evaluation

### Status

Accepted

### Context

The visible-inventory scanner baseline is fast enough for the current product
phase. The next user value comes from better artifact content extraction and a
clearer surface for scanned artifacts, not from another broad speed pass.

The scan page currently has more diagnostic/result detail than the operator
needs during a live scan. Showing every stat, confidence value, and evaluation
while the scanner is running makes the main workspace noisy and increases the
risk that a user treats uncertain OCR as a final artifact judgment.

The desired product flow is:

- left side: screenshot/preview,
- right side: compact live list of finished artifact results,
- later menu item: browsable scanned artifact inventory,
- click-through detail: screenshot, parsed stats, OCR confidence, evaluation
  reasons, and upgrade projection.

### Decision

Adopt a split scan/result/inventory model:

- The active scan page shows the latest screenshot/preview and a compact live
  result rail.
- The live rail shows only scan number, artifact name or compact fallback,
  artifact value score `0-100`, and a result pill after evaluation has finished.
- Extraction confidence and artifact value are separate data concepts. A low
  confidence read becomes `Review`; it must not be silently displayed as a
  normal weak artifact.
- Debug stats, raw OCR, confidence breakdowns, and detailed evaluation belong in
  diagnostics, scan summary, or artifact detail.
- Add a scanned artifact inventory view for browsing, filtering, sorting, and
  opening details.
- Add artifact detail evaluation before promoting broad recommendations.
- Add upgrade projection only as a detail-level feature, with worst/middle/best
  projected value scores and clear uncertainty labeling.
- Keep the producer/consumer screenshot queue as a later implementation phase:
  one game-control worker may click/scroll/capture, while OCR/parse/evaluation
  workers can process bounded queued jobs once the UI/data contracts are stable.

### Consequences

- The scan page becomes calmer and more task-focused.
- The app can surface useful artifact outcomes without hiding OCR uncertainty.
- Inventory and detail views become the natural place for richer analysis.
- Recommendation work has a cleaner dependency chain: trusted scans, compact
  results, artifact inventory, detail evaluation, then build recommendations.
- Renderer-loop speed work remains secondary unless measured timings show a real
  regression; native capture speed work is handled by ADR-013.

## ADR-013: Move High-Speed Scanning Into The Native Helper And Vendor IK Inventorylists

### Status

Accepted

### Context

Inventory Kamera is fast because the game-control loop is native and the UI does
not perform per-artifact work. The Electron renderer path paid for focus,
capture, OCR, parse, persistence, status updates, and UI state inside one loop.
The user also has Inventory Kamera 1.4.4 locally, including `inventorylists`
that can be used 1:1 as scanner reference data.

### Decision

Use the C# input helper as the high-speed scanner service. Electron starts,
stops, packages, and reports status. React remains a visual control surface. The
IK `inventorylists` are copied into `data/ik-inventorylists` and packaged as
extra resources. The first native milestone captures artifact detail card crops
quickly; OCR, parsing, GOOD persistence, and evaluation are separate downstream
steps. Each native run writes `manifest.json`, `capture-jobs.jsonl`, and
`status.json`; downstream workers must consume those files instead of asking the
renderer to do per-artifact scanner work.

The first downstream worker is a post-capture processor that reads
`capture-jobs.jsonl` and writes both `scan-results.json` and
`processing-report.json`. `scan-results.json` is the durable result contract:
one entry per captured artifact card with extraction status, artifact identity
when parsed, review state, and deferred value status. `processing-report.json`
remains the diagnostic OCR/parse report. The worker OCRs/parses after capture
and does not persist by default; DB writes require explicit opt-in after native
crop OCR is validated with live evidence.

### Consequences

- Capture throughput can be optimized without renderer roundtrips per artifact.
- The scanner data source now matches IK's artifact names and set/piece keys.
  Other vendored IK lists are data only until those values are actually scanned.
- Evaluation can remain deferred without blocking capture speed.
- Live safety stays read-only: no memory reads, hooks, injection, game-file
  modification, deleting, feeding, enhancing, or spending resources.
- Non-16:9 layouts currently block in native preflight until layout support is
  expanded.