genshin-assistant/docs/DECISIONS.md

Decisions

This document contains Architecture Decision Records.

ADR Index

ID	Title	Status	Date
ADR-001	Build a local Electron app first	Accepted	2026-07-04
ADR-002	Use screen capture as the primary scanner source	Accepted	2026-07-04
ADR-003	Keep APIs and GOOD compatibility optional	Accepted	2026-07-04
ADR-004	Treat in-game marking as a later opt-in feature	Accepted	2026-07-04
ADR-005	Use a generated Genshin data package for OCR matching	Accepted	2026-07-04
ADR-006	Persistent input helper and JSON artifact store before SQLite	Accepted	2026-07-04
ADR-007	Measure OCR accuracy with a labeled eval harness before reworking the scanner	Accepted	2026-07-05
ADR-008	Replace the PowerShell input/capture helper with a C# sidecar	Accepted	2026-07-05
ADR-009	Resolution-anchored layout profiles and OCR preprocessing over color detection	Accepted	2026-07-05
ADR-010	Elevated dev runner and bounded live automation probes	Accepted	2026-07-07
ADR-011	Quality-gated Inventory Kamera comparison before OCR default changes	Accepted	2026-07-07
ADR-012	Separate live scan results, artifact inventory, and value evaluation	Accepted	2026-07-09

ADR-001: Build A Local Electron App First

Status

Accepted

Context

The product needs a Windows desktop UI, local screen capture, possible overlay windows, and future optional input automation.

Decision

Use Electron with React and TypeScript for the MVP.

Consequences

• Fast UI iteration and easy local packaging.
• Electron main-process code must be treated as a separate boundary from renderer code.
• Native or Rust sidecars can be added later for high-performance capture/OCR work.

ADR-002: Use Screen Capture As The Primary Scanner Source

Status

Accepted

Context

The user wants an app that works without Inventory Kamera, Genshin Optimizer, Enka, or HoYoLAB as core dependencies.

Decision

Use local screen capture as the primary source. Current Smart Capture focuses Genshin, hides the app, captures the primary screen through Windows GDI, detects the artifact detail panel, then OCRs focused crops.

Consequences

• The app remains offline-first.
• OCR and crop reliability are core product risks.
• UI language, resolution, HDR, and game layout changes need explicit test coverage.

ADR-003: Keep APIs And GOOD Compatibility Optional

Status

Accepted

Context

External APIs and existing optimizer formats can speed up setup, but should not define the main user workflow.

Decision

Keep Enka, HoYoLAB, Akasha, Genshin Optimizer, and GOOD import/export as optional future compatibility layers.

Consequences

• The app can work without external accounts or cookies.
• Data package and scanner quality become more important.
• Compatibility can be added when it helps testing, migration, or export.

ADR-004: Treat In-Game Marking As A Later Opt-In Feature

Status

Accepted

Context

Locking or marking artifacts in game may save time, but input automation increases ToS and misclick risk.

Decision

Do not ship in-game marking in the scanner MVP. If implemented later, it must be off by default, reversible, whitelisted, previewed before execution, and stoppable with ESC or user mouse movement.

Consequences

• Early scanner work stays lower risk.
• App-internal triage remains the first decision layer.
• No delete, feed, enhance, or resource-spending automation is allowed.

ADR-005: Use A Generated Genshin Data Package For OCR Matching

Status

Accepted

Context

Hardcoded arrays for characters and artifact sets caused repeated scanner failures whenever the user tested a newer character, set, or artifact name.

Decision

Generate src/data/genshinGameData.json from genshin-db and use it as the local matching dictionary for artifact sets, artifact piece names, characters, slots, main stats, and substats.

Consequences

• The scanner can recognize new characters and sets as soon as the local data package is regenerated from an updated genshin-db.
• Parser logic stays generic and testable instead of growing one-off fixes.
• OCR still needs good crops and text quality; the data package improves recognition but cannot solve unreadable screenshots by itself.

ADR-006: Persistent Input Helper And JSON Artifact Store Before SQLite

Status

Accepted

Context

Per-action PowerShell scripts recompiled the Win32 interop for every click, scroll, and capture (1-2s each) and one Marshal::SizeOf call was broken in Windows PowerShell 5.1, so SendInput clicks silently never executed. Scan results were also not persisted anywhere; only review samples reached disk. Adding better-sqlite3 (native module) was considered too heavy for this step.

Decision

Run one persistent PowerShell helper process (compiled once, JSON protocol over stdin/stdout) for focus, cursor/ESC state, click, scroll, and GDI capture. Persist parsed artifacts into artifact-store.json in userData, deduplicated by a content signature that excludes the equipped character. Keep SQLite as the planned future store; the JSON store is the migration source.

Consequences

• Batch scans become fast enough to be testable and the failsafe (ESC or user mouse movement aborts) can poll cheaply between actions.
• Automated clicks are verified by checking that the parsed detail signature changed; repeated failures abort with a diagnosis hint instead of clicking blindly.
• Leveling an artifact changes its signature and creates a new record; rescan-merge is an open follow-up.
• If the helper process dies it is respawned on the next request; pending requests fail loudly instead of hanging.

ADR-007: Measure OCR Accuracy With A Labeled Eval Harness Before Reworking The Scanner

Status

Accepted

Context

OCR and crop reliability are the core product risk (ADR-002), but there was no way to measure field-level accuracy. Every OCR, crop, or parser change was a blind change - regressions could only be caught by a human re-testing against the live game, and there was no baseline to compare a new engine or preprocessing step against.

Decision

Add a field-level eval harness (src/eval/) that runs the real parseArtifactCandidate over a labeled corpus and reports per-field and per-case accuracy. Seed the corpus from the existing parser test cases (verified labels), and grow it from human-confirmed review samples via reviewSampleToEvalCase. Run it as a gate under npm test (must stay 100% on the verified seed) and as a full report via npm run eval.

Consequences

• OCR/layout/preprocessing changes are measured, not guessed at; a new engine or a preprocessing step has to beat a recorded baseline.
• The review queue does double duty: it flags artifacts for human correction and feeds the eval corpus. Review-sample parsed blocks are label candidates, never ground truth, to avoid the parser grading itself.
• The corpus label is the source of truth. If a code change intentionally alters a correct output, the label is updated in the same commit.

ADR-008: Replace The PowerShell Input/Capture Helper With A C# Sidecar

Status

Accepted

Context

The persistent PowerShell helper (ADR-006) still carries Windows PowerShell 5.1 quirks (the Marshal::SizeOf interop bug), compiles Win32 interop at startup, and captures each frame by writing a PNG to the temp directory and reading it back. Inventory Kamera - the proven reference for automated Genshin scanning - uses a C#/.NET stack with InputSimulator (SendInput) and direct GDI/BitBlt capture.

Decision

Replace the PowerShell helper with a self-contained .NET (C#) sidecar that speaks the same JSON-over-stdin/stdout protocol, so the Electron-side InputHelperService interface stays stable. The sidecar does per-monitor DPI-aware SendInput click/scroll, BitBlt client-rect capture returning bytes without a temp file, and elevation detection.

Consequences

• No PS 5.1 marshalling bugs, no per-call interop compile, no temp-PNG churn; lower latency makes batch scans and the ESC/mouse failsafe polling cheaper.
• Adds a .NET build/publish step and ships a compiled exe with the app.
• The migration is behind the existing service interface, so the renderer and scan loop do not change.

ADR-009: Resolution-Anchored Layout Profiles And OCR Preprocessing Over Color Detection

Status

Accepted

Context

The current pipeline finds the artifact detail panel with hardcoded orange/green color thresholds (inferDetailRect) and then crops fixed percentages of that guessed rectangle. This is brittle against HDR, color profiles, UI scale, aspect ratio, and game UI updates. Inventory Kamera instead requires borderless 16:9 and scales fixed crop coordinates from a reference resolution, then feeds Tesseract preprocessed (grayscale, upscaled, thresholded) crops - which is why general Tesseract is accurate enough for them.

Decision

Adopt the same approach: require borderless 16:9, drive crops from resolution-anchored layout profiles scaled from a reference resolution (color detection only as a fallback), and add a per-region preprocessing pass (grayscale, upscale, threshold, invert) plus a digit-whitelist mode for numeric fields. The set name stays derived from the static piece-to-set data package (no dedicated set-effect crop). English-only OCR is accepted. Every change is validated against the ADR-007 eval harness.

Consequences

• Crop positions become deterministic per resolution instead of per-frame guesses.
• Preprocessing is expected to lift accuracy enough that a custom OCR engine is only pursued if the eval harness shows Tesseract-plus-preprocessing is insufficient.
• Non-16:9 or non-borderless setups are explicitly unsupported for the auto scanner; the app should detect and warn rather than silently misread.

ADR-010: Elevated Dev Runner And Bounded Live Automation Probes

Status

Accepted

Context

Automatic grid scanning needs read-only mouse movement, click, and wheel input to reach the focused Genshin window. A lower-integrity app can fail to deliver input to an elevated or protected target because of Windows UIPI/integrity boundaries. During live testing, npm run dev:admin originally printed that a new Administrator window was started, but the elevated PowerShell received no arguments, so the intended dev process did not reliably start.

The project also needed a smaller live validation path than a full inventory scan. A full scan is too risky as the first proof of input delivery because it can click many tiles before a bad coordinate, focus issue, or blocked input is understood.

Decision

Keep automatic scan input automation read-only and require an elevated runtime when Windows reports that automation would otherwise be blocked. Replace the old dev-admin.cmd entry with scripts/dev-admin.ps1, quote the elevated PowerShell arguments explicitly, and log elevated startup to outputs/admin-start/admin-dev.log.

Add dev-only HTTP checks:

• /automation/probe-click?index=N or ?row=R&col=C performs one safe inventory selection click and verifies whether the detail panel changed.
• /scanner/start?limit=N sends a temporary scan-limit payload to the renderer, so live auto-scan validation can start with two items instead of the UI default.

Document the workflow in AUTOMATION_LIVE_SCAN.md.

Consequences

• The user still has to approve Windows UAC manually; the app must not try to click the Secure Desktop prompt.
• We can distinguish input delivery from OCR/parser quality with a one-click probe before running any broader scan.
• Live validation now has a low-risk path: check elevation and Genshin detection, run a single probe click, then run a bounded limit=2 scan.
• The implementation remains inside the allowed safety boundary: no memory reads, hooks, injection, game-file modification, deleting, feeding, enhancing, locking/unlocking, or spending resources.

ADR-011: Quality-Gated Inventory Kamera Comparison Before OCR Default Changes

Status

Accepted

Context

The product target is not merely to click through 100 artifacts quickly. It is to scan the first 100 artifacts with accuracy at least as good as Inventory Kamera and speed equal to or better than Inventory Kamera. A faster scan that creates too many misses, review samples, or false positives is worse than a slower qualified run.

The app can now compare the current OCR path with Inventory Kamera's genshin_fast_09_04_21.traineddata through the same visible crop set. It also has hot-loop timing fields for capture, OCR, card readiness, scroll readiness, active scan time, and projected 100-artifact time.

Decision

Use a quality-gated live soak and benchmark before changing the default OCR engine or claiming IK parity. A qualified scan result must:

• finish cleanly,
• parse at least the requested count,
• keep miss rate at or below 2%,
• keep review rate at or below 15%,
• report active scan timing and bottlenecks,
• come from a runtime whose /health.appBuild.signature matches the current source.

scripts/live-soak.ps1 writes the evidence bundle and scan-performance-assessment.json. npm run scan:assessment:test verifies that the ranking logic rejects fast but low-quality synthetic runs without needing Genshin. The assessment also records goal100Decision and goal100.comparisonComplete; IK-target claims require a qualified 100-artifact winner and a complete current vs. ik-traineddata comparison. npm run scan:assessment:validate -- --summary prints the assessment path and createdAt timestamp so reports can cite the exact evidence file.

Consequences

• Speed claims cannot be based on click count or elapsed time alone.
• A new OCR engine cannot become the default just because it is theoretically closer to IK; it must win the same-capture benchmark and a qualified live run.
• Stale elevated Electron instances are treated as invalid evidence, not as a harmless warning.
• The validated :wait scan scripts are acceptable for manual post-UAC startup; non-waiting scripts remain useful when automation should fail fast.
• The goal remains open until the 100-artifact qualified comparison is captured.

ADR-012: Separate Live Scan Results, Artifact Inventory, And Value Evaluation

Status

Accepted

Context

The visible-inventory scanner baseline is fast enough for the current product phase. The next user value comes from better artifact content extraction and a clearer surface for scanned artifacts, not from another broad speed pass.

The scan page currently has more diagnostic/result detail than the operator needs during a live scan. Showing every stat, confidence value, and evaluation while the scanner is running makes the main workspace noisy and increases the risk that a user treats uncertain OCR as a final artifact judgment.

The desired product flow is:

• left side: screenshot/preview,
• right side: compact live list of finished artifact results,
• later menu item: browsable scanned artifact inventory,
• click-through detail: screenshot, parsed stats, OCR confidence, evaluation reasons, and upgrade projection.

Decision

Adopt a split scan/result/inventory model:

• The active scan page shows the latest screenshot/preview and a compact live result rail.
• The live rail shows only scan number, artifact name or compact fallback, artifact value score 0-100, and a result pill after evaluation has finished.
• Extraction confidence and artifact value are separate data concepts. A low confidence read becomes Review; it must not be silently displayed as a normal weak artifact.
• Debug stats, raw OCR, confidence breakdowns, and detailed evaluation belong in diagnostics, scan summary, or artifact detail.
• Add a scanned artifact inventory view for browsing, filtering, sorting, and opening details.
• Add artifact detail evaluation before promoting broad recommendations.
• Add upgrade projection only as a detail-level feature, with worst/middle/best projected value scores and clear uncertainty labeling.
• Keep the producer/consumer screenshot queue as a later implementation phase: one game-control worker may click/scroll/capture, while OCR/parse/evaluation workers can process bounded queued jobs once the UI/data contracts are stable.

Consequences

• The scan page becomes calmer and more task-focused.
• The app can surface useful artifact outcomes without hiding OCR uncertainty.
• Inventory and detail views become the natural place for richer analysis.
• Recommendation work has a cleaner dependency chain: trusted scans, compact results, artifact inventory, detail evaluation, then build recommendations.
• Speed work remains possible, but it is secondary unless measured timings show a real regression.