Improve scanner repeatability guardrails

This commit is contained in:
AzuTear
2026-07-09 10:21:13 +02:00
parent 13fd46c104
commit c10f17b4ef
13 changed files with 356 additions and 39 deletions
+38 -4
View File
@@ -144,6 +144,37 @@ Current status after the 2026-07-09 merge to `main`:
end-to-end capture roundtrip time. Use `averageCaptureRoundTripMs` and
`averageCaptureRoundTripOverheadMs` in the next `limit=20` live iteration to
decide whether the next cut belongs in native capture transport or OCR.
- **Repeatability and capture-overhead guardrails (2026-07-09)** -
`scripts/live-soak.ps1` now writes capture roundtrip and roundtrip-overhead
timing into `scan-performance-assessment.json`. The assessment validator can
enforce optional speed budgets with `--max-active-average-ms` and
`--max-capture-roundtrip-overhead-ms`, and `npm run scan:repeatability:wait`
runs 20/45/100 current-engine passes as repeatability evidence without
presenting them as an IK comparison. `-RepeatabilityRun` now sets those limits
inside PowerShell, and the script refuses unsafe limits above 1800 so npm/cmd
argument parsing cannot accidentally turn `20,45,100` into one oversized run.
- **Distinctive partial piece recovery (2026-07-09)** - a live repeatability run
exposed four identical OCR misses where the piece name was read as
`Wontiroms Creation pan`. The parser now derives a piece only when a long OCR
fragment uniquely matches exactly one known artifact piece. This recovered the
local case as `Sharpness That Ceased Upon Wondrous Creation` /
`Disenchantment in Deep Shadow` without adding a broad fuzzy exception.
- **Repeatability live pass after parser fix (2026-07-09)** -
`outputs/live-soak/2026-07-09T09-29-11/scan-performance-assessment.json`
captured a clean current-engine 20-artifact run: `20/20` parsed, `0` review,
`0` misses, `336 ms/artifact` active average, `318 ms` average capture
roundtrip, and `138 ms` average roundtrip overhead. The strict 3 artifacts per
second budget still failed by 3 ms (`336 ms` vs `333 ms`).
- **3/s follow-up experiments (2026-07-09)** - tested and rejected several
shortcut-style optimizations because live runs got slower or added risk:
skipping Paimon-menu analysis, skipping lock-state as a production shortcut,
reducing the artifact-level crop scale, and raising the OCR worker pool to 6.
The kept low-risk changes are fast-profile OCR crop priority and avoiding a
duplicate DataURL string when the native helper already returns Base64. A
follow-up clean 20-artifact run after payload cleanup reached `351 ms/artifact`,
`331 ms` capture roundtrip, and `146 ms` roundtrip overhead, so the next
credible 3/s work is native capture transport/roundtrip reduction, not UI
recommendation work.
- **3/s live attempt (2026-07-08)** - the missing-detail-preview review trigger
was fixed and tested. The best clean 20-artifact run reached `7285 ms`
(`364 ms/artifact`, about `2.75 artifacts/second`) with 0 review and 0 misses.
@@ -218,14 +249,17 @@ resolution or without UI work best tested live:
substat OCR time, or a larger capture pipeline change that avoids full-frame
PNG/Base64 transport without hurting safety checks. The target remains
`<= 6667 ms` elapsed for 20 parsed artifacts with 0 misses and no silent OCR
review regression.
review regression. The latest clean 20-artifact repeatability run reached
`336 ms/artifact`, so 3/s remains close but unproven.
6. **Broader scan soak test** — direct-GDI current-engine runs now passed at
`20/20`, `45/45`, and `100/100` with 0 misses. Continue with repeat runs if
duplicate rate needs tuning.
`20/20`, `45/45`, and `100/100` with 0 misses. Continue with
`npm run scan:repeatability:wait` in later sessions to check duplicate rate,
scroll behavior, and capture roundtrip timing without changing defaults.
7. **Repeatability pass** — repeat the qualified current-vs-IK-traineddata run
in a later live session before making major OCR-engine defaults or speed
claims beyond this environment.
claims beyond this environment. Current-engine-only repeatability is useful
evidence, but it is not an IK parity claim.
Visible-page limits up to 20, scroll/page-transition limit 45, the final
100-artifact current-vs-IK-traineddata comparison, equipped footer live smokes,