Improve scanner repeatability guardrails
This commit is contained in:
@@ -268,10 +268,20 @@ Latest live timing evidence on 2026-07-08:
|
||||
signature `2026-07-08-direct-gdi-reviewfix` completed `20/20`, `18` stored,
|
||||
`0` review, `2` duplicates, `0` misses, `7973 ms` elapsed, or
|
||||
`399 ms/artifact`. 3 artifacts/second is not proven.
|
||||
- Follow-up 3 artifacts/second attempts on 2026-07-09:
|
||||
after the distinctive partial piece parser fix, the best clean repeatability
|
||||
run reached `336 ms/artifact` with `20/20` parsed, `0` review, `0` misses,
|
||||
`318 ms` average capture roundtrip, and `138 ms` roundtrip overhead. Later
|
||||
runs with crop priority and image-payload cleanup stayed clean but ranged
|
||||
around `346-351 ms/artifact`; the strict `333 ms/artifact` budget remains
|
||||
unproven.
|
||||
- Rejected speed experiments:
|
||||
detail-region capture, `GAA_OCR_WORKERS=5`, DataURL-to-buffer decode, and
|
||||
substat OCR `PSM.SINGLE_COLUMN` were all live/benchmark tested and were slower
|
||||
than the direct-GDI baseline. Keep `GAA_OCR_WORKERS=4` for current runs.
|
||||
than the direct-GDI baseline. Later checks also rejected skipping
|
||||
`analyzePaimonMenu`, skipping lock-state as a production shortcut, reducing
|
||||
the artifact-level crop scale, and `GAA_OCR_WORKERS=6` as the default.
|
||||
Keep `GAA_OCR_WORKERS=4` for current runs.
|
||||
- Quality-gated current-vs-IK comparison:
|
||||
`npm run scan:goal:compare:validated` produced
|
||||
`outputs/live-soak/2026-07-08T18-38-35/scan-performance-assessment.json`
|
||||
@@ -449,15 +459,17 @@ uses the default `current` scan engine; use `scan:goal:ik` for a native
|
||||
IK-traineddata scan pass. Use `scan:goal:compare` to run both scan engines
|
||||
back-to-back with the same limits and one combined CSV. The CSV is the quickest evidence for
|
||||
`averageMsPerParsed`, `activeAverageMsPerParsed`, `averageCaptureMs`,
|
||||
`averageCaptureRoundTripMs`, `averageCaptureRoundTripOverheadMs`,
|
||||
`captureP50Ms`, `captureP90Ms`, `averageOcrMs`, `ocrP50Ms`, `ocrP90Ms`,
|
||||
`averageCardReadyMs`, `averageScrollReadyMs`, `artifactsPerMinute`, and
|
||||
`projectedMsFor100`.
|
||||
The run also writes `scan-performance-assessment.json`, which groups results by
|
||||
limit, picks the best qualified engine, and labels the dominant bottleneck as
|
||||
OCR, capture, card-ready, or scroll-ready. A qualified winner must finish the
|
||||
run, parse the requested count, keep miss rate under 2%, and keep review rate
|
||||
at or below 15%; review and miss rates are penalized before active average speed
|
||||
is used as the tie-breaker. For IK-target claims, check `goal100Decision`; it
|
||||
OCR, capture-roundtrip-overhead, capture, card-ready, or scroll-ready. A
|
||||
qualified winner must finish the run, parse the requested count, keep miss rate
|
||||
under 2%, and keep review rate at or below 15%; review and miss rates are
|
||||
penalized before active average speed is used as the tie-breaker. For IK-target
|
||||
claims, check `goal100Decision`; it
|
||||
must read `qualified-comparison: winner=<engine>`, and
|
||||
`goal100.comparisonComplete` must be `true` so a single-engine 100-artifact run
|
||||
is not mistaken for a current-vs-IK comparison.
|
||||
@@ -478,12 +490,38 @@ instead of the final 100-artifact proof. Add `--summary` when you want a short
|
||||
report-ready PASS/FAIL output that includes the input assessment path and
|
||||
assessment `createdAt` timestamp.
|
||||
|
||||
Optional budget flags are useful for the current speed work:
|
||||
|
||||
```powershell
|
||||
npm run scan:assessment:validate -- --latest --summary --limit=20 --max-active-average-ms=333 --max-capture-roundtrip-overhead-ms=120
|
||||
```
|
||||
|
||||
`--max-active-average-ms=333` is the strict 3 artifacts/second check. Use a
|
||||
separate `--max-capture-roundtrip-overhead-ms` budget when deciding whether the
|
||||
next optimization belongs in native capture transport instead of OCR.
|
||||
Single-engine repeatability runs may be validated with `--allow-single-engine`,
|
||||
but that mode is only for repeatability evidence and must not be used for IK
|
||||
parity claims.
|
||||
|
||||
`npm run scan:goal:compare:validated` is the preferred final command: it runs
|
||||
the live preflight first, then the full comparison, and then validates the
|
||||
newest assessment with `--summary`. Use
|
||||
`npm run scan:goal:compare:validated:wait` for the same final flow when starting
|
||||
immediately after UAC.
|
||||
|
||||
For later-session repeatability without changing OCR engines, use:
|
||||
|
||||
```powershell
|
||||
npm run scan:repeatability:wait
|
||||
```
|
||||
|
||||
That command runs the current visible-inventory engine at 20, 45, and 100
|
||||
artifacts, then validates the 100-artifact result as single-engine evidence.
|
||||
It uses the `live-soak.ps1 -RepeatabilityRun` switch instead of passing a
|
||||
comma-separated `-Limits` value through npm/cmd, because Windows argument
|
||||
parsing can collapse `20,45,100` into one unsafe number. The script also refuses
|
||||
limits above 1800 as a final guard.
|
||||
|
||||
The assessment ranking can be verified without Genshin or the Electron app:
|
||||
|
||||
```powershell
|
||||
|
||||
Reference in New Issue
Block a user