feat(corpus): require retrievable visual review evidence
This commit is contained in:
AzuTear
@@ -370,9 +370,11 @@ npm run eval:review-candidates -- --limit=80
|
||||
```
|
||||
|
||||
Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
|
||||
worklist, not ground truth. Only after expected fields are confirmed or
|
||||
corrected against the real artifact should a case move into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
worklist, not ground truth. Only choose a candidate marked
|
||||
`visualEvidence: available`: it had a retrievable local PNG at export time.
|
||||
The export exposes only safe capture/run references, never the local crop path.
|
||||
After expected fields are confirmed or corrected against the real artifact, a
|
||||
case may move into `src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
For a manually checked candidate, generate a paste-ready confirmed-case snippet:
|
||||
|
||||
@@ -380,6 +382,10 @@ For a manually checked candidate, generate a paste-ready confirmed-case snippet:
|
||||
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
|
||||
```
|
||||
|
||||
The preparer refuses candidates without explicit labels or retrievable visual
|
||||
evidence. Re-export after retaining a crop rather than promoting an OCR-only
|
||||
legacy record.
|
||||
|
||||
## Anti-Cheat And Safety Boundary
|
||||
|
||||
Do not describe the implementation as bypassing anti-cheat. The app does not
|
||||
|
||||
@@ -142,12 +142,13 @@ the legacy scorer or emit legacy builds/recommendations.
|
||||
1. The saved 20/50/100/2,211 validation gates passed for this checkpoint;
|
||||
preserve them for later Build-Fit changes and do not weaken scanner review
|
||||
gates.
|
||||
2. Perform a packaged renderer/UI acceptance before calling the new Builds
|
||||
surface live-accepted.
|
||||
2. The packaged renderer/UI acceptance passed. Keep it distinct from the
|
||||
optional elevated packaged-executable runtime/UAC claim.
|
||||
3. Refresh every profile before its explicit expiry and add Golden plus
|
||||
Review/low-confidence adversarial coverage for every new profile.
|
||||
4. Visually label the remaining 136 Review crops before changing parser or
|
||||
profile rules; do not relax gates from aggregate statistics.
|
||||
4. The historical 136 Review crops were visually audited. For all new Review
|
||||
cases, require visual confirmation before changing parser or profile rules;
|
||||
do not relax gates from aggregate statistics.
|
||||
5. Add later profiles only when their team/weapon/rotation assumptions can be
|
||||
represented explicitly without hidden constants. No suggestion may imply
|
||||
deletion, feeding, enhancement, or automatic in-game mutation.
|
||||
|
||||
+18
-6
@@ -45,6 +45,10 @@ The fresh current-build smoke is documented in
|
||||
the current corpus work: 128 are explicit 1–4★ entries, six have the same
|
||||
exact duplicated-decimal OCR defect, one requires slot-safe name resolution,
|
||||
and one incomplete crop intentionally remains `Review`.
|
||||
- The review-corpus exporter now exposes only safe capture/run references and
|
||||
prioritizes candidates with a retrievable local crop. The confirmation
|
||||
preparer refuses OCR-only records, so a parser guess cannot become ground
|
||||
truth without retained visual evidence.
|
||||
|
||||
## Verified Evidence
|
||||
|
||||
@@ -74,6 +78,11 @@ does not certify the current change set by itself:
|
||||
`native-scanner-run-timing-v1` record. Request-to-reconciliation was
|
||||
`28,022 ms` (`~1.78 Artifacts/s`); that is a functional/safety result, not a
|
||||
3/s performance claim.
|
||||
- The follow-on review-evidence workflow passed focused exporter/preparer
|
||||
checks plus `npm run lint`, `npm test` (72 files / 446 tests), and
|
||||
`npm run build`. A read-only export of the current local queue found 177
|
||||
records and three retrievable crops; all three are already-confirmed
|
||||
`native-review-approved` cases, so no new label was promoted.
|
||||
|
||||
## What Still Needs Testing
|
||||
|
||||
@@ -109,9 +118,12 @@ does not certify the current change set by itself:
|
||||
|
||||
1. Keep the passing built-renderer acceptance distinct from the optional
|
||||
elevated packaged-executable runtime/UAC check.
|
||||
2. Clean generated workspace outputs, commit the coherent checkpoint, and push
|
||||
the intended branch; the current-build 50-artifact live gate has passed.
|
||||
3. Continue the next quality phase with visually confirmed corpus growth,
|
||||
fail-closed parser behavior, profile-source refreshes, and packaged UI
|
||||
regression coverage. Do not broaden game-category scope or add in-game
|
||||
mutation.
|
||||
2. The generated workspace outputs were cleaned and the coherent checkpoint
|
||||
was committed and pushed as `8b9f948` on 2026-07-11. The active admin
|
||||
runtime's locked log was intentionally left untouched.
|
||||
3. Re-export future Review cases and only label candidates marked with
|
||||
retrievable visual evidence. No currently retrievable local candidate needs
|
||||
a new corpus label.
|
||||
4. Continue the next quality phase with fail-closed parser behavior,
|
||||
profile-source refreshes, and packaged UI regression coverage. Do not
|
||||
broaden game-category scope or add in-game mutation.
|
||||
|
||||
@@ -102,13 +102,15 @@ not a guessed canonical artifact.
|
||||
`run-timing.json`. It measured `28,022 ms` request-to-reconciliation, so it
|
||||
does not claim the separate 3/s performance target.
|
||||
|
||||
3. Remove generated workspace outputs, commit the coherent checkpoint, and
|
||||
push the intended branch.
|
||||
3. Completed on 2026-07-11: generated workspace outputs were removed and the
|
||||
coherent checkpoint was committed and pushed as `8b9f948`. The running
|
||||
admin runtime's locked log was intentionally preserved.
|
||||
|
||||
## Follow-on Quality Phase
|
||||
|
||||
- Extend the corpus only from visually confirmed crops and preserve ambiguity as
|
||||
`Review`.
|
||||
- Extend the corpus only from new visually confirmed crops and preserve
|
||||
ambiguity as `Review`. The historical 136-item Review population has already
|
||||
been audited; it is not an outstanding relabeling task.
|
||||
- Refresh Build-Fit source profiles before their explicit expiry and add Golden
|
||||
plus adversarial fixtures for each profile expansion.
|
||||
- Keep the bounded wheel reset as the production baseline; test any alternate
|
||||
|
||||
@@ -217,11 +217,14 @@ Nachweis.
|
||||
- Eine echte Review-Korrektur oder Promotion wurde nicht absichtlich in den
|
||||
Store geschrieben. Die statischen und Main-Process-Gates bestehen; der
|
||||
Mutation-Pfad bleibt ein separater, explizit bestaetigter QA-Fall.
|
||||
- Die verbliebenen 136 Review-Faelle des kompletten Corpus sind der naechste
|
||||
Scanner-Qualitaetsbestand: 128 tragen `substats_implausible`, 65
|
||||
`substats_incomplete`, 28 eine niedrige Main-Value-Konfidenz und 4 einen
|
||||
IK-Mismatch; Gruende koennen sich ueberlappen. Sie werden nicht automatisch
|
||||
als sauber umetikettiert.
|
||||
- Die damals verbliebenen 136 Review-Faelle des kompletten Corpus waren der
|
||||
naechste Scanner-Qualitaetsbestand. Sie wurden am 2026-07-11 visuell gegen
|
||||
die gespeicherten Crops auditiert: 128 sind explizite 1–4★-Ausschluesse,
|
||||
sechs belegen die exakte doppelte Dezimalpunkt-Korrektur, ein Fall erhielt
|
||||
eine slot-sichere Namensaufloesung und ein unvollstaendiger Crop bleibt
|
||||
bewusst `Review`. Details stehen in
|
||||
`LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md`; neue Fälle
|
||||
werden weiterhin nicht automatisch als sauber umetikettiert.
|
||||
|
||||
Die score-freie Grundlage von `build-fit-contract-v1` ist inzwischen
|
||||
implementiert: gesourcte Eingaben, OCR-/Review-Gates, Set-/Mainstat-/Substat-
|
||||
|
||||
@@ -36,8 +36,8 @@ reported separately; they are not added together.
|
||||
|
||||
## Follow-Up
|
||||
|
||||
The required scanner release gate is complete. The next approved actions are
|
||||
workspace cleanup, a coherent repository commit, and push of the current
|
||||
checkpoint. The remaining quality work is visually confirmed corpus growth,
|
||||
source-profile refresh, and a separate elevated packaged-runtime check only if
|
||||
that specific binary-runtime claim is needed.
|
||||
The required scanner release gate and its checkpoint publication are complete:
|
||||
the workspace was cleaned and commit `8b9f948` was pushed on 2026-07-11. The
|
||||
remaining quality work is new visually confirmed corpus growth, source-profile
|
||||
refresh, and a separate elevated packaged-runtime check only if that specific
|
||||
binary-runtime claim is needed.
|
||||
|
||||
+16
-8
@@ -282,7 +282,10 @@ confirmation probes. That is not an elevated packaged-executable claim. The
|
||||
current elevated development 50-artifact native smoke now passes with complete
|
||||
timing evidence and zero writes; its `28,022 ms` end-to-end result does not
|
||||
prove the separate 3/s target. The remaining Review corpus continues to grow
|
||||
only from visually confirmed labels.
|
||||
only from visually confirmed labels. The review exporter now prioritizes
|
||||
retrievable retained crops and serializes only safe capture/run references; its
|
||||
confirmation preparer refuses OCR-only candidates so parser guesses cannot be
|
||||
promoted without visual evidence.
|
||||
|
||||
## Product Direction
|
||||
|
||||
@@ -531,18 +534,23 @@ Status:
|
||||
reconciliation, zero Review/errors/writes, and
|
||||
`requestToResultsReconciledMs=28,022`. This is a functional/safety gate;
|
||||
it does not claim 3 Artifacts/s.
|
||||
4. Clean generated workspace output, commit the coherent checkpoint, and push
|
||||
the intended branch. A package-runtime/UAC test is required only for that
|
||||
separate claim.
|
||||
5. Continue review-corpus growth only from visually confirmed crops. Keep
|
||||
4. Completed on 2026-07-11: generated workspace output was cleaned and the
|
||||
coherent checkpoint was committed and pushed as `8b9f948`. A
|
||||
package-runtime/UAC test is required only for that separate claim.
|
||||
5. The visual-evidence review exporter/preparer passed focused checks plus
|
||||
`npm run lint`, `npm test` (72 files / 446 tests), and `npm run build`.
|
||||
Its read-only export of the current 177-record local queue found three
|
||||
retrievable crops, all already-confirmed `native-review-approved` cases;
|
||||
no new corpus label was promoted.
|
||||
6. Continue review-corpus growth only from new visually confirmed crops. Keep
|
||||
explicit 1–4★ evidence excluded, ambiguous evidence in `Review`, and reject
|
||||
aliases that cannot be justified by independent field evidence.
|
||||
6. Refresh source-bound profiles before expiry and add Golden plus
|
||||
7. Refresh source-bound profiles before expiry and add Golden plus
|
||||
Review/low-confidence adversarial fixtures for each additional profile.
|
||||
7. Keep the bounded wheel reset as the production baseline. Benchmark a `B`
|
||||
8. Keep the bounded wheel reset as the production baseline. Benchmark a `B`
|
||||
plus Artifact-tab reopen only as a separate guarded experiment before any
|
||||
reset strategy change.
|
||||
8. Keep Build-Fit profile suggestions read-only and Artifact-only until an
|
||||
9. Keep Build-Fit profile suggestions read-only and Artifact-only until an
|
||||
explicit product decision adds owned-character, weapon, team, and rotation
|
||||
inputs with their own sources.
|
||||
|
||||
|
||||
+28
-14
@@ -52,12 +52,21 @@ This writes:
|
||||
- `outputs/review-eval-candidates/review-eval-candidates.json`
|
||||
- `outputs/review-eval-candidates/review-eval-candidates.md`
|
||||
|
||||
The exporter deduplicates samples, puts complete modern OCR captures first,
|
||||
marks missing fast-profile fields so stale/partial captures do not crowd out
|
||||
useful cases, and surfaces ownership/lock evidence (`artifact-footer`,
|
||||
`equipped`, and `locked=true/false`) for the next validation pass.
|
||||
The exporter deduplicates samples, puts candidates with retrievable local visual
|
||||
evidence first, then preserves the existing complete-OCR/staleness ordering
|
||||
within each evidence group. It marks missing fast-profile fields so
|
||||
stale/partial captures do not crowd out useful cases, and surfaces
|
||||
ownership/lock evidence (`artifact-footer`, `equipped`, and `locked=true/false`)
|
||||
for the next validation pass.
|
||||
|
||||
Current local snapshot from 2026-07-10:
|
||||
Each candidate records a safe `captureId`, `capturedAt`, and
|
||||
`visualEvidence.status`. `available` means the referenced local PNG existed at
|
||||
export time; `unavailable` is an OCR-only legacy/no-path case. The export never
|
||||
serializes the local PNG path. Only label an `available` candidate after opening
|
||||
that retained crop and checking the real artifact. Re-export if the crop was
|
||||
removed; the preparer deliberately refuses `unavailable` candidates.
|
||||
|
||||
Current local snapshot re-exported on 2026-07-11:
|
||||
|
||||
- 177 records read, 0 invalid
|
||||
- 138 unique candidates
|
||||
@@ -65,16 +74,18 @@ Current local snapshot from 2026-07-10:
|
||||
- 36 candidates with complete fast-profile fields
|
||||
- 41 likely stale or partial captures
|
||||
- 14 equipped-footer candidates
|
||||
- 3 `native-review-approved` cases already represented in
|
||||
`confirmedReviewCorpus.ts`; the remaining 77 exported cases still require
|
||||
explicit human labels
|
||||
- 3 candidates with retrievable visual evidence, all
|
||||
`native-review-approved` cases already represented in
|
||||
`confirmedReviewCorpus.ts`
|
||||
- 77 OCR-only/unavailable exported cases, which must not be prepared as
|
||||
confirmed corpus labels until a retained crop is available
|
||||
|
||||
These counts describe the current local queue and may grow after later live
|
||||
sessions. Do not treat the 36 complete-field candidates as automatically
|
||||
correct; complete OCR is still only a review candidate until visually checked.
|
||||
|
||||
After manually checking one candidate against the real artifact, create a
|
||||
confirmed corpus snippet with explicit expected labels:
|
||||
After manually checking one `visualEvidence: available` candidate against the
|
||||
real artifact, create a confirmed corpus snippet with explicit expected labels:
|
||||
|
||||
```powershell
|
||||
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
|
||||
@@ -83,15 +94,18 @@ npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\pat
|
||||
The script reads the latest
|
||||
`outputs/review-eval-candidates/review-eval-candidates.json` by default and
|
||||
writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`.
|
||||
It refuses to run without explicit labels, so parser guesses are not silently
|
||||
promoted to ground truth. Review that snippet, then paste the object into
|
||||
It refuses to run without explicit labels or retrievable visual evidence, so
|
||||
parser guesses and OCR-only legacy records are not silently promoted to ground
|
||||
truth. The generated provenance note includes the safe capture/run reference
|
||||
and timestamp, never an absolute local crop path. Review that snippet, then paste the object into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
The parser's guess is a label **candidate, not ground truth** (using it directly
|
||||
would be the parser grading itself). To add a real case:
|
||||
|
||||
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
|
||||
2. Open each produced case and confirm or correct the `expect` values against
|
||||
1. Export review samples and choose a candidate marked
|
||||
`visualEvidence: available`.
|
||||
2. Open its retained crop and confirm or correct the `expect` values against
|
||||
what the artifact actually is in-game. Set `confirmed: true`.
|
||||
3. Move the corrected case into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports
|
||||
|
||||
@@ -41,11 +41,11 @@ checkpoint, see
|
||||
|
||||
The implementation now has an English-default/Deutsch-selectable renderer,
|
||||
confirmed local artifact removal, a shared-clock native run timing record, and
|
||||
new visually reviewed OCR corpus cases. These are implementation and offline
|
||||
validation results, not a new live-release claim. The remaining release gates
|
||||
include the non-persisting 50-artifact native smoke that produces a complete
|
||||
`run-timing.json`. The built-renderer Builds/UI acceptance already passed; an
|
||||
elevated packaged-executable runtime remains a separate packaging boundary.
|
||||
new visually reviewed OCR corpus cases. The required non-persisting current
|
||||
build 50-artifact native smoke passed with a complete `run-timing.json`, and
|
||||
the resulting checkpoint was published as `8b9f948`. The built-renderer
|
||||
Builds/UI acceptance also passed; an elevated packaged-executable runtime
|
||||
remains a separate packaging boundary.
|
||||
|
||||
## Target User Flow
|
||||
|
||||
|
||||
@@ -108,6 +108,12 @@ Detailed evidence:
|
||||
akzeptiert. 360 widerspruechliche `+0`-Main-Values werden jetzt nur dann aus
|
||||
der kanonischen 5-Sterne-Tabelle repariert, wenn vier Anfangs-Substats die
|
||||
Seltenheit strukturell beweisen; Drei-Substat-Faelle bleiben Review.
|
||||
- Der Review-Corpus-Exporter priorisiert jetzt nur noch Kandidaten mit
|
||||
abrufbarem lokalem Crop und gibt sichere Capture-/Run-Referenzen statt lokaler
|
||||
Pfade aus. Der Snippet-Preparer blockiert OCR-only Kandidaten ohne visuelle
|
||||
Evidenz. Ein read-only Export der aktuellen 177 Eintraege fand drei abrufbare,
|
||||
bereits bestaetigte `native-review-approved` Faelle; keine neue Label-Promotion
|
||||
wurde vorgenommen.
|
||||
- Native Helper und PowerShell-Fallback akzeptieren nur noch die exakten
|
||||
Prozessnamen `GenshinImpact` oder `YuanShen`. Bei geschlossenem Spiel bestand
|
||||
der Package-Smoke mit `genshinFound=false`, HWND 0 und leerem Zielprozess.
|
||||
@@ -153,8 +159,9 @@ Detailed evidence:
|
||||
offen; der frische native 50er-Smoke bestand als Scanner-Release-Gate.
|
||||
2. Profilquellen vor Ablauf erneuern und jedes weitere Profil wieder mit
|
||||
Golden-/adversarial Fixtures absichern.
|
||||
3. Die verbleibenden 136 Review-Faelle nur nach visueller Crop-Bestaetigung in
|
||||
Parser-Corpus oder Lernregeln uebernehmen.
|
||||
3. Neue Review-Faelle nur nach visueller Crop-Bestaetigung in Parser-Corpus
|
||||
oder Lernregeln uebernehmen. Die historische 136er-Population ist bereits
|
||||
auditiert; der eine unvollstaendige Crop bleibt bewusst `Review`.
|
||||
4. Result-Klarheit und explizite Mutation-QA weiter staerken.
|
||||
5. Erst nach eigenen Quellen fuer Charakterbesitz, Team, Waffe und Rotation
|
||||
eine breitere Optimizer- oder Empfehlungsphase erwägen.
|
||||
@@ -162,7 +169,7 @@ Detailed evidence:
|
||||
## Aktueller Nachweis
|
||||
|
||||
- `npm run acceptance:offline`: PASS
|
||||
- `npm test`: 444/444 in 72 Dateien
|
||||
- `npm test`: 446/446 in 72 Dateien
|
||||
- `npm run acceptance:offline`: final PASS nach Lokalisierung, lokaler
|
||||
Loeschung, Timing, Korpus und UI-Acceptance-Runner; Package-Verifier 27/27
|
||||
- Build-Fit-Ranker/Kontext/Profile: offline PASS; gebaute Renderer-UI PASS,
|
||||
|
||||
Reference in New Issue
Block a user