feat(corpus): require retrievable visual review evidence
This commit is contained in:
AzuTear
@@ -370,9 +370,11 @@ npm run eval:review-candidates -- --limit=80
|
||||
```
|
||||
|
||||
Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review
|
||||
worklist, not ground truth. Only after expected fields are confirmed or
|
||||
corrected against the real artifact should a case move into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
worklist, not ground truth. Only choose a candidate marked
|
||||
`visualEvidence: available`: it had a retrievable local PNG at export time.
|
||||
The export exposes only safe capture/run references, never the local crop path.
|
||||
After expected fields are confirmed or corrected against the real artifact, a
|
||||
case may move into `src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
For a manually checked candidate, generate a paste-ready confirmed-case snippet:
|
||||
|
||||
@@ -380,6 +382,10 @@ For a manually checked candidate, generate a paste-ready confirmed-case snippet:
|
||||
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
|
||||
```
|
||||
|
||||
The preparer refuses candidates without explicit labels or retrievable visual
|
||||
evidence. Re-export after retaining a crop rather than promoting an OCR-only
|
||||
legacy record.
|
||||
|
||||
## Anti-Cheat And Safety Boundary
|
||||
|
||||
Do not describe the implementation as bypassing anti-cheat. The app does not
|
||||
|
||||
@@ -142,12 +142,13 @@ the legacy scorer or emit legacy builds/recommendations.
|
||||
1. The saved 20/50/100/2,211 validation gates passed for this checkpoint;
|
||||
preserve them for later Build-Fit changes and do not weaken scanner review
|
||||
gates.
|
||||
2. Perform a packaged renderer/UI acceptance before calling the new Builds
|
||||
surface live-accepted.
|
||||
2. The packaged renderer/UI acceptance passed. Keep it distinct from the
|
||||
optional elevated packaged-executable runtime/UAC claim.
|
||||
3. Refresh every profile before its explicit expiry and add Golden plus
|
||||
Review/low-confidence adversarial coverage for every new profile.
|
||||
4. Visually label the remaining 136 Review crops before changing parser or
|
||||
profile rules; do not relax gates from aggregate statistics.
|
||||
4. The historical 136 Review crops were visually audited. For all new Review
|
||||
cases, require visual confirmation before changing parser or profile rules;
|
||||
do not relax gates from aggregate statistics.
|
||||
5. Add later profiles only when their team/weapon/rotation assumptions can be
|
||||
represented explicitly without hidden constants. No suggestion may imply
|
||||
deletion, feeding, enhancement, or automatic in-game mutation.
|
||||
|
||||
+18
-6
@@ -45,6 +45,10 @@ The fresh current-build smoke is documented in
|
||||
the current corpus work: 128 are explicit 1–4★ entries, six have the same
|
||||
exact duplicated-decimal OCR defect, one requires slot-safe name resolution,
|
||||
and one incomplete crop intentionally remains `Review`.
|
||||
- The review-corpus exporter now exposes only safe capture/run references and
|
||||
prioritizes candidates with a retrievable local crop. The confirmation
|
||||
preparer refuses OCR-only records, so a parser guess cannot become ground
|
||||
truth without retained visual evidence.
|
||||
|
||||
## Verified Evidence
|
||||
|
||||
@@ -74,6 +78,11 @@ does not certify the current change set by itself:
|
||||
`native-scanner-run-timing-v1` record. Request-to-reconciliation was
|
||||
`28,022 ms` (`~1.78 Artifacts/s`); that is a functional/safety result, not a
|
||||
3/s performance claim.
|
||||
- The follow-on review-evidence workflow passed focused exporter/preparer
|
||||
checks plus `npm run lint`, `npm test` (72 files / 446 tests), and
|
||||
`npm run build`. A read-only export of the current local queue found 177
|
||||
records and three retrievable crops; all three are already-confirmed
|
||||
`native-review-approved` cases, so no new label was promoted.
|
||||
|
||||
## What Still Needs Testing
|
||||
|
||||
@@ -109,9 +118,12 @@ does not certify the current change set by itself:
|
||||
|
||||
1. Keep the passing built-renderer acceptance distinct from the optional
|
||||
elevated packaged-executable runtime/UAC check.
|
||||
2. Clean generated workspace outputs, commit the coherent checkpoint, and push
|
||||
the intended branch; the current-build 50-artifact live gate has passed.
|
||||
3. Continue the next quality phase with visually confirmed corpus growth,
|
||||
fail-closed parser behavior, profile-source refreshes, and packaged UI
|
||||
regression coverage. Do not broaden game-category scope or add in-game
|
||||
mutation.
|
||||
2. The generated workspace outputs were cleaned and the coherent checkpoint
|
||||
was committed and pushed as `8b9f948` on 2026-07-11. The active admin
|
||||
runtime's locked log was intentionally left untouched.
|
||||
3. Re-export future Review cases and only label candidates marked with
|
||||
retrievable visual evidence. No currently retrievable local candidate needs
|
||||
a new corpus label.
|
||||
4. Continue the next quality phase with fail-closed parser behavior,
|
||||
profile-source refreshes, and packaged UI regression coverage. Do not
|
||||
broaden game-category scope or add in-game mutation.
|
||||
|
||||
@@ -102,13 +102,15 @@ not a guessed canonical artifact.
|
||||
`run-timing.json`. It measured `28,022 ms` request-to-reconciliation, so it
|
||||
does not claim the separate 3/s performance target.
|
||||
|
||||
3. Remove generated workspace outputs, commit the coherent checkpoint, and
|
||||
push the intended branch.
|
||||
3. Completed on 2026-07-11: generated workspace outputs were removed and the
|
||||
coherent checkpoint was committed and pushed as `8b9f948`. The running
|
||||
admin runtime's locked log was intentionally preserved.
|
||||
|
||||
## Follow-on Quality Phase
|
||||
|
||||
- Extend the corpus only from visually confirmed crops and preserve ambiguity as
|
||||
`Review`.
|
||||
- Extend the corpus only from new visually confirmed crops and preserve
|
||||
ambiguity as `Review`. The historical 136-item Review population has already
|
||||
been audited; it is not an outstanding relabeling task.
|
||||
- Refresh Build-Fit source profiles before their explicit expiry and add Golden
|
||||
plus adversarial fixtures for each profile expansion.
|
||||
- Keep the bounded wheel reset as the production baseline; test any alternate
|
||||
|
||||
@@ -217,11 +217,14 @@ Nachweis.
|
||||
- Eine echte Review-Korrektur oder Promotion wurde nicht absichtlich in den
|
||||
Store geschrieben. Die statischen und Main-Process-Gates bestehen; der
|
||||
Mutation-Pfad bleibt ein separater, explizit bestaetigter QA-Fall.
|
||||
- Die verbliebenen 136 Review-Faelle des kompletten Corpus sind der naechste
|
||||
Scanner-Qualitaetsbestand: 128 tragen `substats_implausible`, 65
|
||||
`substats_incomplete`, 28 eine niedrige Main-Value-Konfidenz und 4 einen
|
||||
IK-Mismatch; Gruende koennen sich ueberlappen. Sie werden nicht automatisch
|
||||
als sauber umetikettiert.
|
||||
- Die damals verbliebenen 136 Review-Faelle des kompletten Corpus waren der
|
||||
naechste Scanner-Qualitaetsbestand. Sie wurden am 2026-07-11 visuell gegen
|
||||
die gespeicherten Crops auditiert: 128 sind explizite 1–4★-Ausschluesse,
|
||||
sechs belegen die exakte doppelte Dezimalpunkt-Korrektur, ein Fall erhielt
|
||||
eine slot-sichere Namensaufloesung und ein unvollstaendiger Crop bleibt
|
||||
bewusst `Review`. Details stehen in
|
||||
`LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md`; neue Fälle
|
||||
werden weiterhin nicht automatisch als sauber umetikettiert.
|
||||
|
||||
Die score-freie Grundlage von `build-fit-contract-v1` ist inzwischen
|
||||
implementiert: gesourcte Eingaben, OCR-/Review-Gates, Set-/Mainstat-/Substat-
|
||||
|
||||
@@ -36,8 +36,8 @@ reported separately; they are not added together.
|
||||
|
||||
## Follow-Up
|
||||
|
||||
The required scanner release gate is complete. The next approved actions are
|
||||
workspace cleanup, a coherent repository commit, and push of the current
|
||||
checkpoint. The remaining quality work is visually confirmed corpus growth,
|
||||
source-profile refresh, and a separate elevated packaged-runtime check only if
|
||||
that specific binary-runtime claim is needed.
|
||||
The required scanner release gate and its checkpoint publication are complete:
|
||||
the workspace was cleaned and commit `8b9f948` was pushed on 2026-07-11. The
|
||||
remaining quality work is new visually confirmed corpus growth, source-profile
|
||||
refresh, and a separate elevated packaged-runtime check only if that specific
|
||||
binary-runtime claim is needed.
|
||||
|
||||
+16
-8
@@ -282,7 +282,10 @@ confirmation probes. That is not an elevated packaged-executable claim. The
|
||||
current elevated development 50-artifact native smoke now passes with complete
|
||||
timing evidence and zero writes; its `28,022 ms` end-to-end result does not
|
||||
prove the separate 3/s target. The remaining Review corpus continues to grow
|
||||
only from visually confirmed labels.
|
||||
only from visually confirmed labels. The review exporter now prioritizes
|
||||
retrievable retained crops and serializes only safe capture/run references; its
|
||||
confirmation preparer refuses OCR-only candidates so parser guesses cannot be
|
||||
promoted without visual evidence.
|
||||
|
||||
## Product Direction
|
||||
|
||||
@@ -531,18 +534,23 @@ Status:
|
||||
reconciliation, zero Review/errors/writes, and
|
||||
`requestToResultsReconciledMs=28,022`. This is a functional/safety gate;
|
||||
it does not claim 3 Artifacts/s.
|
||||
4. Clean generated workspace output, commit the coherent checkpoint, and push
|
||||
the intended branch. A package-runtime/UAC test is required only for that
|
||||
separate claim.
|
||||
5. Continue review-corpus growth only from visually confirmed crops. Keep
|
||||
4. Completed on 2026-07-11: generated workspace output was cleaned and the
|
||||
coherent checkpoint was committed and pushed as `8b9f948`. A
|
||||
package-runtime/UAC test is required only for that separate claim.
|
||||
5. The visual-evidence review exporter/preparer passed focused checks plus
|
||||
`npm run lint`, `npm test` (72 files / 446 tests), and `npm run build`.
|
||||
Its read-only export of the current 177-record local queue found three
|
||||
retrievable crops, all already-confirmed `native-review-approved` cases;
|
||||
no new corpus label was promoted.
|
||||
6. Continue review-corpus growth only from new visually confirmed crops. Keep
|
||||
explicit 1–4★ evidence excluded, ambiguous evidence in `Review`, and reject
|
||||
aliases that cannot be justified by independent field evidence.
|
||||
6. Refresh source-bound profiles before expiry and add Golden plus
|
||||
7. Refresh source-bound profiles before expiry and add Golden plus
|
||||
Review/low-confidence adversarial fixtures for each additional profile.
|
||||
7. Keep the bounded wheel reset as the production baseline. Benchmark a `B`
|
||||
8. Keep the bounded wheel reset as the production baseline. Benchmark a `B`
|
||||
plus Artifact-tab reopen only as a separate guarded experiment before any
|
||||
reset strategy change.
|
||||
8. Keep Build-Fit profile suggestions read-only and Artifact-only until an
|
||||
9. Keep Build-Fit profile suggestions read-only and Artifact-only until an
|
||||
explicit product decision adds owned-character, weapon, team, and rotation
|
||||
inputs with their own sources.
|
||||
|
||||
|
||||
+28
-14
@@ -52,12 +52,21 @@ This writes:
|
||||
- `outputs/review-eval-candidates/review-eval-candidates.json`
|
||||
- `outputs/review-eval-candidates/review-eval-candidates.md`
|
||||
|
||||
The exporter deduplicates samples, puts complete modern OCR captures first,
|
||||
marks missing fast-profile fields so stale/partial captures do not crowd out
|
||||
useful cases, and surfaces ownership/lock evidence (`artifact-footer`,
|
||||
`equipped`, and `locked=true/false`) for the next validation pass.
|
||||
The exporter deduplicates samples, puts candidates with retrievable local visual
|
||||
evidence first, then preserves the existing complete-OCR/staleness ordering
|
||||
within each evidence group. It marks missing fast-profile fields so
|
||||
stale/partial captures do not crowd out useful cases, and surfaces
|
||||
ownership/lock evidence (`artifact-footer`, `equipped`, and `locked=true/false`)
|
||||
for the next validation pass.
|
||||
|
||||
Current local snapshot from 2026-07-10:
|
||||
Each candidate records a safe `captureId`, `capturedAt`, and
|
||||
`visualEvidence.status`. `available` means the referenced local PNG existed at
|
||||
export time; `unavailable` is an OCR-only legacy/no-path case. The export never
|
||||
serializes the local PNG path. Only label an `available` candidate after opening
|
||||
that retained crop and checking the real artifact. Re-export if the crop was
|
||||
removed; the preparer deliberately refuses `unavailable` candidates.
|
||||
|
||||
Current local snapshot re-exported on 2026-07-11:
|
||||
|
||||
- 177 records read, 0 invalid
|
||||
- 138 unique candidates
|
||||
@@ -65,16 +74,18 @@ Current local snapshot from 2026-07-10:
|
||||
- 36 candidates with complete fast-profile fields
|
||||
- 41 likely stale or partial captures
|
||||
- 14 equipped-footer candidates
|
||||
- 3 `native-review-approved` cases already represented in
|
||||
`confirmedReviewCorpus.ts`; the remaining 77 exported cases still require
|
||||
explicit human labels
|
||||
- 3 candidates with retrievable visual evidence, all
|
||||
`native-review-approved` cases already represented in
|
||||
`confirmedReviewCorpus.ts`
|
||||
- 77 OCR-only/unavailable exported cases, which must not be prepared as
|
||||
confirmed corpus labels until a retained crop is available
|
||||
|
||||
These counts describe the current local queue and may grow after later live
|
||||
sessions. Do not treat the 36 complete-field candidates as automatically
|
||||
correct; complete OCR is still only a review candidate until visually checked.
|
||||
|
||||
After manually checking one candidate against the real artifact, create a
|
||||
confirmed corpus snippet with explicit expected labels:
|
||||
After manually checking one `visualEvidence: available` candidate against the
|
||||
real artifact, create a confirmed corpus snippet with explicit expected labels:
|
||||
|
||||
```powershell
|
||||
npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\path\to\expect.json
|
||||
@@ -83,15 +94,18 @@ npm run eval:prepare-confirmed -- --candidate=<candidate-id> --expect-file=.\pat
|
||||
The script reads the latest
|
||||
`outputs/review-eval-candidates/review-eval-candidates.json` by default and
|
||||
writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`.
|
||||
It refuses to run without explicit labels, so parser guesses are not silently
|
||||
promoted to ground truth. Review that snippet, then paste the object into
|
||||
It refuses to run without explicit labels or retrievable visual evidence, so
|
||||
parser guesses and OCR-only legacy records are not silently promoted to ground
|
||||
truth. The generated provenance note includes the safe capture/run reference
|
||||
and timestamp, never an absolute local crop path. Review that snippet, then paste the object into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`.
|
||||
|
||||
The parser's guess is a label **candidate, not ground truth** (using it directly
|
||||
would be the parser grading itself). To add a real case:
|
||||
|
||||
1. Convert review samples with `reviewSamplesToEvalCases(records)`.
|
||||
2. Open each produced case and confirm or correct the `expect` values against
|
||||
1. Export review samples and choose a candidate marked
|
||||
`visualEvidence: available`.
|
||||
2. Open its retained crop and confirm or correct the `expect` values against
|
||||
what the artifact actually is in-game. Set `confirmed: true`.
|
||||
3. Move the corrected case into
|
||||
`src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports
|
||||
|
||||
@@ -41,11 +41,11 @@ checkpoint, see
|
||||
|
||||
The implementation now has an English-default/Deutsch-selectable renderer,
|
||||
confirmed local artifact removal, a shared-clock native run timing record, and
|
||||
new visually reviewed OCR corpus cases. These are implementation and offline
|
||||
validation results, not a new live-release claim. The remaining release gates
|
||||
include the non-persisting 50-artifact native smoke that produces a complete
|
||||
`run-timing.json`. The built-renderer Builds/UI acceptance already passed; an
|
||||
elevated packaged-executable runtime remains a separate packaging boundary.
|
||||
new visually reviewed OCR corpus cases. The required non-persisting current
|
||||
build 50-artifact native smoke passed with a complete `run-timing.json`, and
|
||||
the resulting checkpoint was published as `8b9f948`. The built-renderer
|
||||
Builds/UI acceptance also passed; an elevated packaged-executable runtime
|
||||
remains a separate packaging boundary.
|
||||
|
||||
## Target User Flow
|
||||
|
||||
|
||||
@@ -108,6 +108,12 @@ Detailed evidence:
|
||||
akzeptiert. 360 widerspruechliche `+0`-Main-Values werden jetzt nur dann aus
|
||||
der kanonischen 5-Sterne-Tabelle repariert, wenn vier Anfangs-Substats die
|
||||
Seltenheit strukturell beweisen; Drei-Substat-Faelle bleiben Review.
|
||||
- Der Review-Corpus-Exporter priorisiert jetzt nur noch Kandidaten mit
|
||||
abrufbarem lokalem Crop und gibt sichere Capture-/Run-Referenzen statt lokaler
|
||||
Pfade aus. Der Snippet-Preparer blockiert OCR-only Kandidaten ohne visuelle
|
||||
Evidenz. Ein read-only Export der aktuellen 177 Eintraege fand drei abrufbare,
|
||||
bereits bestaetigte `native-review-approved` Faelle; keine neue Label-Promotion
|
||||
wurde vorgenommen.
|
||||
- Native Helper und PowerShell-Fallback akzeptieren nur noch die exakten
|
||||
Prozessnamen `GenshinImpact` oder `YuanShen`. Bei geschlossenem Spiel bestand
|
||||
der Package-Smoke mit `genshinFound=false`, HWND 0 und leerem Zielprozess.
|
||||
@@ -153,8 +159,9 @@ Detailed evidence:
|
||||
offen; der frische native 50er-Smoke bestand als Scanner-Release-Gate.
|
||||
2. Profilquellen vor Ablauf erneuern und jedes weitere Profil wieder mit
|
||||
Golden-/adversarial Fixtures absichern.
|
||||
3. Die verbleibenden 136 Review-Faelle nur nach visueller Crop-Bestaetigung in
|
||||
Parser-Corpus oder Lernregeln uebernehmen.
|
||||
3. Neue Review-Faelle nur nach visueller Crop-Bestaetigung in Parser-Corpus
|
||||
oder Lernregeln uebernehmen. Die historische 136er-Population ist bereits
|
||||
auditiert; der eine unvollstaendige Crop bleibt bewusst `Review`.
|
||||
4. Result-Klarheit und explizite Mutation-QA weiter staerken.
|
||||
5. Erst nach eigenen Quellen fuer Charakterbesitz, Team, Waffe und Rotation
|
||||
eine breitere Optimizer- oder Empfehlungsphase erwägen.
|
||||
@@ -162,7 +169,7 @@ Detailed evidence:
|
||||
## Aktueller Nachweis
|
||||
|
||||
- `npm run acceptance:offline`: PASS
|
||||
- `npm test`: 444/444 in 72 Dateien
|
||||
- `npm test`: 446/446 in 72 Dateien
|
||||
- `npm run acceptance:offline`: final PASS nach Lokalisierung, lokaler
|
||||
Loeschung, Timing, Korpus und UI-Acceptance-Runner; Package-Verifier 27/27
|
||||
- Build-Fit-Ranker/Kontext/Profile: offline PASS; gebaute Renderer-UI PASS,
|
||||
|
||||
@@ -23,6 +23,43 @@ function simplifyId(value) {
|
||||
.slice(0, 80) || "unknown";
|
||||
}
|
||||
|
||||
function safeCaptureReference(value) {
|
||||
const reference = typeof value === "string" ? value.trim() : "";
|
||||
if (!reference || reference.length > 160 || /[\r\n]/.test(reference)) return null;
|
||||
if (path.isAbsolute(reference) || /^[A-Za-z]:[\\/]/.test(reference) || reference.startsWith("\\\\")) return null;
|
||||
return reference;
|
||||
}
|
||||
|
||||
function safeCapturedAt(value) {
|
||||
const capturedAt = typeof value === "string" ? value.trim() : "";
|
||||
if (!capturedAt || capturedAt.length > 80 || /[\r\n]/.test(capturedAt)) return null;
|
||||
return capturedAt;
|
||||
}
|
||||
|
||||
function hasRetrievableLocalPng(value) {
|
||||
if (typeof value !== "string" || path.extname(value).toLowerCase() !== ".png") return false;
|
||||
try {
|
||||
return fs.statSync(value).isFile();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function visualEvidenceForCapture(capture) {
|
||||
const captureId = safeCaptureReference(capture?.id);
|
||||
const capturedAt = safeCapturedAt(capture?.capturedAt);
|
||||
const available = hasRetrievableLocalPng(capture?.name);
|
||||
return {
|
||||
captureId,
|
||||
capturedAt,
|
||||
visualEvidence: {
|
||||
status: available ? "available" : "unavailable",
|
||||
source: available ? "local-crop" : "none",
|
||||
reference: captureId,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function parsedSummary(parsed) {
|
||||
if (!parsed || typeof parsed !== "object") return {};
|
||||
return {
|
||||
@@ -66,6 +103,7 @@ function candidateFromRecord(record, index) {
|
||||
const reason = record?.sample?.reason || "missing-reason";
|
||||
const savedAt = record?.savedAt || "unknown";
|
||||
const capture = record?.sample?.capture || {};
|
||||
const visualEvidence = visualEvidenceForCapture(capture);
|
||||
const ocrFieldIds = Object.keys(ocr).sort();
|
||||
const missingFastFields = REQUIRED_FAST_OCR_FIELDS.filter((field) => !ocr[field]);
|
||||
const hasEquippedFooterOcr = Boolean(ocr["artifact-footer"]);
|
||||
@@ -76,6 +114,9 @@ function candidateFromRecord(record, index) {
|
||||
reason,
|
||||
confirmed: false,
|
||||
resolution: capture.width && capture.height ? `${capture.width}x${capture.height}` : "",
|
||||
captureId: visualEvidence.captureId,
|
||||
capturedAt: visualEvidence.capturedAt,
|
||||
visualEvidence: visualEvidence.visualEvidence,
|
||||
ocrFieldIds,
|
||||
missingFastFields,
|
||||
hasEquippedFooterOcr,
|
||||
@@ -98,6 +139,9 @@ function candidateFromRecord(record, index) {
|
||||
validationChecks: {
|
||||
equipped: hasEquippedFooterOcr ? "Confirm character name or mark Not detected." : "No footer OCR in this sample.",
|
||||
locked: locked === null ? "No lock-state payload in this sample." : `Confirm locked=${locked}.`,
|
||||
visualEvidence: visualEvidence.visualEvidence.status === "available"
|
||||
? "A local PNG was retrievable at export. Inspect it before confirming labels."
|
||||
: "No retrievable local PNG. Do not promote this case to the confirmed corpus.",
|
||||
},
|
||||
},
|
||||
};
|
||||
@@ -128,9 +172,19 @@ function candidatePriority(candidate) {
|
||||
return 5;
|
||||
}
|
||||
|
||||
function candidateComparator(left, right) {
|
||||
const visualEvidencePriority = Number(left.visualEvidence?.status !== "available")
|
||||
- Number(right.visualEvidence?.status !== "available");
|
||||
if (visualEvidencePriority) return visualEvidencePriority;
|
||||
const priority = candidatePriority(left) - candidatePriority(right);
|
||||
if (priority) return priority;
|
||||
const missingDiff = (left.missingFastFields?.length || 0) - (right.missingFastFields?.length || 0);
|
||||
if (missingDiff) return missingDiff;
|
||||
return String(right.savedAt).localeCompare(String(left.savedAt));
|
||||
}
|
||||
|
||||
async function readCandidates(inputPath, limit) {
|
||||
const candidates = [];
|
||||
const seen = new Set();
|
||||
const candidatesByKey = new Map();
|
||||
let total = 0;
|
||||
let invalid = 0;
|
||||
const rl = readline.createInterface({ input: fs.createReadStream(inputPath, { encoding: "utf8" }) });
|
||||
@@ -147,16 +201,14 @@ async function readCandidates(inputPath, limit) {
|
||||
const candidate = candidateFromRecord(record, total);
|
||||
if (!candidate) continue;
|
||||
const key = candidateKey(candidate);
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
candidates.push(candidate);
|
||||
const existing = candidatesByKey.get(key);
|
||||
if (!existing || candidateComparator(candidate, existing) < 0) {
|
||||
candidatesByKey.set(key, candidate);
|
||||
}
|
||||
}
|
||||
const candidates = [...candidatesByKey.values()];
|
||||
candidates.sort((left, right) => {
|
||||
const priority = candidatePriority(left) - candidatePriority(right);
|
||||
if (priority) return priority;
|
||||
const missingDiff = (left.missingFastFields?.length || 0) - (right.missingFastFields?.length || 0);
|
||||
if (missingDiff) return missingDiff;
|
||||
return String(right.savedAt).localeCompare(String(left.savedAt));
|
||||
return candidateComparator(left, right);
|
||||
});
|
||||
return { total, invalid, candidates: candidates.slice(0, limit), uniqueCandidates: candidates.length };
|
||||
}
|
||||
@@ -174,12 +226,16 @@ function buildExportStats(candidates) {
|
||||
let equippedFooterCandidates = 0;
|
||||
let lockedTrueCandidates = 0;
|
||||
let lockedFalseCandidates = 0;
|
||||
let visualEvidenceAvailable = 0;
|
||||
let visualEvidenceUnavailable = 0;
|
||||
for (const candidate of candidates) {
|
||||
increment(reasonCounts, candidate.reason);
|
||||
if (candidate.likelyStaleCapture) likelyStaleCaptures++;
|
||||
if (candidate.hasEquippedFooterOcr) equippedFooterCandidates++;
|
||||
if (candidate.locked === true) lockedTrueCandidates++;
|
||||
if (candidate.locked === false) lockedFalseCandidates++;
|
||||
if (candidate.visualEvidence?.status === "available") visualEvidenceAvailable++;
|
||||
else visualEvidenceUnavailable++;
|
||||
const missingFields = candidate.missingFastFields || [];
|
||||
if (missingFields.length === 0) completeFastFields++;
|
||||
for (const field of missingFields) increment(missingFastFieldCounts, field);
|
||||
@@ -190,6 +246,8 @@ function buildExportStats(candidates) {
|
||||
equippedFooterCandidates,
|
||||
lockedTrueCandidates,
|
||||
lockedFalseCandidates,
|
||||
visualEvidenceAvailable,
|
||||
visualEvidenceUnavailable,
|
||||
reasonCounts,
|
||||
missingFastFieldCounts,
|
||||
};
|
||||
@@ -209,8 +267,10 @@ function markdownFor(summary, candidates) {
|
||||
`Equipped footer candidates: ${summary.exportStats.equippedFooterCandidates}`,
|
||||
`Locked=true candidates: ${summary.exportStats.lockedTrueCandidates}`,
|
||||
`Locked=false candidates: ${summary.exportStats.lockedFalseCandidates}`,
|
||||
`Visual evidence available: ${summary.exportStats.visualEvidenceAvailable}`,
|
||||
`Visual evidence unavailable: ${summary.exportStats.visualEvidenceUnavailable}`,
|
||||
"",
|
||||
"These cases are not ground truth yet. Confirm or correct the expected fields before committing any case into `src/eval/corpus/`.",
|
||||
"These cases are not ground truth yet. Only manually label candidates with `visualEvidence: available`; then confirm or correct the expected fields before committing any case into `src/eval/corpus/`.",
|
||||
"",
|
||||
"## Export stats",
|
||||
"",
|
||||
@@ -228,6 +288,9 @@ function markdownFor(summary, candidates) {
|
||||
lines.push("");
|
||||
lines.push(`- savedAt: ${candidate.savedAt}`);
|
||||
lines.push(`- reason: ${candidate.reason}`);
|
||||
lines.push(`- captureId: ${candidate.captureId || "unavailable"}`);
|
||||
lines.push(`- capturedAt: ${candidate.capturedAt || "unavailable"}`);
|
||||
lines.push(`- visualEvidence: ${candidate.visualEvidence?.status || "unavailable"} (${candidate.visualEvidence?.source || "none"})`);
|
||||
lines.push(`- missingFastFields: ${candidate.missingFastFields.join(", ") || "none"}`);
|
||||
lines.push(`- likelyStaleCapture: ${candidate.likelyStaleCapture ? "yes" : "no"}`);
|
||||
lines.push(`- hasEquippedFooterOcr: ${candidate.hasEquippedFooterOcr ? "yes" : "no"}`);
|
||||
@@ -281,8 +344,11 @@ if (require.main === module) {
|
||||
module.exports = {
|
||||
buildExportStats,
|
||||
candidateFromRecord,
|
||||
candidateComparator,
|
||||
candidatePriority,
|
||||
defaultReviewSamplesPath,
|
||||
hasRetrievableLocalPng,
|
||||
markdownFor,
|
||||
readCandidates,
|
||||
visualEvidenceForCapture,
|
||||
};
|
||||
|
||||
@@ -60,12 +60,33 @@ function stableId(value) {
|
||||
.slice(0, 80) || "review-case";
|
||||
}
|
||||
|
||||
function safeProvenanceValue(value, fallback = "unknown") {
|
||||
const text = typeof value === "string" ? value.trim() : "";
|
||||
if (!text || text.length > 200 || /[\r\n]/.test(text)) return fallback;
|
||||
if (path.isAbsolute(text) || /[A-Za-z]:[\\/]/.test(text) || text.startsWith("\\\\")) return fallback;
|
||||
return text;
|
||||
}
|
||||
|
||||
function visualEvidenceNote(candidate) {
|
||||
const status = candidate?.visualEvidence?.status === "available" ? "available" : "unavailable";
|
||||
const parts = [`visualEvidence=${status}`];
|
||||
const captureId = safeProvenanceValue(candidate?.captureId, "");
|
||||
const capturedAt = safeProvenanceValue(candidate?.capturedAt, "");
|
||||
if (captureId) parts.push(`captureId=${captureId}`);
|
||||
if (capturedAt) parts.push(`capturedAt=${capturedAt}`);
|
||||
return parts.join(" | ");
|
||||
}
|
||||
|
||||
function objectLiteral(value, indent = 2) {
|
||||
return JSON.stringify(value, null, indent).replace(/"([A-Za-z_$][0-9A-Za-z_$]*)":/g, "$1:");
|
||||
}
|
||||
|
||||
function snippetFor(candidate, expect) {
|
||||
const id = stableId(argValue("id", `confirmed-${candidate.id}`));
|
||||
const reason = safeProvenanceValue(candidate.reason, "review-sample");
|
||||
const savedAt = safeProvenanceValue(candidate.savedAt, "unknown");
|
||||
const resolution = safeProvenanceValue(candidate.resolution, "");
|
||||
const sourceCandidate = stableId(candidate.id);
|
||||
const entry = {
|
||||
id,
|
||||
confirmed: true,
|
||||
@@ -73,8 +94,8 @@ function snippetFor(candidate, expect) {
|
||||
expect,
|
||||
meta: {
|
||||
source: "review-sample",
|
||||
resolution: candidate.resolution || undefined,
|
||||
note: `${candidate.reason || "review-sample"} | savedAt=${candidate.savedAt || "unknown"} | sourceCandidate=${candidate.id}`,
|
||||
resolution: resolution || undefined,
|
||||
note: `${reason} | savedAt=${savedAt} | sourceCandidate=${sourceCandidate} | ${visualEvidenceNote(candidate)}`,
|
||||
},
|
||||
};
|
||||
return `${objectLiteral(entry, 2)},\n`;
|
||||
@@ -90,6 +111,9 @@ function main() {
|
||||
if (!candidateId) throw new Error("Missing candidate id. Pass --candidate=<id>.");
|
||||
const outputPath = path.resolve(argValue("out", path.join(process.cwd(), "outputs", "review-eval-candidates", `${stableId(candidateId)}.confirmed.ts`)));
|
||||
const candidate = loadCandidate(inputPath, candidateId);
|
||||
if (candidate?.visualEvidence?.status !== "available") {
|
||||
throw new Error("Candidate has no retrievable visual evidence. Re-export after retaining a local PNG before preparing a confirmed corpus case.");
|
||||
}
|
||||
const expect = parseExpectedFields();
|
||||
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
|
||||
const snippet = snippetFor(candidate, expect);
|
||||
@@ -106,4 +130,5 @@ module.exports = {
|
||||
parseExpectedFields,
|
||||
snippetFor,
|
||||
stableId,
|
||||
visualEvidenceNote,
|
||||
};
|
||||
|
||||
@@ -4,8 +4,9 @@ import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
function writeCandidatePayload(dir: string) {
|
||||
function writeCandidatePayload(dir: string, visualEvidenceStatus: "available" | "unavailable" = "available") {
|
||||
const inputPath = path.join(dir, "review-eval-candidates.json");
|
||||
const localCropPath = path.join(dir, "private-artifact-crop.png");
|
||||
writeFileSync(
|
||||
inputPath,
|
||||
JSON.stringify({
|
||||
@@ -15,7 +16,16 @@ function writeCandidatePayload(dir: string) {
|
||||
id: "review-2026-07-08T15-02-39-765Z-152",
|
||||
savedAt: "2026-07-08T15:02:39.765Z",
|
||||
reason: "automatic:parser-notes:p1:r2c3",
|
||||
resolution: "1920x1080",
|
||||
resolution: localCropPath,
|
||||
captureId: "20260711-154827:10",
|
||||
capturedAt: "2026-07-11T15:48:28.000Z",
|
||||
capturePath: localCropPath,
|
||||
visualEvidence: {
|
||||
status: visualEvidenceStatus,
|
||||
source: visualEvidenceStatus === "available" ? "local-crop" : "none",
|
||||
reference: "20260711-154827:10",
|
||||
localPath: localCropPath,
|
||||
},
|
||||
ocr: {
|
||||
"artifact-name": "Pristine Plume of the Blessed",
|
||||
"artifact-slot": "Plume of Death",
|
||||
@@ -67,6 +77,10 @@ describe("prepare confirmed review case script", () => {
|
||||
expect(snippet).toContain("Pristine Plume of the Blessed");
|
||||
expect(snippet).toContain('"artifact-name": "Pristine Plume of the Blessed"');
|
||||
expect(snippet).toContain("sourceCandidate=review-2026-07-08T15-02-39-765Z-152");
|
||||
expect(snippet).toContain("visualEvidence=available");
|
||||
expect(snippet).toContain("captureId=20260711-154827:10");
|
||||
expect(snippet).toContain("capturedAt=2026-07-11T15:48:28.000Z");
|
||||
expect(snippet).not.toContain(path.join(dir, "private-artifact-crop.png"));
|
||||
expect(snippet).not.toContain("confirmed: false");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
@@ -88,4 +102,31 @@ describe("prepare confirmed review case script", () => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects candidates without retrievable visual evidence", () => {
|
||||
const dir = mkdtempSync(path.join(tmpdir(), "gaa-confirmed-review-"));
|
||||
try {
|
||||
const inputPath = writeCandidatePayload(dir, "unavailable");
|
||||
let thrown: unknown;
|
||||
try {
|
||||
execFileSync(
|
||||
"node",
|
||||
[
|
||||
"scripts/prepare-confirmed-review-case.cjs",
|
||||
`--input=${inputPath}`,
|
||||
"--candidate=review-2026-07-08T15-02-39-765Z-152",
|
||||
'--expect-json={"name":"Pristine Plume of the Blessed"}',
|
||||
],
|
||||
{ cwd: process.cwd(), stdio: "pipe" },
|
||||
);
|
||||
} catch (error) {
|
||||
thrown = error;
|
||||
}
|
||||
expect(thrown).toBeDefined();
|
||||
const stderr = (thrown as { stderr?: Buffer | string }).stderr;
|
||||
expect(String(stderr)).toContain("retrievable visual evidence");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -4,12 +4,19 @@ import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
function reviewRecord(savedAt: string, reason: string, ocr: Array<{ id: string; text: string }>, locked?: boolean) {
|
||||
function reviewRecord(
|
||||
savedAt: string,
|
||||
reason: string,
|
||||
ocr: Array<{ id: string; text: string }>,
|
||||
locked?: boolean,
|
||||
captureDetails: Record<string, unknown> = {},
|
||||
) {
|
||||
return {
|
||||
savedAt,
|
||||
sample: {
|
||||
reason,
|
||||
capture: {
|
||||
...captureDetails,
|
||||
width: 1920,
|
||||
height: 1080,
|
||||
ocr,
|
||||
@@ -74,7 +81,10 @@ describe("review eval candidate exporter", () => {
|
||||
expect(payload.summary.exportStats.likelyStaleCaptures).toBe(1);
|
||||
expect(payload.summary.exportStats.equippedFooterCandidates).toBe(2);
|
||||
expect(payload.summary.exportStats.lockedTrueCandidates).toBe(1);
|
||||
expect(payload.summary.exportStats.visualEvidenceAvailable).toBe(0);
|
||||
expect(payload.summary.exportStats.visualEvidenceUnavailable).toBe(2);
|
||||
expect(payload.candidates[0].missingFastFields).toEqual([]);
|
||||
expect(payload.candidates[0].visualEvidence.status).toBe("unavailable");
|
||||
expect(payload.candidates[0].parsed.equipped).toBe("Aino");
|
||||
expect(payload.candidates[0].reviewPrompt.expectedFields.equipped).toBe("Aino");
|
||||
expect(payload.candidates[1].likelyStaleCapture).toBe(true);
|
||||
@@ -83,6 +93,63 @@ describe("review eval candidate exporter", () => {
|
||||
expect(markdown).toContain("- artifact-slot: 1");
|
||||
expect(markdown).toContain("equipped=Aino");
|
||||
expect(markdown).toContain("Locked=true candidates: 1");
|
||||
expect(markdown).toContain("Visual evidence unavailable: 2");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("prioritizes retrievable local crops and does not export their local paths", () => {
|
||||
const dir = mkdtempSync(path.join(tmpdir(), "gaa-review-evidence-"));
|
||||
try {
|
||||
const inputPath = path.join(dir, "review-samples.jsonl");
|
||||
const outDir = path.join(dir, "out");
|
||||
const cropPath = path.join(dir, "artifact-0010.png");
|
||||
writeFileSync(cropPath, "png", "utf8");
|
||||
const staleOcr = completeOcr.filter((entry) => entry.id !== "artifact-slot");
|
||||
writeFileSync(
|
||||
inputPath,
|
||||
[
|
||||
JSON.stringify(reviewRecord("2026-07-11T10:00:00.000Z", "automatic:low-field:p1:r0c0", completeOcr)),
|
||||
JSON.stringify(reviewRecord("2026-07-11T10:01:00.000Z", "automatic:low-field:p1:r0c0", completeOcr, false, {
|
||||
id: "20260711-154827:10",
|
||||
name: cropPath,
|
||||
capturedAt: "2026-07-11T10:01:01.000Z",
|
||||
})),
|
||||
JSON.stringify(reviewRecord("2026-07-11T10:02:00.000Z", "automatic:capture-rejected:p1:r0c1", staleOcr, false, {
|
||||
id: "20260711-154827:11",
|
||||
name: cropPath,
|
||||
capturedAt: "2026-07-11T10:02:01.000Z",
|
||||
})),
|
||||
JSON.stringify(reviewRecord("2026-07-11T10:03:00.000Z", "automatic:low-field:p1:r0c2", completeOcr)),
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
execFileSync("node", ["scripts/export-review-eval-candidates.cjs", `--input=${inputPath}`, `--out=${outDir}`, "--limit=10"], {
|
||||
cwd: process.cwd(),
|
||||
stdio: "pipe",
|
||||
});
|
||||
|
||||
const payload = JSON.parse(readFileSync(path.join(outDir, "review-eval-candidates.json"), "utf8"));
|
||||
const markdown = readFileSync(path.join(outDir, "review-eval-candidates.md"), "utf8");
|
||||
|
||||
expect(payload.summary.uniqueCandidates).toBe(3);
|
||||
expect(payload.summary.exportStats.visualEvidenceAvailable).toBe(2);
|
||||
expect(payload.summary.exportStats.visualEvidenceUnavailable).toBe(1);
|
||||
expect(payload.candidates.map((candidate: { captureId: string | null }) => candidate.captureId)).toEqual([
|
||||
"20260711-154827:10",
|
||||
"20260711-154827:11",
|
||||
null,
|
||||
]);
|
||||
expect(payload.candidates.map((candidate: { visualEvidence: { status: string } }) => candidate.visualEvidence.status)).toEqual([
|
||||
"available",
|
||||
"available",
|
||||
"unavailable",
|
||||
]);
|
||||
expect(JSON.stringify(payload)).not.toContain(cropPath);
|
||||
expect(markdown).not.toContain(cropPath);
|
||||
expect(markdown).toContain("Only manually label candidates with `visualEvidence: available`");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user