diff --git a/docs/AUTOMATION_LIVE_SCAN.md b/docs/AUTOMATION_LIVE_SCAN.md index 9e4f74b..7c6bed8 100644 --- a/docs/AUTOMATION_LIVE_SCAN.md +++ b/docs/AUTOMATION_LIVE_SCAN.md @@ -370,9 +370,11 @@ npm run eval:review-candidates -- --limit=80 ``` Read `outputs/review-eval-candidates/review-eval-candidates.md`. It is a review -worklist, not ground truth. Only after expected fields are confirmed or -corrected against the real artifact should a case move into -`src/eval/corpus/confirmedReviewCorpus.ts`. +worklist, not ground truth. Only choose a candidate marked +`visualEvidence: available`: it had a retrievable local PNG at export time. +The export exposes only safe capture/run references, never the local crop path. +After expected fields are confirmed or corrected against the real artifact, a +case may move into `src/eval/corpus/confirmedReviewCorpus.ts`. For a manually checked candidate, generate a paste-ready confirmed-case snippet: @@ -380,6 +382,10 @@ For a manually checked candidate, generate a paste-ready confirmed-case snippet: npm run eval:prepare-confirmed -- --candidate= --expect-file=.\path\to\expect.json ``` +The preparer refuses candidates without explicit labels or retrievable visual +evidence. Re-export after retaining a crop rather than promoting an OCR-only +legacy record. + ## Anti-Cheat And Safety Boundary Do not describe the implementation as bypassing anti-cheat. The app does not diff --git a/docs/BUILD_FIT_CONTRACT_V1.md b/docs/BUILD_FIT_CONTRACT_V1.md index 2a36142..db12ed4 100644 --- a/docs/BUILD_FIT_CONTRACT_V1.md +++ b/docs/BUILD_FIT_CONTRACT_V1.md @@ -142,12 +142,13 @@ the legacy scorer or emit legacy builds/recommendations. 1. The saved 20/50/100/2,211 validation gates passed for this checkpoint; preserve them for later Build-Fit changes and do not weaken scanner review gates. -2. Perform a packaged renderer/UI acceptance before calling the new Builds - surface live-accepted. +2. The packaged renderer/UI acceptance passed. Keep it distinct from the + optional elevated packaged-executable runtime/UAC claim. 3. Refresh every profile before its explicit expiry and add Golden plus Review/low-confidence adversarial coverage for every new profile. -4. Visually label the remaining 136 Review crops before changing parser or - profile rules; do not relax gates from aggregate statistics. +4. The historical 136 Review crops were visually audited. For all new Review + cases, require visual confirmation before changing parser or profile rules; + do not relax gates from aggregate statistics. 5. Add later profiles only when their team/weapon/rotation assumptions can be represented explicitly without hidden constants. No suggestion may imply deletion, feeding, enhancement, or automatic in-game mutation. diff --git a/docs/CURRENT_STATUS.md b/docs/CURRENT_STATUS.md index 3e9c174..5ee6f9c 100644 --- a/docs/CURRENT_STATUS.md +++ b/docs/CURRENT_STATUS.md @@ -45,6 +45,10 @@ The fresh current-build smoke is documented in the current corpus work: 128 are explicit 1–4★ entries, six have the same exact duplicated-decimal OCR defect, one requires slot-safe name resolution, and one incomplete crop intentionally remains `Review`. +- The review-corpus exporter now exposes only safe capture/run references and + prioritizes candidates with a retrievable local crop. The confirmation + preparer refuses OCR-only records, so a parser guess cannot become ground + truth without retained visual evidence. ## Verified Evidence @@ -74,6 +78,11 @@ does not certify the current change set by itself: `native-scanner-run-timing-v1` record. Request-to-reconciliation was `28,022 ms` (`~1.78 Artifacts/s`); that is a functional/safety result, not a 3/s performance claim. +- The follow-on review-evidence workflow passed focused exporter/preparer + checks plus `npm run lint`, `npm test` (72 files / 446 tests), and + `npm run build`. A read-only export of the current local queue found 177 + records and three retrievable crops; all three are already-confirmed + `native-review-approved` cases, so no new label was promoted. ## What Still Needs Testing @@ -109,9 +118,12 @@ does not certify the current change set by itself: 1. Keep the passing built-renderer acceptance distinct from the optional elevated packaged-executable runtime/UAC check. -2. Clean generated workspace outputs, commit the coherent checkpoint, and push - the intended branch; the current-build 50-artifact live gate has passed. -3. Continue the next quality phase with visually confirmed corpus growth, - fail-closed parser behavior, profile-source refreshes, and packaged UI - regression coverage. Do not broaden game-category scope or add in-game - mutation. +2. The generated workspace outputs were cleaned and the coherent checkpoint + was committed and pushed as `8b9f948` on 2026-07-11. The active admin + runtime's locked log was intentionally left untouched. +3. Re-export future Review cases and only label candidates marked with + retrievable visual evidence. No currently retrievable local candidate needs + a new corpus label. +4. Continue the next quality phase with fail-closed parser behavior, + profile-source refreshes, and packaged UI regression coverage. Do not + broaden game-category scope or add in-game mutation. diff --git a/docs/LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md b/docs/LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md index 21db17b..d6f58b0 100644 --- a/docs/LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md +++ b/docs/LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md @@ -102,13 +102,15 @@ not a guessed canonical artifact. `run-timing.json`. It measured `28,022 ms` request-to-reconciliation, so it does not claim the separate 3/s performance target. -3. Remove generated workspace outputs, commit the coherent checkpoint, and - push the intended branch. +3. Completed on 2026-07-11: generated workspace outputs were removed and the + coherent checkpoint was committed and pushed as `8b9f948`. The running + admin runtime's locked log was intentionally preserved. ## Follow-on Quality Phase -- Extend the corpus only from visually confirmed crops and preserve ambiguity as - `Review`. +- Extend the corpus only from new visually confirmed crops and preserve + ambiguity as `Review`. The historical 136-item Review population has already + been audited; it is not an outstanding relabeling task. - Refresh Build-Fit source profiles before their explicit expiry and add Golden plus adversarial fixtures for each profile expansion. - Keep the bounded wheel reset as the production baseline; test any alternate diff --git a/docs/NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md b/docs/NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md index 2371faa..bc2282e 100644 --- a/docs/NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md +++ b/docs/NATIVE_SCANNER_LIVE_ACCEPTANCE_2026-07-10.md @@ -217,11 +217,14 @@ Nachweis. - Eine echte Review-Korrektur oder Promotion wurde nicht absichtlich in den Store geschrieben. Die statischen und Main-Process-Gates bestehen; der Mutation-Pfad bleibt ein separater, explizit bestaetigter QA-Fall. -- Die verbliebenen 136 Review-Faelle des kompletten Corpus sind der naechste - Scanner-Qualitaetsbestand: 128 tragen `substats_implausible`, 65 - `substats_incomplete`, 28 eine niedrige Main-Value-Konfidenz und 4 einen - IK-Mismatch; Gruende koennen sich ueberlappen. Sie werden nicht automatisch - als sauber umetikettiert. +- Die damals verbliebenen 136 Review-Faelle des kompletten Corpus waren der + naechste Scanner-Qualitaetsbestand. Sie wurden am 2026-07-11 visuell gegen + die gespeicherten Crops auditiert: 128 sind explizite 1–4★-Ausschluesse, + sechs belegen die exakte doppelte Dezimalpunkt-Korrektur, ein Fall erhielt + eine slot-sichere Namensaufloesung und ein unvollstaendiger Crop bleibt + bewusst `Review`. Details stehen in + `LOCALIZATION_LOCAL_DELETION_TIMING_CHECKPOINT_2026-07-11.md`; neue Fälle + werden weiterhin nicht automatisch als sauber umetikettiert. Die score-freie Grundlage von `build-fit-contract-v1` ist inzwischen implementiert: gesourcte Eingaben, OCR-/Review-Gates, Set-/Mainstat-/Substat- diff --git a/docs/NATIVE_SCANNER_LIVE_SMOKE_2026-07-11.md b/docs/NATIVE_SCANNER_LIVE_SMOKE_2026-07-11.md index 74af13d..ee7ace5 100644 --- a/docs/NATIVE_SCANNER_LIVE_SMOKE_2026-07-11.md +++ b/docs/NATIVE_SCANNER_LIVE_SMOKE_2026-07-11.md @@ -36,8 +36,8 @@ reported separately; they are not added together. ## Follow-Up -The required scanner release gate is complete. The next approved actions are -workspace cleanup, a coherent repository commit, and push of the current -checkpoint. The remaining quality work is visually confirmed corpus growth, -source-profile refresh, and a separate elevated packaged-runtime check only if -that specific binary-runtime claim is needed. +The required scanner release gate and its checkpoint publication are complete: +the workspace was cleaned and commit `8b9f948` was pushed on 2026-07-11. The +remaining quality work is new visually confirmed corpus growth, source-profile +refresh, and a separate elevated packaged-runtime check only if that specific +binary-runtime claim is needed. diff --git a/docs/PROJECT.md b/docs/PROJECT.md index bd21327..e2d39f6 100644 --- a/docs/PROJECT.md +++ b/docs/PROJECT.md @@ -282,7 +282,10 @@ confirmation probes. That is not an elevated packaged-executable claim. The current elevated development 50-artifact native smoke now passes with complete timing evidence and zero writes; its `28,022 ms` end-to-end result does not prove the separate 3/s target. The remaining Review corpus continues to grow -only from visually confirmed labels. +only from visually confirmed labels. The review exporter now prioritizes +retrievable retained crops and serializes only safe capture/run references; its +confirmation preparer refuses OCR-only candidates so parser guesses cannot be +promoted without visual evidence. ## Product Direction @@ -531,18 +534,23 @@ Status: reconciliation, zero Review/errors/writes, and `requestToResultsReconciledMs=28,022`. This is a functional/safety gate; it does not claim 3 Artifacts/s. -4. Clean generated workspace output, commit the coherent checkpoint, and push - the intended branch. A package-runtime/UAC test is required only for that - separate claim. -5. Continue review-corpus growth only from visually confirmed crops. Keep +4. Completed on 2026-07-11: generated workspace output was cleaned and the + coherent checkpoint was committed and pushed as `8b9f948`. A + package-runtime/UAC test is required only for that separate claim. +5. The visual-evidence review exporter/preparer passed focused checks plus + `npm run lint`, `npm test` (72 files / 446 tests), and `npm run build`. + Its read-only export of the current 177-record local queue found three + retrievable crops, all already-confirmed `native-review-approved` cases; + no new corpus label was promoted. +6. Continue review-corpus growth only from new visually confirmed crops. Keep explicit 1–4★ evidence excluded, ambiguous evidence in `Review`, and reject aliases that cannot be justified by independent field evidence. -6. Refresh source-bound profiles before expiry and add Golden plus +7. Refresh source-bound profiles before expiry and add Golden plus Review/low-confidence adversarial fixtures for each additional profile. -7. Keep the bounded wheel reset as the production baseline. Benchmark a `B` +8. Keep the bounded wheel reset as the production baseline. Benchmark a `B` plus Artifact-tab reopen only as a separate guarded experiment before any reset strategy change. -8. Keep Build-Fit profile suggestions read-only and Artifact-only until an +9. Keep Build-Fit profile suggestions read-only and Artifact-only until an explicit product decision adds owned-character, weapon, team, and rotation inputs with their own sources. diff --git a/docs/ocr-eval.md b/docs/ocr-eval.md index d12191e..c75df2b 100644 --- a/docs/ocr-eval.md +++ b/docs/ocr-eval.md @@ -52,12 +52,21 @@ This writes: - `outputs/review-eval-candidates/review-eval-candidates.json` - `outputs/review-eval-candidates/review-eval-candidates.md` -The exporter deduplicates samples, puts complete modern OCR captures first, -marks missing fast-profile fields so stale/partial captures do not crowd out -useful cases, and surfaces ownership/lock evidence (`artifact-footer`, -`equipped`, and `locked=true/false`) for the next validation pass. +The exporter deduplicates samples, puts candidates with retrievable local visual +evidence first, then preserves the existing complete-OCR/staleness ordering +within each evidence group. It marks missing fast-profile fields so +stale/partial captures do not crowd out useful cases, and surfaces +ownership/lock evidence (`artifact-footer`, `equipped`, and `locked=true/false`) +for the next validation pass. -Current local snapshot from 2026-07-10: +Each candidate records a safe `captureId`, `capturedAt`, and +`visualEvidence.status`. `available` means the referenced local PNG existed at +export time; `unavailable` is an OCR-only legacy/no-path case. The export never +serializes the local PNG path. Only label an `available` candidate after opening +that retained crop and checking the real artifact. Re-export if the crop was +removed; the preparer deliberately refuses `unavailable` candidates. + +Current local snapshot re-exported on 2026-07-11: - 177 records read, 0 invalid - 138 unique candidates @@ -65,16 +74,18 @@ Current local snapshot from 2026-07-10: - 36 candidates with complete fast-profile fields - 41 likely stale or partial captures - 14 equipped-footer candidates -- 3 `native-review-approved` cases already represented in - `confirmedReviewCorpus.ts`; the remaining 77 exported cases still require - explicit human labels +- 3 candidates with retrievable visual evidence, all + `native-review-approved` cases already represented in + `confirmedReviewCorpus.ts` +- 77 OCR-only/unavailable exported cases, which must not be prepared as + confirmed corpus labels until a retained crop is available These counts describe the current local queue and may grow after later live sessions. Do not treat the 36 complete-field candidates as automatically correct; complete OCR is still only a review candidate until visually checked. -After manually checking one candidate against the real artifact, create a -confirmed corpus snippet with explicit expected labels: +After manually checking one `visualEvidence: available` candidate against the +real artifact, create a confirmed corpus snippet with explicit expected labels: ```powershell npm run eval:prepare-confirmed -- --candidate= --expect-file=.\path\to\expect.json @@ -83,15 +94,18 @@ npm run eval:prepare-confirmed -- --candidate= --expect-file=.\pat The script reads the latest `outputs/review-eval-candidates/review-eval-candidates.json` by default and writes a `.confirmed.ts` snippet under `outputs/review-eval-candidates/`. -It refuses to run without explicit labels, so parser guesses are not silently -promoted to ground truth. Review that snippet, then paste the object into +It refuses to run without explicit labels or retrievable visual evidence, so +parser guesses and OCR-only legacy records are not silently promoted to ground +truth. The generated provenance note includes the safe capture/run reference +and timestamp, never an absolute local crop path. Review that snippet, then paste the object into `src/eval/corpus/confirmedReviewCorpus.ts`. The parser's guess is a label **candidate, not ground truth** (using it directly would be the parser grading itself). To add a real case: -1. Convert review samples with `reviewSamplesToEvalCases(records)`. -2. Open each produced case and confirm or correct the `expect` values against +1. Export review samples and choose a candidate marked + `visualEvidence: available`. +2. Open its retained crop and confirm or correct the `expect` values against what the artifact actually is in-game. Set `confirmed: true`. 3. Move the corrected case into `src/eval/corpus/confirmedReviewCorpus.ts`. The main eval gate imports diff --git a/docs/scanner-results-inventory-roadmap.md b/docs/scanner-results-inventory-roadmap.md index d8b8f17..095c227 100644 --- a/docs/scanner-results-inventory-roadmap.md +++ b/docs/scanner-results-inventory-roadmap.md @@ -41,11 +41,11 @@ checkpoint, see The implementation now has an English-default/Deutsch-selectable renderer, confirmed local artifact removal, a shared-clock native run timing record, and -new visually reviewed OCR corpus cases. These are implementation and offline -validation results, not a new live-release claim. The remaining release gates -include the non-persisting 50-artifact native smoke that produces a complete -`run-timing.json`. The built-renderer Builds/UI acceptance already passed; an -elevated packaged-executable runtime remains a separate packaging boundary. +new visually reviewed OCR corpus cases. The required non-persisting current +build 50-artifact native smoke passed with a complete `run-timing.json`, and +the resulting checkpoint was published as `8b9f948`. The built-renderer +Builds/UI acceptance also passed; an elevated packaged-executable runtime +remains a separate packaging boundary. ## Target User Flow diff --git a/docs/scanner-rework-status.md b/docs/scanner-rework-status.md index 2659c41..b35d941 100644 --- a/docs/scanner-rework-status.md +++ b/docs/scanner-rework-status.md @@ -108,6 +108,12 @@ Detailed evidence: akzeptiert. 360 widerspruechliche `+0`-Main-Values werden jetzt nur dann aus der kanonischen 5-Sterne-Tabelle repariert, wenn vier Anfangs-Substats die Seltenheit strukturell beweisen; Drei-Substat-Faelle bleiben Review. +- Der Review-Corpus-Exporter priorisiert jetzt nur noch Kandidaten mit + abrufbarem lokalem Crop und gibt sichere Capture-/Run-Referenzen statt lokaler + Pfade aus. Der Snippet-Preparer blockiert OCR-only Kandidaten ohne visuelle + Evidenz. Ein read-only Export der aktuellen 177 Eintraege fand drei abrufbare, + bereits bestaetigte `native-review-approved` Faelle; keine neue Label-Promotion + wurde vorgenommen. - Native Helper und PowerShell-Fallback akzeptieren nur noch die exakten Prozessnamen `GenshinImpact` oder `YuanShen`. Bei geschlossenem Spiel bestand der Package-Smoke mit `genshinFound=false`, HWND 0 und leerem Zielprozess. @@ -153,8 +159,9 @@ Detailed evidence: offen; der frische native 50er-Smoke bestand als Scanner-Release-Gate. 2. Profilquellen vor Ablauf erneuern und jedes weitere Profil wieder mit Golden-/adversarial Fixtures absichern. -3. Die verbleibenden 136 Review-Faelle nur nach visueller Crop-Bestaetigung in - Parser-Corpus oder Lernregeln uebernehmen. +3. Neue Review-Faelle nur nach visueller Crop-Bestaetigung in Parser-Corpus + oder Lernregeln uebernehmen. Die historische 136er-Population ist bereits + auditiert; der eine unvollstaendige Crop bleibt bewusst `Review`. 4. Result-Klarheit und explizite Mutation-QA weiter staerken. 5. Erst nach eigenen Quellen fuer Charakterbesitz, Team, Waffe und Rotation eine breitere Optimizer- oder Empfehlungsphase erwägen. @@ -162,7 +169,7 @@ Detailed evidence: ## Aktueller Nachweis - `npm run acceptance:offline`: PASS -- `npm test`: 444/444 in 72 Dateien +- `npm test`: 446/446 in 72 Dateien - `npm run acceptance:offline`: final PASS nach Lokalisierung, lokaler Loeschung, Timing, Korpus und UI-Acceptance-Runner; Package-Verifier 27/27 - Build-Fit-Ranker/Kontext/Profile: offline PASS; gebaute Renderer-UI PASS, diff --git a/scripts/export-review-eval-candidates.cjs b/scripts/export-review-eval-candidates.cjs index e585b8f..e42fbb2 100644 --- a/scripts/export-review-eval-candidates.cjs +++ b/scripts/export-review-eval-candidates.cjs @@ -23,6 +23,43 @@ function simplifyId(value) { .slice(0, 80) || "unknown"; } +function safeCaptureReference(value) { + const reference = typeof value === "string" ? value.trim() : ""; + if (!reference || reference.length > 160 || /[\r\n]/.test(reference)) return null; + if (path.isAbsolute(reference) || /^[A-Za-z]:[\\/]/.test(reference) || reference.startsWith("\\\\")) return null; + return reference; +} + +function safeCapturedAt(value) { + const capturedAt = typeof value === "string" ? value.trim() : ""; + if (!capturedAt || capturedAt.length > 80 || /[\r\n]/.test(capturedAt)) return null; + return capturedAt; +} + +function hasRetrievableLocalPng(value) { + if (typeof value !== "string" || path.extname(value).toLowerCase() !== ".png") return false; + try { + return fs.statSync(value).isFile(); + } catch { + return false; + } +} + +function visualEvidenceForCapture(capture) { + const captureId = safeCaptureReference(capture?.id); + const capturedAt = safeCapturedAt(capture?.capturedAt); + const available = hasRetrievableLocalPng(capture?.name); + return { + captureId, + capturedAt, + visualEvidence: { + status: available ? "available" : "unavailable", + source: available ? "local-crop" : "none", + reference: captureId, + }, + }; +} + function parsedSummary(parsed) { if (!parsed || typeof parsed !== "object") return {}; return { @@ -66,6 +103,7 @@ function candidateFromRecord(record, index) { const reason = record?.sample?.reason || "missing-reason"; const savedAt = record?.savedAt || "unknown"; const capture = record?.sample?.capture || {}; + const visualEvidence = visualEvidenceForCapture(capture); const ocrFieldIds = Object.keys(ocr).sort(); const missingFastFields = REQUIRED_FAST_OCR_FIELDS.filter((field) => !ocr[field]); const hasEquippedFooterOcr = Boolean(ocr["artifact-footer"]); @@ -76,6 +114,9 @@ function candidateFromRecord(record, index) { reason, confirmed: false, resolution: capture.width && capture.height ? `${capture.width}x${capture.height}` : "", + captureId: visualEvidence.captureId, + capturedAt: visualEvidence.capturedAt, + visualEvidence: visualEvidence.visualEvidence, ocrFieldIds, missingFastFields, hasEquippedFooterOcr, @@ -98,6 +139,9 @@ function candidateFromRecord(record, index) { validationChecks: { equipped: hasEquippedFooterOcr ? "Confirm character name or mark Not detected." : "No footer OCR in this sample.", locked: locked === null ? "No lock-state payload in this sample." : `Confirm locked=${locked}.`, + visualEvidence: visualEvidence.visualEvidence.status === "available" + ? "A local PNG was retrievable at export. Inspect it before confirming labels." + : "No retrievable local PNG. Do not promote this case to the confirmed corpus.", }, }, }; @@ -128,9 +172,19 @@ function candidatePriority(candidate) { return 5; } +function candidateComparator(left, right) { + const visualEvidencePriority = Number(left.visualEvidence?.status !== "available") + - Number(right.visualEvidence?.status !== "available"); + if (visualEvidencePriority) return visualEvidencePriority; + const priority = candidatePriority(left) - candidatePriority(right); + if (priority) return priority; + const missingDiff = (left.missingFastFields?.length || 0) - (right.missingFastFields?.length || 0); + if (missingDiff) return missingDiff; + return String(right.savedAt).localeCompare(String(left.savedAt)); +} + async function readCandidates(inputPath, limit) { - const candidates = []; - const seen = new Set(); + const candidatesByKey = new Map(); let total = 0; let invalid = 0; const rl = readline.createInterface({ input: fs.createReadStream(inputPath, { encoding: "utf8" }) }); @@ -147,16 +201,14 @@ async function readCandidates(inputPath, limit) { const candidate = candidateFromRecord(record, total); if (!candidate) continue; const key = candidateKey(candidate); - if (seen.has(key)) continue; - seen.add(key); - candidates.push(candidate); + const existing = candidatesByKey.get(key); + if (!existing || candidateComparator(candidate, existing) < 0) { + candidatesByKey.set(key, candidate); + } } + const candidates = [...candidatesByKey.values()]; candidates.sort((left, right) => { - const priority = candidatePriority(left) - candidatePriority(right); - if (priority) return priority; - const missingDiff = (left.missingFastFields?.length || 0) - (right.missingFastFields?.length || 0); - if (missingDiff) return missingDiff; - return String(right.savedAt).localeCompare(String(left.savedAt)); + return candidateComparator(left, right); }); return { total, invalid, candidates: candidates.slice(0, limit), uniqueCandidates: candidates.length }; } @@ -174,12 +226,16 @@ function buildExportStats(candidates) { let equippedFooterCandidates = 0; let lockedTrueCandidates = 0; let lockedFalseCandidates = 0; + let visualEvidenceAvailable = 0; + let visualEvidenceUnavailable = 0; for (const candidate of candidates) { increment(reasonCounts, candidate.reason); if (candidate.likelyStaleCapture) likelyStaleCaptures++; if (candidate.hasEquippedFooterOcr) equippedFooterCandidates++; if (candidate.locked === true) lockedTrueCandidates++; if (candidate.locked === false) lockedFalseCandidates++; + if (candidate.visualEvidence?.status === "available") visualEvidenceAvailable++; + else visualEvidenceUnavailable++; const missingFields = candidate.missingFastFields || []; if (missingFields.length === 0) completeFastFields++; for (const field of missingFields) increment(missingFastFieldCounts, field); @@ -190,6 +246,8 @@ function buildExportStats(candidates) { equippedFooterCandidates, lockedTrueCandidates, lockedFalseCandidates, + visualEvidenceAvailable, + visualEvidenceUnavailable, reasonCounts, missingFastFieldCounts, }; @@ -209,8 +267,10 @@ function markdownFor(summary, candidates) { `Equipped footer candidates: ${summary.exportStats.equippedFooterCandidates}`, `Locked=true candidates: ${summary.exportStats.lockedTrueCandidates}`, `Locked=false candidates: ${summary.exportStats.lockedFalseCandidates}`, + `Visual evidence available: ${summary.exportStats.visualEvidenceAvailable}`, + `Visual evidence unavailable: ${summary.exportStats.visualEvidenceUnavailable}`, "", - "These cases are not ground truth yet. Confirm or correct the expected fields before committing any case into `src/eval/corpus/`.", + "These cases are not ground truth yet. Only manually label candidates with `visualEvidence: available`; then confirm or correct the expected fields before committing any case into `src/eval/corpus/`.", "", "## Export stats", "", @@ -228,6 +288,9 @@ function markdownFor(summary, candidates) { lines.push(""); lines.push(`- savedAt: ${candidate.savedAt}`); lines.push(`- reason: ${candidate.reason}`); + lines.push(`- captureId: ${candidate.captureId || "unavailable"}`); + lines.push(`- capturedAt: ${candidate.capturedAt || "unavailable"}`); + lines.push(`- visualEvidence: ${candidate.visualEvidence?.status || "unavailable"} (${candidate.visualEvidence?.source || "none"})`); lines.push(`- missingFastFields: ${candidate.missingFastFields.join(", ") || "none"}`); lines.push(`- likelyStaleCapture: ${candidate.likelyStaleCapture ? "yes" : "no"}`); lines.push(`- hasEquippedFooterOcr: ${candidate.hasEquippedFooterOcr ? "yes" : "no"}`); @@ -281,8 +344,11 @@ if (require.main === module) { module.exports = { buildExportStats, candidateFromRecord, + candidateComparator, candidatePriority, defaultReviewSamplesPath, + hasRetrievableLocalPng, markdownFor, readCandidates, + visualEvidenceForCapture, }; diff --git a/scripts/prepare-confirmed-review-case.cjs b/scripts/prepare-confirmed-review-case.cjs index b15731c..a1848e0 100644 --- a/scripts/prepare-confirmed-review-case.cjs +++ b/scripts/prepare-confirmed-review-case.cjs @@ -60,12 +60,33 @@ function stableId(value) { .slice(0, 80) || "review-case"; } +function safeProvenanceValue(value, fallback = "unknown") { + const text = typeof value === "string" ? value.trim() : ""; + if (!text || text.length > 200 || /[\r\n]/.test(text)) return fallback; + if (path.isAbsolute(text) || /[A-Za-z]:[\\/]/.test(text) || text.startsWith("\\\\")) return fallback; + return text; +} + +function visualEvidenceNote(candidate) { + const status = candidate?.visualEvidence?.status === "available" ? "available" : "unavailable"; + const parts = [`visualEvidence=${status}`]; + const captureId = safeProvenanceValue(candidate?.captureId, ""); + const capturedAt = safeProvenanceValue(candidate?.capturedAt, ""); + if (captureId) parts.push(`captureId=${captureId}`); + if (capturedAt) parts.push(`capturedAt=${capturedAt}`); + return parts.join(" | "); +} + function objectLiteral(value, indent = 2) { return JSON.stringify(value, null, indent).replace(/"([A-Za-z_$][0-9A-Za-z_$]*)":/g, "$1:"); } function snippetFor(candidate, expect) { const id = stableId(argValue("id", `confirmed-${candidate.id}`)); + const reason = safeProvenanceValue(candidate.reason, "review-sample"); + const savedAt = safeProvenanceValue(candidate.savedAt, "unknown"); + const resolution = safeProvenanceValue(candidate.resolution, ""); + const sourceCandidate = stableId(candidate.id); const entry = { id, confirmed: true, @@ -73,8 +94,8 @@ function snippetFor(candidate, expect) { expect, meta: { source: "review-sample", - resolution: candidate.resolution || undefined, - note: `${candidate.reason || "review-sample"} | savedAt=${candidate.savedAt || "unknown"} | sourceCandidate=${candidate.id}`, + resolution: resolution || undefined, + note: `${reason} | savedAt=${savedAt} | sourceCandidate=${sourceCandidate} | ${visualEvidenceNote(candidate)}`, }, }; return `${objectLiteral(entry, 2)},\n`; @@ -90,6 +111,9 @@ function main() { if (!candidateId) throw new Error("Missing candidate id. Pass --candidate=."); const outputPath = path.resolve(argValue("out", path.join(process.cwd(), "outputs", "review-eval-candidates", `${stableId(candidateId)}.confirmed.ts`))); const candidate = loadCandidate(inputPath, candidateId); + if (candidate?.visualEvidence?.status !== "available") { + throw new Error("Candidate has no retrievable visual evidence. Re-export after retaining a local PNG before preparing a confirmed corpus case."); + } const expect = parseExpectedFields(); fs.mkdirSync(path.dirname(outputPath), { recursive: true }); const snippet = snippetFor(candidate, expect); @@ -106,4 +130,5 @@ module.exports = { parseExpectedFields, snippetFor, stableId, + visualEvidenceNote, }; diff --git a/src/eval/prepareConfirmedReviewCaseScript.test.ts b/src/eval/prepareConfirmedReviewCaseScript.test.ts index f7a80c4..d3e2972 100644 --- a/src/eval/prepareConfirmedReviewCaseScript.test.ts +++ b/src/eval/prepareConfirmedReviewCaseScript.test.ts @@ -4,8 +4,9 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; -function writeCandidatePayload(dir: string) { +function writeCandidatePayload(dir: string, visualEvidenceStatus: "available" | "unavailable" = "available") { const inputPath = path.join(dir, "review-eval-candidates.json"); + const localCropPath = path.join(dir, "private-artifact-crop.png"); writeFileSync( inputPath, JSON.stringify({ @@ -15,7 +16,16 @@ function writeCandidatePayload(dir: string) { id: "review-2026-07-08T15-02-39-765Z-152", savedAt: "2026-07-08T15:02:39.765Z", reason: "automatic:parser-notes:p1:r2c3", - resolution: "1920x1080", + resolution: localCropPath, + captureId: "20260711-154827:10", + capturedAt: "2026-07-11T15:48:28.000Z", + capturePath: localCropPath, + visualEvidence: { + status: visualEvidenceStatus, + source: visualEvidenceStatus === "available" ? "local-crop" : "none", + reference: "20260711-154827:10", + localPath: localCropPath, + }, ocr: { "artifact-name": "Pristine Plume of the Blessed", "artifact-slot": "Plume of Death", @@ -67,6 +77,10 @@ describe("prepare confirmed review case script", () => { expect(snippet).toContain("Pristine Plume of the Blessed"); expect(snippet).toContain('"artifact-name": "Pristine Plume of the Blessed"'); expect(snippet).toContain("sourceCandidate=review-2026-07-08T15-02-39-765Z-152"); + expect(snippet).toContain("visualEvidence=available"); + expect(snippet).toContain("captureId=20260711-154827:10"); + expect(snippet).toContain("capturedAt=2026-07-11T15:48:28.000Z"); + expect(snippet).not.toContain(path.join(dir, "private-artifact-crop.png")); expect(snippet).not.toContain("confirmed: false"); } finally { rmSync(dir, { recursive: true, force: true }); @@ -88,4 +102,31 @@ describe("prepare confirmed review case script", () => { rmSync(dir, { recursive: true, force: true }); } }); + + it("rejects candidates without retrievable visual evidence", () => { + const dir = mkdtempSync(path.join(tmpdir(), "gaa-confirmed-review-")); + try { + const inputPath = writeCandidatePayload(dir, "unavailable"); + let thrown: unknown; + try { + execFileSync( + "node", + [ + "scripts/prepare-confirmed-review-case.cjs", + `--input=${inputPath}`, + "--candidate=review-2026-07-08T15-02-39-765Z-152", + '--expect-json={"name":"Pristine Plume of the Blessed"}', + ], + { cwd: process.cwd(), stdio: "pipe" }, + ); + } catch (error) { + thrown = error; + } + expect(thrown).toBeDefined(); + const stderr = (thrown as { stderr?: Buffer | string }).stderr; + expect(String(stderr)).toContain("retrievable visual evidence"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); }); diff --git a/src/eval/reviewEvalCandidatesScript.test.ts b/src/eval/reviewEvalCandidatesScript.test.ts index ae5080c..3e7d3e7 100644 --- a/src/eval/reviewEvalCandidatesScript.test.ts +++ b/src/eval/reviewEvalCandidatesScript.test.ts @@ -4,12 +4,19 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; -function reviewRecord(savedAt: string, reason: string, ocr: Array<{ id: string; text: string }>, locked?: boolean) { +function reviewRecord( + savedAt: string, + reason: string, + ocr: Array<{ id: string; text: string }>, + locked?: boolean, + captureDetails: Record = {}, +) { return { savedAt, sample: { reason, capture: { + ...captureDetails, width: 1920, height: 1080, ocr, @@ -74,7 +81,10 @@ describe("review eval candidate exporter", () => { expect(payload.summary.exportStats.likelyStaleCaptures).toBe(1); expect(payload.summary.exportStats.equippedFooterCandidates).toBe(2); expect(payload.summary.exportStats.lockedTrueCandidates).toBe(1); + expect(payload.summary.exportStats.visualEvidenceAvailable).toBe(0); + expect(payload.summary.exportStats.visualEvidenceUnavailable).toBe(2); expect(payload.candidates[0].missingFastFields).toEqual([]); + expect(payload.candidates[0].visualEvidence.status).toBe("unavailable"); expect(payload.candidates[0].parsed.equipped).toBe("Aino"); expect(payload.candidates[0].reviewPrompt.expectedFields.equipped).toBe("Aino"); expect(payload.candidates[1].likelyStaleCapture).toBe(true); @@ -83,6 +93,63 @@ describe("review eval candidate exporter", () => { expect(markdown).toContain("- artifact-slot: 1"); expect(markdown).toContain("equipped=Aino"); expect(markdown).toContain("Locked=true candidates: 1"); + expect(markdown).toContain("Visual evidence unavailable: 2"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("prioritizes retrievable local crops and does not export their local paths", () => { + const dir = mkdtempSync(path.join(tmpdir(), "gaa-review-evidence-")); + try { + const inputPath = path.join(dir, "review-samples.jsonl"); + const outDir = path.join(dir, "out"); + const cropPath = path.join(dir, "artifact-0010.png"); + writeFileSync(cropPath, "png", "utf8"); + const staleOcr = completeOcr.filter((entry) => entry.id !== "artifact-slot"); + writeFileSync( + inputPath, + [ + JSON.stringify(reviewRecord("2026-07-11T10:00:00.000Z", "automatic:low-field:p1:r0c0", completeOcr)), + JSON.stringify(reviewRecord("2026-07-11T10:01:00.000Z", "automatic:low-field:p1:r0c0", completeOcr, false, { + id: "20260711-154827:10", + name: cropPath, + capturedAt: "2026-07-11T10:01:01.000Z", + })), + JSON.stringify(reviewRecord("2026-07-11T10:02:00.000Z", "automatic:capture-rejected:p1:r0c1", staleOcr, false, { + id: "20260711-154827:11", + name: cropPath, + capturedAt: "2026-07-11T10:02:01.000Z", + })), + JSON.stringify(reviewRecord("2026-07-11T10:03:00.000Z", "automatic:low-field:p1:r0c2", completeOcr)), + ].join("\n"), + "utf8", + ); + + execFileSync("node", ["scripts/export-review-eval-candidates.cjs", `--input=${inputPath}`, `--out=${outDir}`, "--limit=10"], { + cwd: process.cwd(), + stdio: "pipe", + }); + + const payload = JSON.parse(readFileSync(path.join(outDir, "review-eval-candidates.json"), "utf8")); + const markdown = readFileSync(path.join(outDir, "review-eval-candidates.md"), "utf8"); + + expect(payload.summary.uniqueCandidates).toBe(3); + expect(payload.summary.exportStats.visualEvidenceAvailable).toBe(2); + expect(payload.summary.exportStats.visualEvidenceUnavailable).toBe(1); + expect(payload.candidates.map((candidate: { captureId: string | null }) => candidate.captureId)).toEqual([ + "20260711-154827:10", + "20260711-154827:11", + null, + ]); + expect(payload.candidates.map((candidate: { visualEvidence: { status: string } }) => candidate.visualEvidence.status)).toEqual([ + "available", + "available", + "unavailable", + ]); + expect(JSON.stringify(payload)).not.toContain(cropPath); + expect(markdown).not.toContain(cropPath); + expect(markdown).toContain("Only manually label candidates with `visualEvidence: available`"); } finally { rmSync(dir, { recursive: true, force: true }); }