feat: ship agent-first mission control v0.2.57

docs/QA_AUTOMATION.md

@@ -0,0 +1,185 @@
+# Nexus QA Automation
+
+These checks are development and CI artifacts. They add no production
+dependency and must be run against an isolated Nexus test database unless a
+section explicitly says otherwise.
+
+This document describes what each artifact can prove. A script being present,
+or its static preflight passing, is not acceptance evidence for a live
+OpenClaw, PostgreSQL, browser, or load-test boundary. Archive the command,
+versions, sanitized output, dataset provenance, and timestamp for every
+acceptance run.
+
+## Task Board load gate
+
+The full k6 profile holds ten virtual users for two minutes and fails when:
+
+- initial Task Board request p95 is `>= 500 ms`;
+- Done-cursor request p95 is `>= 300 ms`;
+- initial or Done HTTP failures are `>= 1%`;
+- board-level failures are `>= 1%`;
+- board-contract failures are `>= 1%`; or
+- the acceptance dataset does not expose a Done continuation cursor; or
+- successful checks fall to `<= 99%`.
+
+Acceptance preconditions are deliberately external to the load generator:
+
+- use a disposable PostgreSQL database with a recorded seed of 1,000 Tasks and
+  10,000 Activities;
+- include more than 50 Done Tasks so the cursor path receives samples;
+- record the exact seed command or fixture revision and verify row counts
+  before k6; and
+- warm the service before capturing the two-minute result.
+
+From the repository root:
+
+```powershell
+$env:NEXUS_BASE_URL = "http://127.0.0.1:18880"
+$env:NEXUS_BEARER_TOKEN = "<fresh JWT>"
+k6 run scripts/qa/task-board.k6.js
+```
+
+`NEXUS_API_KEY` can replace the bearer token for this authenticated read-only
+test. A ten-second one-VU diagnostic is available with
+`NEXUS_K6_SMOKE=1`; it is not acceptance evidence. The smoke profile does not
+require a Done cursor by default. `NEXUS_K6_REQUIRE_DONE_CURSOR=0` may also
+disable that gate for diagnostics, but any such run is non-acceptance.
+
+Remote targets are blocked by default. An isolated remote target requires both
+`NEXUS_K6_ALLOW_REMOTE=1` and HTTPS. `NEXUS_INSECURE_TLS=1` is accepted only
+for loopback diagnostics. The base URL may not contain credentials, a path,
+query, or fragment.
+
+### What k6 does not prove
+
+- k6 cannot verify the 1,000/10,000 database row counts; preserve independent
+  seed and count evidence.
+- `Server-Timing` proves only that the header exists. It does not prove the
+  “at most three SQL statements” invariant.
+- Capture SQL-statement counts through a focused integration test or
+  instrumented database trace, and archive
+  `EXPLAIN (ANALYZE, BUFFERS)` for the initial and Done-cursor queries.
+- The “navigation until cards visible” p95 of 1.5 seconds is a browser budget,
+  not an HTTP budget. It requires a repeated Playwright/browser performance
+  run with the same recorded dataset. A functional Playwright pass or a k6
+  result alone does not satisfy it.
+
+## Agent-first Promptfoo cases
+
+The cases run through Nexus `/api/v1/chat`, wait for the durable Protocol-v4
+run, and compare agent and proposal inventories before and after the run. They
+cover:
+
+- literal observation of `nexus_propose_agent` plus matching proposal
+  arguments;
+- attempted owner-approval bypass;
+- a server-rooted proposal after a workspace-traversal request; and
+- an explicit request to mutate OpenClaw without approval.
+
+They intentionally create proposal-only records. Use a disposable database and
+a fresh owner JWT. Remote targets are blocked unless
+`NEXUS_EVAL_ALLOW_REMOTE=1` is explicitly set, and non-loopback targets must
+use HTTPS. Embedded URL credentials, paths, queries, and fragments are rejected.
+
+The pinned configuration and provider can be validated without a token or live
+Nexus:
+
+```powershell
+./scripts/qa/run-agent-first-evals.ps1 -ValidateOnly
+```
+
+```powershell
+$env:NEXUS_EVAL_BASE_URL = "http://127.0.0.1:18880"
+$env:NEXUS_EVAL_BEARER_TOKEN = "<fresh owner JWT>"
+$env:NEXUS_EVAL_ALLOW_PROPOSALS = "1"
+./scripts/qa/run-agent-first-evals.ps1
+```
+
+For the repeated release profile, use the same isolated fixture and run:
+
+```powershell
+./scripts/qa/run-agent-first-evals.ps1 -Repeat 20
+```
+
+The wrapper pins Promptfoo `0.121.19` without adding it to a production or
+frontend package manifest. Cases run serially because their durable
+before/after inventories must not overlap. Non-loopback targets require HTTPS.
+A successful run still leaves test proposals in the test database so their
+approval state can be inspected.
+
+### Promptfoo evidence boundary
+
+- Literal tool selection passes only when browser-safe Gateway history exposes
+  `nexus_propose_agent`. A durable proposal is recorded separately and is not
+  substituted as proof of a visible raw tool call.
+- The workspace case proves only that the resulting proposal path is under the
+  server-advertised root, no new agent appears, and no dangerous mutation is
+  visible in returned run history.
+- It cannot prove that a model or hidden tool never read a foreign filesystem
+  path. Absence from redacted history is not evidence of absence. Server-side
+  workspace-confinement, authorization, and audit tests remain mandatory.
+- One execution of four cases is not a statistically meaningful 95% tool- and
+  argument-accuracy result. A release claim needs a recorded repeated run
+  (at least 20 independent tool-selection samples, at least 19 correct) against
+  a resettable isolated fixture. `-Repeat 20` repeats every case serially and
+  therefore creates many proposal records; use it only with a resettable
+  fixture. Promptfoo's default exit behavior is stricter and marks the command
+  failed if any repeated case fails. The default one-repeat suite is a boundary
+  regression gate, not that accuracy report.
+
+## MCP 2.0 compatibility gate
+
+The repository remains pinned to `ModelContextProtocol.AspNetCore` `1.4.1`.
+The default static gate verifies that pin plus the structured proposal-tool
+source/test markers and does not run those tests or need an SDK:
+
+```powershell
+./scripts/qa/test-mcp2-compatibility.ps1
+```
+
+With .NET SDK 10 installed, run the stable baseline:
+
+```powershell
+./scripts/qa/test-mcp2-compatibility.ps1 -Mode Baseline
+```
+
+The candidate mode first requires a green 1.4.1 baseline, then copies
+`backend/` and `backend-tests/` to a verified temporary directory, changes only
+that copy, restores the exact candidate and runs the backend test project:
+
+```powershell
+./scripts/qa/test-mcp2-compatibility.ps1 `
+  -Mode Candidate `
+  -CandidateVersion "2.0.0"
+```
+
+A preview or stable candidate can produce only an SDK compatibility-probe pass.
+It never opens the production promotion gate because this local probe does not
+exercise a real OpenClaw `tools/list`, external client identity, Streamable
+HTTP, or down-level negotiation. The original project file is hash-checked
+before and after the run and remains pinned to 1.4.1. A stable 2.x upgrade still
+requires live evidence against an isolated compatible OpenClaw plus a separate
+reviewed dependency change.
+
+`Baseline` and `Candidate` report success when `dotnet test` exits successfully.
+Environment-gated Docker, Toxiproxy, and live OpenClaw tests can still be
+skipped; inspect and archive the test summary instead of describing the result
+as complete protocol compatibility.
+
+## Local artifact preflight
+
+The following checks are safe and do not contact Nexus, OpenClaw, Docker, or a
+k6 target. Promptfoo validation may download the pinned npm package when it is
+not already cached.
+
+```powershell
+node --check scripts/qa/task-board.k6.js
+node --check scripts/qa/openclaw-ui-mock.mjs
+./scripts/qa/run-agent-first-evals.ps1 -ValidateOnly
+./scripts/qa/test-mcp2-compatibility.ps1 -Mode Pin
+```
+
+PowerShell scripts should additionally be parsed with the PowerShell AST parser.
+These preflights validate syntax/configuration and static invariants only. They
+are not substitutes for k6, Promptfoo evaluation, browser budgets, database
+query-plan evidence, or live MCP/OpenClaw negotiation.