feat: ship agent-first mission control v0.2.57

2026-07-31 22:39:47 +02:00
parent 3bc7622977
commit f5552218bc
535 changed files with 95242 additions and 8791 deletions
@@ -0,0 +1,347 @@
+# Nexus Agent-first and Performance V2 — implementation and acceptance
+
+**Milestone date:** 2026-07-30
+**Final documentation review:** 2026-07-30
+**Scope:** local combined working tree; no commit, push, deployment or VPS write
+**Outcome:** core architecture implemented locally; production provisioning,
+live OpenClaw acceptance and measured load gates remain blocked or unproven
+
+## Executive verdict
+
+Nexus now has a coherent durable path for owner-approved agent proposals,
+PostgreSQL-backed workflow events, typed frontend contracts and an efficient
+Task Board read model. The implementation materially reduces duplicate
+requests and broad refreshes for the migrated domains.
+
+This is not a production-readiness or performance-budget claim:
+
+- productive OpenClaw writes remain intentionally disabled until a pinned
+  OpenClaw release officially supports an external Nexus or generic operator
+  Client ID;
+- no real OpenClaw pairing, agent creation or
+  `Nexus -> OpenClaw -> OpenAI -> Nexus` write flow was performed;
+- the controlled Playwright server is synthetic;
+- Docker/Testcontainers, Toxiproxy, k6, Promptfoo live evaluation,
+  `EXPLAIN (ANALYZE, BUFFERS)` and the browser p95 budget require separate,
+  isolated acceptance runs.
+
+## Implemented architecture
+
+### One typed API contract
+
+- ASP.NET Core emits OpenAPI 3.1 to
+  `backend/openapi/Nexus.Api.json`.
+- `openapi-typescript` generates
+  `frontend/src/api/generated/schema.d.ts`; `openapi-fetch` is the typed
+  transport for migrated endpoints.
+- CI regenerates the schema and rejects a diff.
+- `ProblemDetails` and `ValidationProblemDetails` are the common HTTP error
+  shape. The server attaches `traceId`; the frontend uses one error adapter.
+- `EntityRefDto`, `OperationResultDto`, `DomainEventDto` and
+  `TaskBoardPageDto` are explicit public contracts rather than view-model
+  inference.
+
+### Server-state migration
+
+TanStack Vue Query owns the migrated server state:
+
+- Task Board;
+- projects and project-scoped tasks;
+- agent proposals and create options;
+- Nexus activity and notifications;
+- OpenClaw overview, agents, runs, cron and models;
+- agent details and configuration reads; and
+- owner-only Memory, Docs, Incidents and Security reads.
+
+The Query-key factory makes multiple consumers share one request. Background
+refresh keeps previous visible data. Pinia still owns authentication, command
+and Iris modal state, setup/wizard workflow state, local drafts and shared
+mutation facades; it does not duplicate canonical runtime collections.
+
+After route and error-state parity checks, the legacy operations/task/
+notification/dashboard server-state stores, static agent inventory and
+mappers, duplicate `liveSync.ts`/`live-sync.ts` modules and the old generic
+live-service reader were removed. The backend compatibility endpoints remain
+temporarily available, but no active frontend consumer uses them as its
+canonical cache.
+
+Memory, Docs and Incidents no longer use host workspace mounts. Their
+owner-only services read OpenClaw workspace content through confined RPC,
+return source-agent and workspace-path provenance and expose no arbitrary
+workspace write path.
+
+### PostgreSQL workflow and event backbone
+
+Migration `20260730224500_AddAgentProvisioningAndBoardIndexes` adds:
+
+- `AgentProposals`;
+- `AgentProvisionRequests`;
+- `OperationClaims`; and
+- `OutboxEvents`.
+
+For migrated Nexus-owned mutations, domain state and outbox event are stored in
+the same EF transaction. The registered OpenClaw mutation claim store also uses
+PostgreSQL `OperationClaims` plus a database transaction and lock; the former
+JSONL path is exposed only for discovery of an immutable legacy archive and is
+not read or appended. The background worker:
+
+- leases pending work with `FOR UPDATE SKIP LOCKED`;
+- uses a bounded single-process wake-up channel plus database polling for
+  restart recovery;
+- publishes content-minimized events to subscriber queues of 64 entries;
+- retains at least 24 hours and at least 10,000 global sequences; and
+- exposes at most 512 replay deltas per reconnect.
+
+`GET /api/v1/events?afterSequence=` resumes by global sequence. A missing or
+expired range produces `resync_required`; the browser refreshes affected Query
+domains rather than silently accepting a gap. The existing OpenClaw event
+projection remains a separate sanitized Runtime adapter during migration.
+
+### Authenticated SSE hub
+
+The frontend uses one fetch-based `AuthenticatedSseHub` with:
+
+- bearer authentication and the existing refresh path;
+- `eventsource-parser`;
+- abort and subscriber lifecycle;
+- heartbeat detection;
+- jittered reconnect; and
+- a 256 KiB maximum parser buffer.
+
+Task events normally reconcile one card through
+`GET /api/v1/tasks/{id}/board-card`. Other events batch small Query-domain
+invalidations. The Dashboard legacy SSE path was hardened for cursor
+continuation and cancellation but remains a compatibility surface.
+
+## Agent proposal and provisioning flow
+
+### Shared flow
+
+The manual `/agents/new` UI and Iris use the same durable service:
+
+```text
+local form draft -> awaiting_approval -> provisioning
+                                      -> ready | partial | failed | in_doubt
+                 -> rejected
+```
+
+New APIs:
+
+- `GET /api/v1/openclaw/agents/create-options`
+- `GET|POST /api/v1/openclaw/agent-proposals`
+- `GET /api/v1/openclaw/agent-proposals/{proposalId}`
+- `POST /api/v1/openclaw/agent-proposals/{proposalId}/approve`
+- `POST /api/v1/openclaw/agent-proposals/{proposalId}/reject`
+- `POST /api/v1/openclaw/agent-proposals/{proposalId}/retry`
+
+Proposal lists use a stable `(CreatedAt, Id)` keyset cursor, including equal
+timestamps. Owner mutations require an Idempotency Key, correlation ID,
+optimistic proposal revision and authenticated actor.
+
+### Iris boundary
+
+Iris receives only:
+
+- `nexus_propose_agent`; and
+- `nexus_get_agent_proposal`.
+
+Both MCP tools return structured content and stable state/error information.
+They cannot approve a proposal or call OpenClaw. Tool annotations describe
+read-only, destructive, idempotent and open-world behavior but are not used as
+authorization.
+
+### Provisioning safety
+
+After explicit owner approval, Nexus rechecks:
+
+- local `ManagementEnabled`;
+- official external Client-ID support;
+- normalized endpoint and TLS trust;
+- current capability hash and required advertised methods;
+- `operator.admin`;
+- proposal revision; and
+- idempotency claim.
+
+The workspace root is server-controlled and checked against live OpenClaw
+configuration. Nexus calls `agents.create` at most once. After a possible
+dispatch timeout it records `in_doubt` and permits only read-only inventory
+reconciliation before another action. A successful create is not `ready` until
+`agents.list` confirms it. Approved standard files are written through
+`agents.files.set` and read back; a later file failure becomes `partial` and
+does not trigger automatic deletion.
+
+The production button remains disabled because OpenClaw `2026.7.1` does not
+register the required external Nexus/generic operator identity. Nexus does not
+impersonate CLI, Control UI or `gateway-client/backend`.
+
+## Task Board V2
+
+`GET /api/v1/tasks/board?doneLimit=50&doneCursor=` returns:
+
+- every non-Done card on the initial page;
+- the newest 50 Done cards by default;
+- an opaque `(UpdatedAt, Id)` keyset cursor for further Done pages; and
+- a stable board revision.
+
+The repository uses `AsNoTracking` and direct DTO projection. Child counts and
+latest activity are correlated scalar projections rather than N+1 reads. The
+initial path uses at most three SQL statements; a Done continuation skips the
+active-card query and uses two. The migration adds state, partial Done,
+activity, child and agent-workflow indexes.
+
+The Vue client uses `useInfiniteQuery`, deduplicates Done cards and keeps all
+active columns mounted for drag-and-drop. Moves are optimistic and roll back on
+failure. Create, update, move and domain events fetch only the affected
+`board-card`; full invalidation is a recovery path. Dashboard, sidebar,
+Command Palette and Task Board use the same Query key instead of parallel board
+loads.
+
+No claim is made that the p95 budgets are met until the recorded
+1,000-task/10,000-activity dataset, k6 run, SQL trace/plans and repeated browser
+timing run have been executed.
+
+## Cross-page result navigation
+
+- `EntityRefDto` contains type, ID and optional label but no backend-generated
+  URL.
+- The frontend resolver maps agent, proposal, project, task, run, cron,
+  incident, document, notification and event-stream references to registered
+  routes.
+- Iris and proposal mutations can show `OperationResultCard` with primary and
+  affected entities plus operation/trace metadata.
+- Projects now have a real `/projects` index and
+  `GET /api/v1/projects/{id}/tasks`; Project Detail no longer loads unrelated
+  tasks.
+- Activity and Notifications use typed, authenticated domain queries and
+  navigate to available related entities.
+
+The 2026-07-31
+[structured operation-results follow-up](../../2026-07-31/operation-results-deep-links/IMPLEMENTATION_AND_ACCEPTANCE.md)
+closed the remaining task, project, notification, cron, config, approval,
+session and agent-file mutation gap. A global result tray now exposes the
+typed references and the addressed target surfaces consume their deep-link
+selection.
+
+## Resilience and observability
+
+- `Microsoft.Extensions.Http.Resilience` protects safe OpenClaw reads with a
+  10-second attempt timeout, 30-second total budget, bounded jittered retries
+  and a circuit breaker.
+- Management mutations and Chat/Run have no automatic retry; Gateway
+  WebSocket reconnect remains inside the connector.
+- OpenTelemetry instruments ASP.NET Core, HttpClient, Npgsql, Task Board,
+  Gateway RPC, proposals/provisioning, outbox and SSE-related metrics.
+- OTLP export is opt-in through configuration; no extra production container
+  is introduced.
+- `web-vitals` reports an allow-list of metric, value, rating, route name,
+  build version, live mode and correlation ID to
+  `POST /api/v1/telemetry/browser`.
+- Redaction removes URL queries/full URLs, SQL text, exception messages/stacks
+  and content-bearing attributes. Prompts, chat text, Markdown, tool arguments,
+  credentials, headers and entity names are not telemetry.
+
+## Dependency decisions
+
+Implemented and pinned:
+
+- `@tanstack/vue-query` `5.101.4`
+- `eventsource-parser` `3.1.0`
+- `openapi-fetch` `0.17.0`
+- `openapi-typescript` `7.13.0`
+- `web-vitals` `5.3.0`
+- Playwright `1.62.0`
+- Testcontainers PostgreSQL/Toxiproxy `4.13.0` in the test project only
+- OpenTelemetry `1.17.0`
+- `Microsoft.Extensions.Http.Resilience` `10.0.0`
+
+`ModelContextProtocol.AspNetCore` remains pinned to `1.4.1`. The
+`scripts/qa/test-mcp2-compatibility.ps1` candidate mode copies the project to a
+temporary directory and may probe `2.0.0`, but it cannot open the upgrade gate
+without isolated live OpenClaw negotiation. The local isolated `2.0.0`
+compile/test probe is green; production deliberately remains on `1.4.1`.
+
+Not introduced: Redis, NATS, Kafka, RabbitMQ, Temporal, Hangfire, Quartz,
+GraphQL, SignalR, RxJS, direct OpenAI Agents/Responses orchestration, OPA,
+OpenFGA, pgvector or a second production service.
+
+## Automated verification
+
+The following non-live checks were completed on 2026-07-30:
+
+| Gate | Result |
+|---|---|
+| .NET 10 / MCP `1.4.1` baseline | **Passed: 360; failed: 0; skipped: 5; total: 365** |
+| Isolated MCP `2.0.0` candidate copy | **Passed: 360; failed: 0; skipped: 5**; compatibility probe only |
+| MCP production pin/static markers | **Passed**; production project remains `1.4.1` |
+| Promptfoo wrapper `-ValidateOnly` | **Passed**; configuration only, no live evaluation |
+| PowerShell AST parse for all QA scripts | **Passed** |
+| `node --check` for k6, QA mock and Promptfoo provider | **Passed** |
+| Frontend typecheck | **Passed**; full Vue application and E2E TypeScript projects |
+| Frontend unit tests | **Passed: 12 files, 28 tests** |
+| Frontend production build | **Passed: 1,984 modules transformed** |
+| Playwright controlled contract suite | **Passed: 24 tests** across all 20 core routes and 375/768/1024/1440/1920 px |
+| Production dependency audit | **Passed**; no known vulnerabilities after pinning PostCSS `8.5.18` |
+| OpenAPI generation repeatability | **Passed**; checked-in backend contract and generated TypeScript schema were stable |
+
+The five skips are three explicitly gated PostgreSQL/Testcontainers
+provisioning tests, one Toxiproxy test and one PostgreSQL operation-claim
+concurrency test. Docker CLI was available but its daemon was not running.
+They were not executed and must not be described as passed integration
+evidence. The Playwright server is a controlled local contract fixture, not a
+real OpenClaw.
+
+## Not executed or not proven
+
+The following acceptance evidence was not produced by this milestone:
+
+1. real pairing or scope upgrade against Bao's OpenClaw;
+2. a live agent create, file write or recovery mutation;
+3. a complete `Nexus -> OpenClaw -> OpenAI -> Nexus` run;
+4. Docker-backed PostgreSQL/Toxiproxy evidence when the environment gate or
+   Docker daemon is unavailable;
+5. k6 p95 thresholds on a verified 1,000-task/10,000-activity fixture;
+6. `EXPLAIN (ANALYZE, BUFFERS)` and independent SQL-statement-count evidence;
+7. repeated browser `navigation -> cards visible` p95 evidence;
+8. live Promptfoo accuracy/injection evaluation against an isolated Nexus and
+   test OpenClaw;
+9. live MCP 2.0 `tools/list`, Streamable HTTP and down-level negotiation; and
+10. production OTLP or `pg_stat_statements` activation.
+
+The repository contains guarded scripts and tests for several of these checks.
+Static syntax/config validation proves only the artifacts, not their external
+systems or thresholds.
+
+## Residual work and release gates
+
+1. Obtain and pin official external Nexus/generic-operator Client-ID support.
+2. Run read-only pairing and inventory acceptance, then a separately approved
+   disposable management-write sequence.
+3. Run Docker integration, Toxiproxy, k6, SQL-plan, repeated browser and
+   Promptfoo release gates with archived sanitized evidence.
+4. Enable `pg_stat_statements` only through a separately approved PostgreSQL
+   maintenance window because it requires server configuration/restart.
+5. Reconsider virtualization, pgvector/QMD and external policy engines only
+   after measurements or multi-operator/tenant requirements justify them.
+
+## Primary code and evidence pointers
+
+- `backend/openapi/Nexus.Api.json`
+- `backend/Data/Migrations/20260730224500_AddAgentProvisioningAndBoardIndexes.cs`
+- `backend/Services/AgentProposalService.cs`
+- `backend/Services/DomainEventStreamService.cs`
+- `backend/Repositories/TaskRepository.cs`
+- `frontend/src/api/queryClient.ts`
+- `frontend/src/api/taskBoard.ts`
+- `frontend/src/services/sseHub.ts`
+- `frontend/src/services/domainEvents.ts`
+- `frontend/e2e/agent-proposals.e2e.ts`
+- `frontend/e2e/route-smoke.e2e.ts`
+- `frontend/e2e/task-board.e2e.ts`
+- [QA automation and evidence boundaries](../../../QA_AUTOMATION.md)
+- [Agent-first target contract](../../../AGENT_FIRST_MISSION_CONTROL.md)
+- [OpenClaw Attach & Adopt evidence](../openclaw-attach-adopt/IMPLEMENTATION_AND_ACCEPTANCE.md)
+
+## External compatibility reference
+
+- [OpenClaw `2026.7.1` client-ID registry](https://github.com/openclaw/openclaw/blob/v2026.7.1/packages/gateway-protocol/src/client-info.ts)
+- [OpenClaw Gateway protocol](https://github.com/openclaw/openclaw/blob/v2026.7.1/docs/gateway/protocol.md)