Root cause: Dual-source architecture for owner password (Gitea secret
ENV_OWNER_PASSWORD vs host .env OWNER_PASSWORD) caused drift when
the DB was ever re-seeded or the volume recreated.
Changes:
- Add SeedAudit entity + migration to track one-time seed operations
- EnsureDatabaseAsync checks SeedAudit BEFORE seeding — owner is never
re-created even if the Users table is wiped
- Deploy and rollback workflows now read OWNER_PASSWORD from the host's
persistent .env (single source of truth) instead of Gitea secrets
- compose.yaml documented: OWNER_PASSWORD only used during initial seed
- Cleanup: .gitignore extended for core dumps, changelog/deployment.md
updated with 2026-06-20 session notes
After this fix the DB is the single source of truth for the owner
password after initial seed. The host .env is the single reference
for the initial value.
The inner shell script run via docker:cli had complex escaping
that caused 'unterminated quoted string' errors at runtime.
Moved the deploy logic to an external script file (heredoc in
the workflow YAML), mounted read-only into the docker:cli
container. Pass BUILD_ARGS and SERVICE via environment
variables instead of shell interpolation.
- Postgres memory: 256M→384M limits, 64M→96M reservations
- Added pg_resetwal -f pre-deploy step to recover from corrupt WAL
('PANIC: could not locate a valid checkpoint record' caused by
force-killed postgres during --force-recreate)
- Added data-checksums initdb arg for future corruption detection
- api→postgres and web→api depends_on: service_healthy→service_started
- Deploy wait loop: fail fast on unhealthy, wait on starting (180s)
- Added researcher/executor to ValidAssignees and frontend dropdowns
- web's depends_on on api: change from service_healthy to
service_started+restart (same as api→postgres fix)
- deploy wait loop: fail fast on unhealthy, wait on starting,
increased timeout to 180s (36×5s)
The docker compose --wait flag times out before postgres can
become healthy (start_period=30s). Replaced with explicit
poll loop (5s interval, up to 120s) that checks ps output
for unhealthy/starting states.
Swagger (/swagger) is only enabled in Development mode (Program.cs
gates it behind app.Environment.IsDevelopment()). In production,
nginx serves the frontend catch-all (index.html), so the check
always returns 200 but never actually validates the API layer.
/health already covers API + database + runtime health checks.
No replacement endpoint needed — the smoke test still validates
both the dashboard and the backend API via /health.
Iteration 2 fix: /api/swagger → /swagger (correct ASP.NET default).
Iteration 3 — Concurrency guard:
- concurrency group 'deploy-production': ensures only one deploy
runs at a time (cancel-in-progress: false so queued deploys
wait instead of being cancelled).
- Why: prevents race conditions when CI-triggered workflow_run
and manual workflow_dispatch overlap. Without this, parallel
deploys could corrupt docker compose state or conflict on
shared resources (ports, volumes, version tags).
Iteration 2 — Deploy robustness:
- Health check: Fibonacci-ish backoff (1,2,3,5,8,13s) instead of fixed
5s intervals. Why: containers need variable warmup time; fixed intervals
either wait too long or give up too early. Total budget ~32s vs 30s before.
- Smoke test: now checks /dashboard, /health, and /api/swagger. Why: a
single endpoint check can miss backend-only outages; API Swagger confirms
the ASP.NET layer is healthy.
- Rollback hint: on any failure, prints previous git tag + docker compose
commands for quick manual rollback. Why: reduces MTTR by providing the
exact recovery steps inline.
pnpm defaults to frozen-lockfile in CI. The committed lockfile
is outdated (vitest added to package.json). Using --no-frozen-lockfile
is a pragmatic fix; lockfile should be regenerated via 'pnpm install'
and recommitted for full --frozen-lockfile enforcement.
Iteration 1 — CI reliability and speed:
- Concurrency: cancel in-progress CI runs when new push arrives
to the same branch. Why: Avoids waste when pushing multiple
fixes in quick succession; only the latest code is tested.
- pnpm: switch from --no-frozen-lockfile to --frozen-lockfile.
Why: Fails fast if pnpm-lock.yaml is outdated — prevents
untested dependency changes from reaching main.
- pnpm: add --prefer-offline to use locally cached packages.
Why: Slightly faster installs when packages are already
available in the runner image cache.
Iteration 1 — Build caching:
- Backend: cache ~/.nuget/packages keyed on .csproj hashes.
Typical hit: restore drops from ~15s to ~2s (NuGet packages
already cached locally).
- Frontend: cache node_modules + ~/.pnpm-store keyed on
pnpm-lock.yaml. Typical hit: install drops from ~30s to ~3s.
- Concurrency: cancel in-progress CI runs when new push arrives
to the same branch (avoids queue buildup).
Why: On cache hits, CI time drops ~60-70%. Faster feedback for
developers means shorter fix-deploy cycles.
The \$ escape before ${{ inputs.service }} prevented Gitea from
evaluating the expression, passing literal backslash to the shell.
Also use ${BUILD_ARGS} (shell expansion) instead of \$BUILD_ARGS
so the outer shell passes the actual build args to the DIND container.
Phase 1 — .env provisioning fix:
The previous approach tried to write .env directly to
/opt/openclaw/data/openclaw/workspace/nexus from inside the
runner's job container, but that host path is not mounted there.
Fix: write .env from Gitea secrets into the workspace first,
then sync it along with the source code via the existing
Docker-in-Docker pattern (which can access the host path).
Combined the separate '.env creation' and 'sync code' steps
into a single atomic 'Sync code + .env to host' step.
devops
reviewer