apps/dashboard — Dashboard deploy
Fro Bot operator dashboard for dashboard.fro.bot. A two-service Docker Compose stack (dashboard +
caddy) on a dedicated DigitalOcean droplet, fronted by Caddy for automatic HTTPS. The dashboard image
is the upstream released image from ghcr.io/fro-bot/dashboard, pinned by digest in
apps/dashboard/docker-compose.yaml. The deploy pulls the digest-pinned image — no on-droplet build.
| Service | Image | Role |
|---|---|---|
dashboard |
ghcr.io/fro-bot/dashboard:<tag>@sha256:<digest> (digest-pinned in compose) |
Hono app on :3000; GitHub App + OAuth; healthcheck at /api/healthz |
caddy |
caddy Alpine image (tag and digest pinned in compose) |
Auto-TLS reverse proxy :443 → dashboard:3000 |
The dashboard image tag and digest are pinned directly in apps/dashboard/docker-compose.yaml.
Versioned dispatches deploy the requested release, then open or update the dashboard-pin audit PR
to record the deployed pin. The deploy router skips commits whose subject starts with
chore(dashboard): pin image to , so merging that audit PR does not deploy the image again.
Caddy depends on the dashboard service being healthy before starting.
Deploy flow
Section titled “Deploy flow”bun run deploy:dashboard from the repository root (loads the root .env), or the Deploy Dashboard GitHub workflow. Both entry paths use the same deploy engine. The deploy:
- Validates env (
DASHBOARD_DOMAIN,DASHBOARD_GITHUB_APP_ID,DASHBOARD_GITHUB_APP_KEY,DASHBOARD_FRO_BOT_APP_ID(positive integer),DASHBOARD_FRO_BOT_APP_KEY(PEM private key),DASHBOARD_OAUTH_CLIENT_ID,DASHBOARD_OAUTH_CLIENT_SECRET,DASHBOARD_OPERATOR_LOGIN,DASHBOARD_COOKIE_KEY, plus SSH context) and the host string before any SSH argv is built. - DNS preflight — resolves
DASHBOARD_DOMAINand fails fast if it does not resolve. - Generates the target digest and Compose content locally. Versioned deploys resolve the GHCR
digest and cross-check any dispatched digest; local/no-version deploys use the committed Compose
pin. The
.env, Caddyfile, and PEM payload are also assembled locally before SSH. - Starts one SSH process whose remote transaction validates
/run/dashboard-deployas a canonical, root-owned0700directory and acquires/run/dashboard-deploy/lockwith a 180-second bounded wait. The kernel lock is held by the process doing the work and releases on process death; there is no stale-lock cleanup or separate lock-holder. The remote Bash transaction is wrapped in a fixed 900-secondtimeout(TERM, thenKILLafter 15 seconds), and the local caller has a 960-second watchdog that escalates the SSH process if its lifecycle does not settle. - After the lock, validates
/opt/dashboard,/opt/dashboard/config, and/opt/dashboard/datawithout accepting symlinks or unsafe types, then records baseline storage, Docker disk, container, active Compose, and running dashboard evidence. - Always runs
docker image prune -af. A prune failure stops the deploy. Docker may reclaim image data through this command, but the deployment never directly deletes containerd storage or files; it never prunes containers or volumes or uses Compose teardown. The first 6 GiB free-space gate is checked after post-prune evidence. - Enumerates the staged Compose image set and verifies every exact
repository@digestlocally first. If all are cached, it skips the pull and recordsacquisition:mode=cache. Otherwise it runs the staged Compose pull, verifies the complete exact set, and recordspull; a failed pull is allowed only when the complete exact cache is present afterward, recorded ascache-fallback. Tags are never trusted. The second 6 GiB gate is checked after acquisition and before active mutation. - Converges
/opt/dashboard/datato a real directory owned by1000:1000with mode0700, then publishes.env,Caddyfile, the GitHub App PEM, and Compose one file at a time, with Compose last. The PEM is a regular file owned by1000:1000, mode0600;.envis root-owned mode0600; Caddyfile and Compose are root-owned mode0644. - Runs
docker compose up -d --no-build --force-recreate --wait --wait-timeout 120 dashboard, so the container is recreated on every deploy and picks up the newly published PEMs (single-file bind mounts keep the old inode), verifies the running dashboard digest against the expected digest and its health, then force-recreates and waits for Caddy. The lock remains held through this convergence and releases when the SSH transaction exits. - After unlock, runs the same-origin operator-health and public HTTPS probes as advisory checks. Versioned deploys write the local Compose pin for the audit PR only after the remote transaction succeeds and both advisory probe attempts finish; probe warnings or failures do not block the write-back.
In CI the SSH key is materialized from DASHBOARD_SSH_KEY to a temp file with a trailing newline
(GitHub strips trailing whitespace from secrets) and chmod 600; locally it uses the ssh-agent.
Staleness guard
Section titled “Staleness guard”- The first
deploy-dashboardstep,Reject stale dashboard dispatch, runs before app-token minting and checkout. Onworkflow_dispatch, it listsdeploy-dashboard.yamlruns with adatabaseIdgreater than the current run’s and fails if any exist. - It is scoped to
workflow_dispatchonly. Underworkflow_callfrom the deploy router, the callee has no run of its own andgithub.run_idbelongs to the router, so the comparison is meaningless. - The job declares
contents: readandactions: read; a job-levelpermissions:block replaces the workflow-level block rather than merging. The router caller grants the same pair, andpackages/cli/src/conventions.test.tsenforces the parity. - The Actions cancel API clears an environment-gated run, with the owner’s token or the workflow
token (
actions: write). It once accepted a cancel and did nothing, so always read back the run status. If a stale run is approved anyway, it fails quickly, freeing the concurrency slot for the newer run. - Any newer run takes precedence regardless of its outcome. If it was rejected or failed, the older run still
refuses to deploy; dispatch a fresh run rather than approving the original. A transient
gh run listAPI failure hard-fails before deployment secrets are touched — deliberate fail-closed behavior that costs one approval click. cancel-superseded-dashboard-deploys.yamlcancels older waitingworkflow_dispatchruns onmainwhen a newerDeploy Dashboardrun is requested (it fails open and reads back each run’s status); router-dispatched runs (Deployviaworkflow_call) are not cancelled.
Retention and failure handling
Section titled “Retention and failure handling”Pruning is pre-deploy only. The replaced image is left locally as one temporary rollback generation after a successful replacement, but the next deployment attempt may prune it. This is not an automatic rollback guarantee. Stopped containers pin their images just like running containers. If those pins keep free space below 6 GiB, inspect the stopped containers, remove only specifically obsolete ones manually, and rerun; no automatic override flag exists.
The operator must resolve the reported condition before rerunning. Every remote failure returns a stable lowercase-hyphen code plus the last completed stage; remote stderr is intentionally not surfaced. Deterministic failure classes are:
- Lock contention: wait for the other transaction to finish, then rerun.
- Prune failure: resolve the Docker cleanup error; do not bypass pruning.
- Post-prune low headroom: inspect storage and stopped-container pins; remove only specifically obsolete stopped containers manually, then rerun.
- Acquisition/cache mismatch: restore registry access or make every staged exact digest available; a tag-only or wrong-digest cache entry does not qualify.
- Post-acquisition low headroom: resolve capacity before allowing active publication.
- Active publication/runtime failure: inspect the reported stage and actual Compose/runtime state; do not assume automatic rollback, then correct the condition and rerun.
- Transaction timeout: inspect the reported stage and host state after the fixed remote deadline;
retry only after confirming the previous process is gone. The caller watchdog also terminates a
hung local SSH process. GNU
timeoutstatuses 124 and 137 are conservatively normalized to the reserved transaction-timeout code because the utility cannot distinguish a child’s independent exit 137 from kill-after escalation.
Image source
Section titled “Image source”The dashboard image is the upstream released image from ghcr.io/fro-bot/dashboard, pinned by tag
and digest in apps/dashboard/docker-compose.yaml. A versioned Deploy Dashboard dispatch resolves
the release digest and deploys it before recording the pin through the dashboard-pin audit PR.
The audit PR records an already completed deployment; its merge intentionally skips router redeployment.
Renovate also tracks the image, but only opens a bump PR once it is approved on the Dependency
Dashboard; merging that PR deploys the new pin through the router and the dashboard gate.
The former ghcr.io/marcusrbrown/infra-dashboard image (built by this repo’s CI) is retired. It can
be deleted manually from the GitHub Container Registry once no references remain.
Container hardening
Section titled “Container hardening”The dashboard service runs with a deliberate security hardening block — a delta from apps/umami:
read_only: truecap_drop: - ALLsecurity_opt: - no-new-privileges:trueuser: nodetmpfs: - /tmpread_only: true— the container filesystem is read-only;/tmp(tmpfs) and the listener data mount are the writable surfaces. The App-key mount remains read-only.cap_drop: [ALL]— drops all Linux capabilities; the Hono app needs none.no-new-privileges: true— prevents privilege escalation via setuid/setgid binaries.user: node— runs as the non-rootnodeuser baked into the upstream image.tmpfs: /tmp— provides a writable scratch space without touching the host filesystem.
Persistent listener storage:
volumes: - type: bind source: /opt/dashboard/data target: /data bind: create_host_path: falseThe /data bind mount is the writable surface for the dashboard’s listener SQLite state. The host
directory is converged at deploy time to mode 0700, owner 1000:1000, and persists across container
recreation. bind.create_host_path: false prevents Compose from silently creating a missing or
mis-typed host path; deploy must establish the validated directory first.
GitHub App private key mount:
volumes: - /opt/dashboard/config/github-app.pem:/run/secrets/github-app.pem:roThe key is bind-mounted read-only into the container at /run/secrets/github-app.pem. The .env
sets DASHBOARD_GITHUB_APP_KEY_FILE=/run/secrets/github-app.pem so the app reads the key via file
path, never via an env-string fallback.
Fro Bot App private key mount:
volumes: - /opt/dashboard/config/fro-bot-app.pem:/run/secrets/fro-bot-app.pem:roThe wiki editor writes wiki pages as the Fro Bot App. The dashboard Environment secrets
DASHBOARD_FRO_BOT_APP_ID and DASHBOARD_FRO_BOT_APP_KEY are required; the deploy fails before any
SSH when either is missing or malformed. The key travels in the remote-deploy payload over SSH stdin
(field fro_bot_app_key) and is published as /opt/dashboard/config/fro-bot-app.pem (0600,
1000:1000). The .env sets DASHBOARD_FRO_BOT_APP_ID,
DASHBOARD_FRO_BOT_APP_KEY_FILE=/run/secrets/fro-bot-app.pem, and the constant
DASHBOARD_OPERATOR_ID=831617 (the operator’s public numeric GitHub user ID; no input exists for it).
DASHBOARD_GATEWAY_OPERATOR_ORIGIN stays unset.
The .env also carries the constant DASHBOARD_TRUSTED_PROXIES=172.18.0.0/16, the dashboard_default
Docker network subnet Caddy connects from; the dashboard reads X-Forwarded-For only from that peer
range so its rate limiter keys on real clients. It is not a deploy input or a secret. Re-read the subnet
with docker network inspect dashboard_default --format '{{(index .IPAM.Config 0).Subnet}}' (a droplet
rebuild can change it). A malformed value stops the dashboard at boot; a /0 prefix is refused.
The Fro Bot App is not the mrbro-bot App behind the repo secrets APPLICATION_ID /
APPLICATION_PRIVATE_KEY; never reuse those here.
Why a file mount instead of an env var for the App private key:
docker inspectexposes all env vars in plaintext to any process with Docker socket access.- Process listings (
/proc/<pid>/environ) expose env vars to other processes on the host. - Crash dumps and OOM reports may capture the process environment.
- A PEM key is long-lived key material — the blast radius of an env-var leak is permanent until the
key is revoked. File mounts with
0600permissions andread_only: truecontainers minimize exposure to the app process only.
Secret rotation
Section titled “Secret rotation”DASHBOARD_SSH_KEY. Replace the secret and update the authorized key on the droplet. No redeploy required for the running stack; the new key takes effect on the next SSH connection.DASHBOARD_DOMAIN. Update the secret and DNS record, then redeploy.DASHBOARD_GITHUB_APP_ID. Update the secret and redeploy.DASHBOARD_GITHUB_APP_KEY. Generate a new private key in the GitHub App settings, update theDASHBOARD_GITHUB_APP_KEYsecret, and redeploy. The deploy uploads the new PEM to/opt/dashboard/config/github-app.pem(0600) via SSH stdin. After confirming the new key works, revoke the old key in the GitHub App settings.DASHBOARD_FRO_BOT_APP_ID. Update the secret and redeploy.DASHBOARD_FRO_BOT_APP_KEY. Generate a new private key in the Fro Bot App settings, update the secret, and redeploy. The deploy uploads the new PEM to/opt/dashboard/config/fro-bot-app.pem(0600) via SSH stdin. After confirming the new key works, revoke the old key in the App settings.DASHBOARD_OAUTH_CLIENT_ID/DASHBOARD_OAUTH_CLIENT_SECRET. Rotate in the GitHub OAuth App settings, update the secrets, and redeploy. Existing sessions are invalidated.DASHBOARD_OPERATOR_LOGIN. Update the secret and redeploy.DASHBOARD_COOKIE_KEY. Rotating invalidates all existing sessions (users re-authenticate). Update the secret and redeploy.
Upgrade flow
Section titled “Upgrade flow”Dispatch Deploy Dashboard on main with the release’s CalVer version and, optionally, its
expected digest, or use infra dashboard deploy --image-version <calver> [--digest <sha256>].
The pre-gate job validates the inputs; approve the dashboard environment gate to deploy. After a
successful versioned deploy, review and merge the dashboard-pin audit PR to record the deployed
image in main. Merging that audit PR does not trigger another dashboard deployment.
Omit both inputs to deploy the committed Compose pin without opening an audit PR. Other dashboard runtime changes selected by the router still deploy the committed pin through the environment gate.
For rollback procedures, see docs/runbooks/dashboard-released-image-rollback.md.
| Command | Purpose |
|---|---|
infra dashboard status |
SSH docker compose ps → service/state/health rows (MCP-exposed) |
infra dashboard deploy [--image-version <calver>] [--digest <sha256>] |
Dispatch the Deploy Dashboard workflow (default), optionally with explicit image inputs, or --local; image flags are remote-only and --digest requires --image-version |
infra dashboard logs [service] [--tail N] |
Stream container logs via SSH (CI-guarded; emits a sensitive-data warning) |
infra status includes a dashboard row. With --json, the schema-versioned envelope’s apps array contains an entry with app: 'dashboard', typed observations, and facts.
Valid services for dashboard logs: dashboard, caddy.
Provisioning
Section titled “Provisioning”One-time: bun run provision:dashboard (root wrapper — loads the repo-root .env; --cwd apps/dashboard would miss it) creates the s-1vcpu-1gb droplet (image docker-20-04, region
nyc1, name dashboard), selects the SSH key by name (DASHBOARD_SSH_KEY_NAME, default
fro-bot-dashboard), waits for SSH, and pins both the domain and droplet-IP host keys into
.github/known_hosts (commit the result before the first CI deploy).
SSH auth during provisioning: when DASHBOARD_SSH_KEY is set, the script materializes it to a 0600
temp key file and pins it with -i + IdentitiesOnly=yes (no ssh-agent needed; cleaned up after).
When unset, it falls back to ssh-agent.
Operator UI (same-origin)
Section titled “Operator UI (same-origin)”The operator UI is enabled same-origin. The deploy sets DASHBOARD_OPERATOR_UI_ENABLED=true
and DASHBOARD_GATEWAY_OPERATOR_SESSION_ENABLED=true as static constants in the .env written by
buildEnvFileContents — no new secrets required. The SSE run-stream UI is reachable at
https://dashboard.fro.bot/operator/* behind the operator auth boundary. The gateway operator
session is the single auth authority; DASHBOARD_GATEWAY_OPERATOR_ORIGIN defaults to
https://dashboard.fro.bot and is not set explicitly.
The ratified browser-visible operator API origin is https://dashboard.fro.bot/operator/*. The
dashboard Caddy instance owns the /operator/* route and proxies it to the gateway operator
listener over the shared DigitalOcean VPC. This makes the operator API same-origin with the
dashboard UI.
Current state: The dashboard Caddy /operator/* route is live. The route is a handle
block before the dashboard:3000 catch-all in apps/dashboard/config/Caddyfile, targeting
{$GATEWAY_VPC_IP}:9300 over the VPC. The gateway operator listener is reachable from the
dashboard droplet via the shared DigitalOcean VPC (nyc1; gateway VPC IP 10.116.0.3, dashboard
VPC IP 10.116.0.5 — example values; actual IPs are set via GATEWAY_VPC_IP).
Docker network alias: The caddy service declares a Docker network alias dashboard.fro.bot
on the shared default network. The dashboard server validates every request by calling
https://dashboard.fro.bot/operator/session server-side. Without the alias, that call hairpins to
the droplet’s public IP — DigitalOcean has no NAT loopback, so it times out. The alias routes the
call to Caddy via Docker DNS; Caddy’s /operator/* handle block proxies it to the gateway VPC.
Both services are explicitly attached to the default network so existing DNS (dashboard:3000)
remains intact. The alias is in the committed docker-compose.yaml base file — not an override —
because the deploy removes any docker-compose.override.yaml on every run.
Route configuration:
handle /operator/* { reverse_proxy {$GATEWAY_VPC_IP}:9300 { flush_interval -1 header_up Host dashboard.fro.bot header_up X-Forwarded-Proto https }}flush_interval -1— disables response buffering for future SSE streams.header_up Host dashboard.fro.bot+header_up X-Forwarded-Proto https— satisfies the gateway daemon’s forwarded-header guard (X-Forwarded-Hostmust matchPUBLIC_ORIGINhost).{$GATEWAY_VPC_IP}— Caddy native env expansion; the dashboardcaddyservice receivesGATEWAY_VPC_IPfrom the deploy.env. Never use a literal IP in the Caddyfile.- The
handleblock must appear before thereverse_proxy dashboard:3000catch-all — Caddy compiles directives in fixed order, not source order; a bare catch-all sorts ahead and self-404s a matched route. - The dashboard app owns all other routing. The Caddyfile has two handles:
/operator/*→ gateway, everything else →dashboard:3000with no rewrite. A new public page needs only the app’sisPublicPathallowlist infro-bot/dashboard; client path routes, if ever added, need an app-side fallback, not a Caddyrewrite.
Required env var:
| Variable | Where set | Description |
|---|---|---|
GATEWAY_VPC_IP |
dashboard GitHub Environment + local .env |
Gateway droplet’s DigitalOcean VPC IP (e.g. 10.116.0.3). The dashboard caddy service expands {$GATEWAY_VPC_IP} from .env. Caddy fails to start if the var is set but unresolved. |
Seed GATEWAY_VPC_IP in the dashboard GitHub Environment before deploying the route. The
same-origin path only works after the gateway VPC bridge is live (see
apps/gateway/AGENTS.md Operator private path and
docs/plans/2026-06-18-003-feat-dashboard-operator-private-path-plan.md).
See docs/plans/2026-06-18-001-feat-dashboard-operator-same-origin-plan.md for the full decision
record.
OPERATOR PUSH
Section titled “OPERATOR PUSH”DASHBOARD_OPERATOR_PUSH_ENABLED is an independent, server-side dashboard flag, currently set to
true in the dashboard Environment. Only the exact string true is rendered as
DASHBOARD_OPERATOR_PUSH_ENABLED=true; absent, false, whitespace-variant, or malformed input is
omitted and disables push — silently, with no deploy error, unlike the gateway quartet which hard-fails
on a partial set.
The dashboard deploy accepts no VAPID material and no endpoint pointer. At runtime, the existing
consent flow and a trusted user gesture own subscription; the flag alone never auto-prompts or
auto-subscribes. That flow fetches the public key from the gateway’s authenticated
GET /operator/push/vapid-key route. A disabled or guard-denied gateway route is a disabled result.
Dashboard and gateway activation are independently actionable, but real end-to-end push requires both
sides and the fro-bot/dashboard#238 privacy-policy prerequisite.
Anti-patterns
Section titled “Anti-patterns”- Never put the GitHub App private key in an env var — use the file mount
(
/run/secrets/github-app.pem) andDASHBOARD_GITHUB_APP_KEY_FILE. Env vars are visible viadocker inspect, process listings, and crash dumps; long-lived key material must not travel that path. - Never
docker compose down -v— destroys thecaddy_datavolume (Caddy TLS certificates and ACME state). Do not use Compose teardown to recover a failed deploy. - Never remove
/opt/dashboard/datacasually — it contains persistent listener SQLite state and removing it destroys Inbox data across container recreation. Treat deletion as an explicit destructive storage reset. - Never add
--buildto the dashboard deploy — the deploy pulls the digest-pinned image fromghcr.io/fro-bot/dashboard; on-droplet builds are not supported and--no-buildis enforced in the deploy script. - Never put secret values in SSH argv — the deploy sends them in the framed SSH stdin payload. Shell metacharacters in secret values are rejected before any SSH connection is opened.
- Never skip
validateDashboardHost— it rejects--prefixed values and characters outside the allowed alphabet. SSH treats--prefixed hostnames as flags (including-oProxyCommand=). - Never wire
apps/dashboard/config/Caddyfileto proxy/operator/*togateway.fro.bot— that would route browser operator calls to the public gateway edge, not through the private VPC path. The dashboard→gateway path must use{$GATEWAY_VPC_IP}(config), never a literal IP or the public gateway hostname. See the operator same-origin target section above. - Never put the
/operator/*handle block after thedashboard:3000catch-all — Caddy compiles directives in fixed order; a barereverse_proxycatch-all sorts ahead and self-404s any route that follows it. The/operator/*block must be ahandleblock before the catch-all. - Never set
GATEWAY_VPC_IPto a literal IP in the Caddyfile — use{$GATEWAY_VPC_IP}and inject the value via thecaddyservice env. Hardcoded IPs break on droplet rebuild and make the config non-auditable. - Never move the
dashboard.fro.botalias to adocker-compose.override.yaml— the deploy removes/opt/dashboard/docker-compose.override.yamlon every deploy. The alias must live in the committed basedocker-compose.yaml. - Never set
DASHBOARD_GATEWAY_OPERATOR_SESSION_ENABLED=false— the gateway operator session is the single auth authority. Setting it false disables session validation and breaks the operator UI auth boundary.