Skip to content

Gateway Deploy Package

The gateway is the Fro Bot Discord client and workspace runner — a 3-service Docker Compose stack (gateway daemon, workspace executor, mitmproxy egress filter) deployed on a dedicated DigitalOcean droplet at gateway.fro.bot. The upstream source is fro-bot/agent, pinned in apps/gateway/upstream.json, which serves the operator contract version reported by /operator/health and must be paired with a dashboard release that declares the same contract. The workspace image’s Node, OpenCode, and Systematic versions come from that pinned upstream release.

With the operator surface enabled the daemon refuses to start unless GATEWAY_OPERATOR_TRUSTED_PROXIES is set; the deploy derives it from DASHBOARD_VPC_IP (see Operator Listener). The operator listener is enabled when GATEWAY_OPERATOR_BIND_HOST, GATEWAY_OPERATOR_BIND_PORT, and GATEWAY_OPERATOR_PUBLIC_ORIGIN are all set. Its auth config — GitHub OAuth client ID/secret, CSRF secret, and allowlist — is materialized as _FILE mounts. The operator surface provides a session/OAuth/CSRF/allowlist auth layer (the OAuth callback redirects to the validated return_to path after minting the session), an authenticated SSE run-stream, and server-side operator token retention, with metadata/repos.yaml redacted on operator surfaces. GET /operator/runs is a read-only, session-gated run index; POST /operator/runs launches runs and caps idempotencyKey at 256 characters (400 above that); POST /operator/dispatch exists; web tool-approval is supported.

The workspace runs OpenCode as uid 10001 (checkout ownership migrates one-way on first boot — see Rollback) and bakes OPENCODE_EXPERIMENTAL_DISABLE_FILEWATCHER=true. Do not override it back on. The only state that watcher refreshes is OpenCode’s per-directory cached current branch, read solely by GET /vcs and branch-mode GET /vcs/diff — neither of which the gateway mention loop calls, so the staleness is inert here. Two corrections to the upstream note: a stale value skews branch-mode diff results, not just a displayed branch name; and clearing this flag does not enable recursive working-tree watching, which is a separate opt-in (OPENCODE_EXPERIMENTAL_FILEWATCHER). If reused checkouts under /workspace/repos drift behind GitHub, that is a checkout-update policy problem — /clone returns 409 repo-exists and never fetches or switches branches — and no amount of file watching fixes it. /fro-bot dispatch needs the GitHub App to hold Actions: read and write; that permission is granted on the Fro Bot Agent App and accepted by the installations that use it.

The deploy consumes these upstream files: deploy/compose.yaml, deploy/gateway.Dockerfile, deploy/workspace.Dockerfile, deploy/mitmproxy/allowlist.py, and deploy/workspace-entrypoint.sh. The workspace has native always-on session tools. Mention runs are serialized by a per-channel run queue (one active run per Discord channel at a time), and /fro-bot force-release-lock lets an operator release a channel lock. Mention runs show a live-status message and typing indicator (GATEWAY_STATUS_MODE, defaults to live-status, no infra wiring needed). The daily_digest presence/announce event reuses the existing /v1/announce HMAC webhook ingress (GATEWAY_WEBHOOK_SECRET + GATEWAY_PRESENCE_CHANNEL_ID).

WORKSPACE_EGRESS_HOSTS is consumed by deploy/mitmproxy/allowlist.py (comma-separated exact hosts the sandboxed workspace may reach through the mitmproxy egress proxy; fail-closed if empty). The deploy emits WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai so the workspace OpenCode can reach the cliproxy endpoint. models.dev and models.opencode.ai are OpenCode model-catalog hosts: they stay allowlisted but are currently unused, because the workspace OpenCode V2 server runs with OPENCODE_DISABLE_MODELS_FETCH=1 and starts from its bundled catalog snapshot. The workspace executor exposes /healthz + /clone on :9100 and an OpenCode bearer proxy on :9200, all on the internal sandbox-net with no host ports, enabling both /fro-bot add-project repo cloning and the @fro-bot mention loop. GATEWAY_WEBHOOK_SECRET and GATEWAY_PRESENCE_CHANNEL_ID are readOptionalSecret (opt-in) in the upstream daemon — set both to enable the announce/presence webhook (see Announce/presence webhook below), or leave both unset to keep the gateway outbound-only. The deploy materializes GitHub App credentials (github-app-id, github-app-private-key) from GH_APP_ID / GH_APP_PRIVATE_KEY environment secrets.

The deploy script materializes secrets as files on the droplet (never via argv), bootstraps the mitmproxy CA on first run, brings up the Compose stack, and gates completion on Discord command registration. A secrets checksum written only after a fully successful deploy prevents silent stale-credentials states across retries.

Task Location Notes
Upstream pin apps/gateway/upstream.json fro-bot/agent ref — update to upgrade
Deploy script apps/gateway/src/deploy.ts Secrets materialization, compose up, registration poll
Provision droplet apps/gateway/server/provision-droplet.ts One-time. Refuses re-run on existing droplet without --force
CLI commands packages/cli/src/commands/gateway/ status, deploy, logs, backup, restore

The deploy runs in two stages: a CI build-images job builds and pushes the gateway and workspace Docker images to GHCR, then the deploy-gateway job SSHes to the droplet to pull those images and bring up the stack. The droplet never builds images — it only pulls prebuilt artifacts.

Runs on ubuntu-latest with packages: write permission (job-scoped only; the deploy job is package-read-only). Steps:

  1. Reads the pinned ref from apps/gateway/upstream.json.
  2. Checks out fro-bot/agent at that ref.
  3. Logs in to ghcr.io with GITHUB_TOKEN.
  4. Builds deploy/gateway.Dockerfile (context: upstream root) and pushes to ghcr.io/marcusrbrown/infra-gateway:<ref>-<github.sha>. Captures the pushed digest from build-push-action outputs. The tag is keyed on the infra commit as well as the upstream ref so repeated builds at the same upstream ref no longer overwrite each other’s tag.
  5. Builds deploy/workspace.Dockerfile (context: upstream root) with --build-arg OPENCODE_VERSION=<version> and pushes to ghcr.io/marcusrbrown/infra-workspace:<ref>-<github.sha>. Captures the pushed digest. The version is read from packages/runtime/src/shared/opencode-v2-pin.json in the checked-out upstream source (the Read workspace OpenCode V2 pin step), so it always matches the asset digests the Dockerfile verifies. The Dockerfile’s version-gate stage refuses any other OPENCODE_VERSION, including a V1 harness build, so the workspace image runs OpenCode V2 at the pinned version only; the gateway build takes no OpenCode arg.
  6. Exposes both digests as job outputs (gateway_digest, workspace_digest).

Both build steps pin a docker/setup-buildx-action builder to a fixed moby/buildkit image, set SOURCE_DATE_EPOCH as a build-arg, and export with rewrite-timestamp=true — this makes config/history/index timestamps and in-layer tar timestamps deterministic for a given upstream ref. Bit-reproducibility of a ref is not guaranteed. The upstream Dockerfiles install Alpine packages without version pins, because Alpine’s branch index drops older versions and an exact pin would fail the build. Rebuilding the same ref can therefore pick up newer packages and produce a different digest. The deployed image is always the digest this job pushed. To check whether a ref reproduced, compare the digests of two published tags sharing an upstream ref prefix (<ref>-<sha>). Layers map to instructions via the image config history (skipping empty_layer entries), so a drifting index names the instruction that caused it; equal size with a different digest is timestamp drift, not content.

Runs between build-images and the gateway environment approval gate, so results are visible on the run page when the operator decides whether to approve. Scans both images by the exact digest from build-images outputs, never by tag. Uses the official Trivy container pinned by digest (ghcr.io/aquasecurity/trivy:0.74.0@sha256:62b1e65e...) rather than aquasecurity/trivy-action, because that action had all but one of its version tags force-pushed to credential-stealing malware in March 2026 (GHSA-69fq-xp46-6x23) and is kept out of the trust path. Report-only — --exit-code 0 and job-level continue-on-error: true — so scan findings or a scanner outage never block a deploy. Table output goes to the job step summary. On main only, each step also emits SARIF (CRITICAL,HIGH, --ignore-unfixed) and uploads it with github/codeql-action/upload-sarif; alerts appear under Security → Code scanning with categories trivy-gateway and trivy-workspace. A missing or empty SARIF file produces a ::warning:: and skips the upload — it is not a clean scan. Permissions are contents: read + packages: read + security-events: write; the deploy.yaml caller job must grant the same security-events: write.

The deploy-gateway job declares needs: [build-images, scan-images, notify]. A build or push failure in build-images blocks the deploy entirely — the old stack stays live. scan-images is continue-on-error: true, so neither findings nor a scanner outage block the deploy. The ungated, main-only notify job runs after the build and scan, before the environment approval gate; the deploy condition ignores its result, so a skipped or failed notification does not block deployment. See Deploy-Gate Notifications for delivery verification and troubleshooting.

  1. Validate host — validateGatewayHost rejects --prefixed values and characters outside the allowed alphabet. SSH treats --prefixed hostnames as flags (including -oProxyCommand=); this check is mandatory before any SSH invocation.
  2. Checksum computation — SHA-256 of all secret values (including the compose.override.yaml content) is computed locally. The result is compared against /opt/gateway/.secrets-checksum on the droplet. If the checksums differ, --force-recreate is added to docker compose up so containers pick up the new secret files. If they match and --force-recreate was not explicitly passed, Compose converges the stack without forcing recreation: unchanged running containers can be reused, so a restart is not guaranteed. In both cases the deploy continues through its health and verification gates — there is no early exit. The checksum file lives outside the deploy clone so git clean -xfd on subsequent deploys doesn’t wipe it.
  3. Source materialization — SSH to the droplet; clone or fetch fro-bot/agent at the pinned ref from upstream.json, then git reset --hard && git clean -xfd to the pinned SHA. The upstream checkout is still needed for compose.yaml, init-certs.sh, and the Dockerfiles compose references — but the droplet does not build from them.
  4. Secret files — 11 required files written atomically under /opt/gateway/deploy/secrets/ (discord-token, discord-application-id, discord-guild-id, aws-access-key-id, aws-secret-access-key, s3-bucket, s3-region, github-app-id, github-app-private-key, workspace-opencode-token, workspace-opencode-auth) plus optional-shaped files (s3-endpoint, aws-session-token, discord-privileged-intents, workspace-opencode-url, gateway-binder-allowlist (always written, empty when GATEWAY_BINDER_ALLOWLIST is unset: compose bind-mounts it with create_host_path: false, so a missing file stops the stack), and gateway-trigger-role-id — written as 0-byte placeholders when unset, though GATEWAY_TRIGGER_ROLE_ID is enforced non-empty via REQUIRED_ENV_VARS, so the mention-loop authz gate fails closed rather than opening to all guild members). The /opt/gateway/deploy/.env carries OBJECT_STORE_HOSTS (computed from S3_BUCKET/S3_REGION/S3_ENDPOINT), WORKSPACE_OPENCODE_MODEL, WORKSPACE_OPENCODE_CONFIG — the latter is JSON-validated (via JSON.parse, not the shell-metachar guard) and $→$$-escaped so docker-compose ${VAR} interpolation passes the JSON through intact — WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai (v0.52.1+ requirement; consumed by deploy/mitmproxy/allowlist.py to allow the sandboxed workspace to reach the cliproxy OpenCode proxy and the OpenCode model catalog through mitmproxy; fail-closed if unset), and optionally WORKSPACE_OPENCODE_READY_TIMEOUT_MS (v0.53.1+; absent/empty uses the upstream default of 60000ms; non-numeric/zero/negative values are rejected before any SSH). Secret bytes enter via SSH stdin only — never via argv. Filenames on disk are kebab-case; the compose contract maps each to /run/secrets/<snake_case> and exposes via ${NAME}_FILE env vars. The GitHub App credentials come from the GH_APP_ID / GH_APP_PRIVATE_KEY environment secrets (GitHub reserves the GITHUB_ prefix for secret names) and materialize to the upstream-fixed github-app-id / github-app-private-key files. See docs/runbooks/discord-token-lifecycle.md for the full mapping table.
  5. Compose override — compose.override.yaml is written on every deploy. It pins gateway.image to ghcr.io/marcusrbrown/infra-gateway@<digest> and workspace.image to ghcr.io/marcusrbrown/infra-workspace@<digest> using the exact digests from the build-images job. It mounts the workspace-repos named volume at /workspace/repos on the workspace service (persists cloned repo checkouts across container recreation and daemon upgrades) and sets WORKSPACE_OPENCODE_READY_TIMEOUT_MS in the workspace container environment via Docker Compose interpolation from the .env file. It also sets mem_limit and memswap_limit on the workspace service to 1024 MiB, so the workspace is hard-capped with no swap (WORKSPACE_MEMORY_LIMIT_MIB in apps/gateway/src/deploy.ts, issue #1652). Sizing, in MiB as free -m reports them, from a measured run: with the workspace using 848 the host had 490 available, so a 1024 limit leaves about 314 available, above the 256 floor. Docker Compose deep-merges the override; the upstream build: stanzas remain present but are inert without --build. When announce is enabled, the override also adds the Caddy service and announce secret wiring (see Announce/presence webhook).
  6. CA bootstrap — init-certs.sh runs on the droplet (idempotent; skips if the CA already exists in the fro-bot_mitmproxy-certs named volume).
  7. Pull images — docker compose --project-directory /opt/gateway/deploy pull fetches the pinned GHCR images by digest. A missing or unpullable image errors here and does not fall back to an on-droplet build. Pull happens before any container disruption so that if the pull fails, gateway/caddy containers are never removed.
  8. Stale gateway-net cleanup — After pulling images, the deploy inspects the remote fro-bot_gateway-net Docker network. If it exists with the old 172.20.0.0/16 subnet (from a previous failed operator deploy), it is removed so Docker can recreate it with the correct 172.21.0.0/16 subnet. If the network is missing or already has 172.21.0.0/16, this step is a no-op. If the first removal attempt fails because containers are still attached (active endpoints — gateway and caddy services), the deploy removes those two containers directly via docker rm -f fro-bot-gateway-1 and docker rm -f fro-bot-caddy-1 (tolerating missing containers so recovery works when caddy is absent from the current compose config) and retries the removal. Note: docker compose stop alone is not sufficient — stopped containers can still keep active endpoints; only removing the containers releases their network endpoints. If container removal or the second removal fails, the deploy fails closed before docker compose up with an error message stating that gateway/caddy containers were removed. This step runs after docker compose pull and before docker compose up — images are already cached before any container disruption.
  9. Compose up — docker compose --project-directory /opt/gateway/deploy up -d --no-build --wait --wait-timeout 600 --remove-orphans (plus --force-recreate when the checksum changed or --force-recreate was passed). mitmproxy starts first, then the gateway daemon. --no-build is explicit — the droplet never builds images. The 600s wait budget outlasts the workspace healthcheck start_period (360s), which covers the one-time checkout ownership migration on first boot (default deadline 300s, WORKSPACE_MIGRATION_DEADLINE_MS). If the migration deadline is widened for a large workspace-repos volume, raise the start_period and this wait together.
  10. Registration poll — GET /applications/{app_id}/guilds/{guild_id}/commands on the Discord API. Polls Discord registration with ~90s default budget (10 attempts × (3s interval + 6s per-attempt timeout); defaults from apps/gateway/src/deploy.ts:344-368). 429 honors Retry-After and doesn’t count against the attempt budget (each 429 retry adds up to 60s; pathological all-429 ceiling ~11 min). 401/403/404 abort immediately with token-sanitized errors. 5xx retries.
  11. Running image digest verification — for each of gateway and workspace, the deploy reads the running container’s RepoDigests and asserts they match the CI-pushed digest. This confirms the droplet is running the GHCR artifact, not a stale or locally-built image. Throws if the digest does not match — deploy fails loudly. Manual verification:
    Terminal window
    docker inspect --format '{{json .RepoDigests}}' "$(docker inspect --format '{{.Image}}' "$(docker compose --project-directory /opt/gateway/deploy ps -q gateway)")"
    docker inspect --format '{{json .RepoDigests}}' "$(docker inspect --format '{{.Image}}' "$(docker compose --project-directory /opt/gateway/deploy ps -q workspace)")"
  12. Workspace memory limit readback — the deploy resolves the workspace container with docker compose ps -q workspace (exactly one ID, else it throws), then reads HostConfig.Memory and HostConfig.MemorySwap from docker inspect. Both must equal 1024 MiB in bytes; any other value (empty, 0, -1, a different limit, including one set by another compose file) throws before the checksum write with the expected and actual values. The read is single-shot with no retry. The stack stays running and the old checksum persists, so the next deploy recreates the containers. Success logs one line naming the limit.
  13. Checksum write — /opt/gateway/.secrets-checksum is written only after compose + registration + digest verification + memory limit readback all succeed. If any step fails mid-rotation, the old checksum persists and the next deploy force-recreates again.

A deploy that adds or changes the workspace limit changes the override, and so the checksum, which recreates every gateway service (gateway, mitmproxy, Caddy, workspace). In-flight runs are interrupted and must be cancelled by hand. Checkouts survive on the fro-bot_workspace-repos volume.

  • Monitoring — GitHub Actions deploy logs for the deploy run; gateway status for live service states; gateway logs <service> to stream container output.
  • Roll back a bad repo change that does not touch the pin — revert the commit and dispatch Deploy Gateway. The deploy is idempotent; containers either updated or didn’t.
  • Roll back an upstream pin bump — see Rollback is not pin reversal. Never revert only apps/gateway/upstream.json.
  • Roll back a bad secret rotation — restore the previous value in the gateway GitHub Environment and redeploy. The checksum-after-success invariant means a failed rotation leaves the old checksum in place, so the next deploy will force-recreate — but the credential value must be rolled back manually first.
  • A failed deploy does not destroy the live droplet — containers either updated or didn’t. Fix the underlying issue and retry; the deploy is safe to re-run.
  • Scaling or region change — provision a new droplet, update GATEWAY_HOST in the GitHub Environment, commit the updated .github/known_hosts, and trigger a deploy. Decommission the old droplet after verifying the new one is healthy.

The workspace limit is 1024 MiB (sizing in Deploy flow step 5). The Workspace memory block of gateway status shows the limit, current usage, max events (times usage hit the limit), OOM kills, restarts with the container start time, OOMKilled, and the OpenCode supervisor state (starting, ready, down, degraded, skipped, or unknown). The OOM kill count is since the container last started: it resets whenever the container is recreated or restarted. The block is display-only; the status line and exit code follow compose state, and a failed or timed-out probe reports the whole block as unknown. gateway status --json prints one line, {ok, services, workspace}, with the same data; the unified status --json envelope carries it as facts.workspace on the gateway entry, with a workspace-memory resource observation alongside.

  • opencode: skipped — the supervisor probe was deliberately not run, because the workspace has no limit or is within 64 MiB of it, so the probe cannot cause a kill. The display says which. unknown means the probe ran (or its outcome could not be established) and gave no usable answer.
  • degraded workspace — recover with cd /opt/gateway/deploy && docker compose restart workspace.
  • Run interrupted by an OOM kill — the run stays running and must be cancelled from its card.
  • Checkout left dirty by a kill — check it read-only with cd /opt/gateway/deploy && docker compose exec -u 10001 workspace git -c core.fsmonitor=false -C /workspace/repos/<owner>/<repo> status --porcelain. Remove files only with operator approval, also as UID 10001 inside the container.

Workspace checkout ownership migrates one-way: on first boot the workspace entrypoint hands every owner/repo checkout on the workspace-repos volume to uid 10001. An older workspace image runs git as root, which refuses to operate on those checkouts (“dubious ownership”). Reverting upstream.json and redeploying therefore leaves the old stack unable to use existing checkouts.

Rollback means restoring a pre-upgrade snapshot of the workspace-repos volume and redeploying the old gateway and workspace images together (they share the workspace bearer contract; a mismatch yields HTTP 401 / workspace-unavailable). Take the snapshot before any pin bump that crosses the uid migration. Upstream’s alternative is chowning the checkout trees back to root — see deploy/README.md in fro-bot/agent, “Workspace uid isolation” → Rollback. Never revert only the pin.

The ownership migration ran at the v0.118.2 cutover. Rolling back between releases at or after v0.118.2 is a paired pin reversal with the dashboard and needs no volume restore; the snapshot requirement applies only when a rollback crosses the migration. The target must also be at or after v0.130.1 (see Revert to a previous gateway and workspace pair).

Rolling back to a release without durable operator sessions leaves the session records in operator-web/sessions/ in place, and the older daemon cannot revoke them on logout. To invalidate them, rotate GATEWAY_OPERATOR_CSRF_SECRET or delete operator-web/sessions/*.

Revert to a previous gateway and workspace pair

Section titled “Revert to a previous gateway and workspace pair”

The gateway and workspace run OpenCode V2 only. apps/gateway/upstream.json holds the current pin, and the workspace image carries the OpenCode version in that release’s opencode-v2-pin.json. There is no OpenCode generation to select or fall back to: fro-bot/agent removed the V1 path in v0.139.0, the workspace Dockerfile refuses any other OPENCODE_VERSION, and the gateway rejects a run unless the workspace /readyz reports an OpenCode V2 version. A workspace that reports no version or an older one is not ready.

The gateway and workspace roll back together, to a pair that can run V2. v0.130.1 is the floor: it is the oldest release that provisions provider credentials into each V2 spawn, and on v0.130.0 every model call fails with 401. Releases before v0.130.1 have no working V2 path and are not rollback targets. Before reverting, confirm no run is in flight: recreating the containers interrupts runs, which must then be cancelled by hand. V2 keeps no OpenCode session state across workspace restarts (it recreates its database on every spawn), so a mention thread loses OpenCode context at the revert.

To revert, use either path. Both ship the gateway and workspace together, and in both the pin and the images must describe the same upstream ref.

  • Previous images (fast, byte-identical). gateway deploy --local takes the images only from GATEWAY_IMAGE_DIGEST and WORKSPACE_IMAGE_DIGEST, but takes the upstream ref from apps/gateway/upstream.json in the checkout it runs from, and resets the droplet’s /opt/gateway clone (the source of the Compose files and scripts) to that ref (resolveUpstreamPin and the source-materialization phase in apps/gateway/src/deploy.ts). Nothing checks the digests against the ref, and the command has no flag for either, so the operator makes them agree. In order:
    1. Provenance of the pair. In the last successful deploy-gateway run before the bump, take the gateway and workspace digests from its build-images outputs (that run’s digest readback verified them on the droplet) and the tags it pushed, <ref>-<sha>: <ref> is the upstream ref, <sha> is the infra commit the run built from.
    2. Deploy files for the same ref. Check out infra commit <sha>, so upstream.json and the deploy code match the pair, and run the deploy from that checkout (the root .env is needed there). Editing only upstream.json at the current commit leaves the current deploy code, which has not been checked against the older ref.
    3. Deploy. With GATEWAY_IMAGE_DIGEST and WORKSPACE_IMAGE_DIGEST set to the pair, run bunx @marcusrbrown/infra gateway deploy --local. No rebuild is needed.
    4. Verify after the deploy. The live RepoDigests must equal the recorded pair (command in Droplet: deploy-gateway job step 11). The deploy’s own readback checks the digests it was given, not where they came from, so step 1 is the only provenance check.
  • Revert the commit. Reverting the pin-bump commit restores upstream.json to the previous ref, and build-images rebuilds that ref with the OPENCODE_VERSION from its own pin file, so the pin and images agree by construction. The digests may differ from the previous pair (see CI: build-images job). A rebuild of a ref older than the current pin has not been verified against the current deploy code; prefer the recorded pair.

No workspace-repos volume restore is needed while both sides stay at or after v0.118.2 (see Rollback is not pin reversal). The previous run’s digests must serve the operator contract the pinned dashboard requires.

Command Purpose
bunx @marcusrbrown/infra gateway status SSH to droplet, run docker compose ps, show service states + ages + healthchecks, then a workspace memory block (limit, usage, OOM kills, restarts, OpenCode state). --json prints one {ok, services, workspace} line instead.
bunx @marcusrbrown/infra gateway deploy Trigger the deploy workflow via gh workflow run (remote, default). --local runs the deploy script directly (requires SSH_AUTH_SOCK). --dry-run prints the plan without side effects.
bunx @marcusrbrown/infra gateway logs <service> [--tail N] Stream docker compose logs from the droplet. Services: gateway, workspace, mitmproxy. --tail defaults to 50. Logs surface unredacted; operator-only via SSH boundary. CI/headless contexts require explicit --allow-ci to discourage scripted log capture.
bunx @marcusrbrown/infra gateway backup [--output FILE] [--include-ca] Pull the mitmproxy CA cert + key from the fro-bot_mitmproxy-certs named volume as a tarball (errors if the volume does not exist). Local file created with mode 0600 via O_EXCL|O_CREAT (no chmod race). --include-ca is required (only CA backup is currently implemented).
bunx @marcusrbrown/infra gateway restore --input FILE [--include-ca] Validate the tarball locally (must contain exactly mitmproxy-ca-cert.pem + mitmproxy-ca.pem; otherwise rejected before any remote mutation), upload to an unguessable mktemp path, extract into the named volume, restart mitmproxy + gateway, byte-equal confirm restored cert + key match the input archive.

Prerequisites:

  • DIGITALOCEAN_ACCESS_TOKEN and GATEWAY_HOST in .env; doctl auth init run locally. GATEWAY_HOST is the FQDN used for host-key pinning (e.g. gateway.fro.bot) — DNS does not need to point at the new droplet yet.
  • Discord application created at https://discord.com/developers/applications with bot scope; token + application ID + guild ID captured
  • S3 or R2 bucket created; AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, S3_BUCKET, S3_REGION captured (add S3_ENDPOINT for R2/MinIO); IAM identity granted the actions in S3 State-Bucket IAM Actions
  • gateway GitHub Environment created with required reviewer set
  • All required secrets seeded in the gateway GitHub Environment (see Required Secrets below)

Run:

Terminal window
bun run provision:gateway

(Root wrapper — loads the repo-root .env; --cwd apps/gateway would miss it.)

The script will:

  1. Confirm doctl is authenticated
  2. Reject if a gateway droplet already exists (aborts; --force to override)
  3. Create an s-1vcpu-2gb droplet in nyc1, tagged gateway
  4. Append unhashed domain host keys and hashed IP host keys to .github/known_hosts (ssh-keyscan <domain> + ssh-keyscan -H <ip>)
  5. Print operator setup steps for finalizing the GitHub Environment

SSH auth during provisioning: when GATEWAY_SSH_KEY is set, the script materializes it to a 0600 temp key file and pins it with -i + IdentitiesOnly=yes (no ssh-agent needed; cleaned up after). When unset, it falls back to ssh-agent.

After provisioning: commit the updated .github/known_hosts.

The fro-bot-gateway identity (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) owns the state bucket: it is the only principal with access, and the bucket has no bucket policy. The grant is the managed policy fro-bot-gateway-s3-access, which is the source of truth for the action set below. It is scoped to the whole bucket rather than a prefix, so it covers the run-state prefix and the operator session keys under operator-web/ alike. Grant it before seeding the secrets in Required Secrets.

Object-level (resource arn:aws:s3:::<bucket>/*):

Action Why
s3:GetObject Read run-state objects
s3:PutObject Write run-state objects
s3:PutObjectTagging Required alongside s3:PutObject — run-state writes send an x-amz-tagging header (object-type=run-state), and AWS requires both actions to accept a tagged PutObject. A missing grant rejects every createRun with AccessDenied on both the Discord @fro-bot mention path and the dashboard launch path — this is not a dashboard-only dependency. Call sites in upstream fro-bot/agent: packages/runtime/src/coordination/run-state.ts (create, transition) and packages/runtime/src/coordination/heartbeat.ts.
s3:DeleteObject Remove run-state objects
s3:GetObjectTagging Read an object’s tags, so an operator can confirm the object-type=run-state tag landed with the same credential
s3:GetObjectVersion Read a specific object version; harmless while the bucket is unversioned

Bucket-level (resource arn:aws:s3:::<bucket>):

Action Why
s3:ListBucket List run-state and operator session keys
s3:GetBucketLocation Resolve the bucket’s region

Durable operator sessions keep one signed record per session under operator-web/sessions/ at the bucket root, and the startup self-test writes under operator-web/_self-test/. These writes are untagged. If a narrower grant ever leaves these keys out, the gateway still starts but operator logins return 503 until the grant is fixed and the gateway is restarted. Keep the Action’s S3 principal on its own bucket so nothing else can write session records.

Lifecycle configuration is operator-owned, not provisioning-owned. A bucket lifecycle rule is bucket-level AWS state, independent of the droplet — destroying and rebuilding the droplet never touches it, and it is set-once configuration rather than per-provision or per-deploy state. apps/gateway/server/provision-droplet.ts does not apply it. The fro-bot-gateway runtime identity intentionally does not hold s3:GetLifecycleConfiguration / s3:PutLifecycleConfiguration, and should not be granted them — the only candidate for holding that bucket-administration power permanently is a credential that lives on the droplet and is used by the running daemon, which is a bad trade for automating a change that happens approximately once.

ensureRunStateLifecycleRule (exported from apps/gateway/server/provision-droplet.ts) remains available as a correct, idempotent helper for an operator to run by hand with a privileged, provisioning-scoped credential — never the runtime fro-bot-gateway identity.

The rule’s canonical shape — RUN_STATE_LIFECYCLE_RULE in apps/gateway/server/provision-droplet.ts is the source of truth, referenced here rather than duplicated:

  • ID: run-state-30d-expiration
  • Filter: Tag {Key: object-type, Value: run-state}
  • Status: Enabled
  • Expiration: Days: 30

Why the filter must stay tag-scoped, not prefix-scoped: the same <prefix> also holds durable Discord channel/repo bindings (bindings/repo.json and bindings/by-channel/<id>.json) that must never expire. A prefix-wide rule would delete them and silently unbind every repository. The tag filter — matching only objects the daemon explicitly tags object-type=run-state — is what makes a retention rule safe to have at all.

Operator procedure — read first, merge, then put. PutBucketLifecycleConfiguration REPLACES the bucket’s entire lifecycle configuration. A blind PUT silently destroys any other rules present (e.g. a NoncurrentVersionExpiration rule covering the IAM probe prefix, see the versioning caveat below). Never PUT without reading first:

Terminal window
# 1. Read the current configuration. NoSuchLifecycleConfiguration means there is
# nothing to merge — proceed straight to writing just the run-state rule.
aws s3api get-bucket-lifecycle-configuration --bucket <bucket>
# 2. Merge: keep every existing rule whose ID is not run-state-30d-expiration,
# then add/replace the run-state-30d-expiration rule with the canonical shape above.
# 3. Put the merged document (never a bare/blind PUT of just the new rule).
aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://merged.json

Or invoke ensureRunStateLifecycleRule(env) directly with a privileged credential — it already performs this read-check-write-readback sequence correctly and merges rather than replaces.

IAM policy management is out-of-band; tagged-write capability is checked at deploy time. No code here manages or inspects the fro-bot-gateway identity’s IAM policy. The gateway deploy verifies the runtime identity’s tagged-write and probe-cleanup capabilities through the fail-closed preflight described below; it does not audit every grant in the tables above. For a manual capability check, use a tagged write, since an untagged write does not exercise s3:PutObjectTagging:

Terminal window
aws s3api put-object --bucket <bucket> --key <prefix>/_iam-probe/<uuid>.json \
--body /dev/null --tagging 'object-type=run-state'
aws s3api delete-object --bucket <bucket> --key <prefix>/_iam-probe/<uuid>.json

The probe key must sit inside the same resource scope (<bucket>/<prefix>/*) the real run-state objects use — a probe outside a narrowly scoped grant’s prefix produces a false negative. Cleanup needs s3:DeleteObject.

The gateway deploy runs this same probe automatically as a fail-closed preflight (assertS3TaggedWriteCapability in apps/gateway/src/deploy.ts), before any remote mutation and before containers are replaced. It is skipped under --dry-run, and skipped with an explicit logged reason when S3_ENDPOINT is set, because upstream suppresses object tagging entirely for non-AWS endpoints.

Versioning caveat: on a versioning-enabled bucket, deleting the probe writes a delete marker and retains a noncurrent version, so one small object accumulates per deploy. The runtime identity cannot read bucket versioning state (no s3:GetBucketVersioning), so neither the probe nor the deploy can detect this. If the state bucket is versioned, cover the probe prefix with a NoncurrentVersionExpiration lifecycle rule rather than granting s3:DeleteObjectVersion to the runtime identity.

Secret Required Description
GATEWAY_SSH_KEY ✓ Ed25519 private key for the gateway.fro.bot droplet
DISCORD_TOKEN ✓ Discord bot token
DISCORD_APPLICATION_ID ✓ Discord application ID
DISCORD_GUILD_ID ✓ Discord guild (server) ID
AWS_ACCESS_KEY_ID ✓ S3/R2 access key
AWS_SECRET_ACCESS_KEY ✓ S3/R2 secret key
S3_BUCKET ✓ Bucket name
S3_REGION ✓ Bucket region
GATEWAY_HOST ✓ FQDN or IP of the droplet
GH_APP_ID ✓ GitHub App ID for /fro-bot add-project repo access (materializes to github-app-id)
GH_APP_PRIVATE_KEY ✓ GitHub App private key PEM (materializes to github-app-private-key)
WORKSPACE_OPENCODE_TOKEN ✓ Internal shared bearer between the gateway and the workspace :9200 OpenCode proxy (random value; fail-closed if empty)
WORKSPACE_OPENCODE_AUTH ✓ OpenCode provider auth.json for the workspace — a dedicated scoped cliproxy key, not the repo’s OPENCODE_AUTH_JSON (separate revocation + blast radius)
WORKSPACE_OPENCODE_MODEL ✓ OpenCode model id for the mention loop (e.g. openai/gpt-5.5). MUST be a real cliproxy /v1/models catalog id — an unlisted id (e.g. gpt-5.5-fast) 502s and OpenCode retries silently, yielding an empty run. Written to .env
WORKSPACE_OPENCODE_CONFIG ✓ OpenCode provider/baseURL config JSON routing through cliproxy.fro.bot/v1; JSON-validated, $-escaped, written to .env
GATEWAY_TRIGGER_ROLE_ID ✓ Discord role ID allowed to trigger the @fro-bot mention loop — required (fail-closed); empty would open LLM execution to every guild member
S3_ENDPOINT — Custom endpoint URL (R2, MinIO, etc.)
WORKSPACE_OPENCODE_URL — Override the workspace OpenCode proxy URL (default http://workspace:9200)
OBJECT_STORE_HOSTS — Comma-separated hostnames the mitmproxy egress filter allows through to S3
WORKSPACE_OPENCODE_READY_TIMEOUT_MS — Workspace OpenCode supervisor ready timeout in ms (1–600000; default: 60000). Absent or empty uses the upstream default. Whitespace-only, non-numeric, zero, negative, non-integer, and above-max values are rejected before any SSH. Written to .env and wired into the workspace container environment via the compose override.
DISCORD_PRIVILEGED_INTENTS — Opt-in privileged intents (e.g. MessageContent); materializes to discord-privileged-intents, empty = baseline intents
GATEWAY_WEBHOOK_SECRET opt-in† HMAC signing key for the announce webhook — strong random value; materialized via SSH stdin, never argv. Set together with GATEWAY_PRESENCE_CHANNEL_ID to enable the announce endpoint; leave both unset to keep the gateway outbound-only. Setting exactly one fails the deploy before any SSH.
GATEWAY_PRESENCE_CHANNEL_ID opt-in† Discord channel ID where the daemon posts presence embeds as the Fro Bot user. Set together with GATEWAY_WEBHOOK_SECRET.
GATEWAY_BINDER_ALLOWLIST — Discord user IDs allowed to run /add-project. A gateway Environment variable (read via vars.), not a secret. One ID per line; commas and spaces also work, blank lines and full-line # comments are ignored. Each entry must be a Discord snowflake (17–20 digits); a malformed entry fails the deploy before any SSH (--dry-run included) and names the bad line. Written on every deploy to gateway-binder-allowlist, normalized to one ID per line; unset or empty writes an empty file and /add-project then refuses everyone. A change alters the secrets checksum, so the gateway is recreated.
GATEWAY_OPERATOR_BIND_HOST opt-in‡ Gateway-net IPv4 address for the operator listener bind (e.g. 172.21.0.2). Rejects 0.0.0.0, loopback, sandbox-net (10.*), 172.20.* (the known conflicted Docker subnet from the failed operator deploy), and IPv6.
GATEWAY_OPERATOR_BIND_PORT opt-in‡ Port for the operator listener (e.g. 9300). Must be a positive integer in [1, 65535].
GATEWAY_OPERATOR_PUBLIC_ORIGIN opt-in‡ HTTPS origin for the browser-visible operator API. The ratified value is https://dashboard.fro.bot — set this for production use. Must be a bare HTTPS origin (no path, query, hash, credentials, or non-default port).
GATEWAY_VPC_IP opt-in§ Gateway droplet’s DigitalOcean VPC IP (e.g. 10.116.0.3). Required for the VPC-IP-scoped operator port publish and DOCKER-USER rule. All-or-none with DASHBOARD_VPC_IP and the operator listener trio.
DASHBOARD_VPC_IP opt-in§ Dashboard droplet’s DigitalOcean VPC IP (e.g. 10.116.0.5). Required whenever the operator listener is enabled: it is the DOCKER-USER source restriction and the derived GATEWAY_OPERATOR_TRUSTED_PROXIES value. Must be an exact IPv4 address (not unspecified, multicast, CIDR, hostname, or IPv6). Deploy fails before any SSH if absent or invalid.
GATEWAY_OPERATOR_TRUSTED_PROXIES derived Not an input. The deploy sets it in the gateway service environment of compose.override.yaml to DASHBOARD_VPC_IP (plain env entry, not a _FILE secret) whenever the operator listener is enabled; omitted otherwise. The daemon refuses to start without it.
DIGITALOCEAN_ACCESS_TOKEN provisioning-only DigitalOcean API token used by provision-droplet.ts to create the gateway-operator-fw Cloud Firewall. Not a deploy secret — not required in the gateway GitHub Environment.
GATEWAY_IMAGE_DIGEST CI-injected sha256:<digest> of the ghcr.io/marcusrbrown/infra-gateway image pushed by the build-images job. Threaded from needs.build-images.outputs.gateway_digest. Required for the deploy to pin and verify the running image. For a local/break-glass deploy, supply manually (see Break-glass runbook).
WORKSPACE_IMAGE_DIGEST CI-injected sha256:<digest> of the ghcr.io/marcusrbrown/infra-workspace image pushed by the build-images job. Threaded from needs.build-images.outputs.workspace_digest. Required for the deploy to pin and verify the running image. For a local/break-glass deploy, supply manually.

†Both-or-neither: set both to enable the announce/presence webhook; set neither to disable. Setting exactly one is an error — the deploy fails fast with a message naming the missing input, before any SSH connection is made.

‡All-or-none: set all three to enable the operator listener; set none to disable. Setting one or two is an error — the deploy fails fast before any SSH. See Operator Listener for bind host/port/origin constraints.

§Required with the operator listener trio: GATEWAY_VPC_IP and DASHBOARD_VPC_IP must both be set whenever the listener trio is set; the deploy fails before any SSH otherwise. With the listener trio unset, both are ignored. DIGITALOCEAN_ACCESS_TOKEN is a provisioning-only concern (used locally by provision-droplet.ts to create the Cloud Firewall) — it is not part of this all-or-none group and is not required in the gateway GitHub Environment. See Operator private path for the full bridge topology.

The workspace runs an OpenCode Go model only through cliproxy.fro.bot. Three values select it:

  • WORKSPACE_OPENCODE_MODEL=opencode-go/<id> — <id> is a bare id listed by cliproxy /v1/models with owned_by: opencode-go.
  • WORKSPACE_OPENCODE_CONFIG — carries provider["opencode-go"].options.baseURL set to https://cliproxy.fro.bot/v1.
  • WORKSPACE_OPENCODE_AUTH — carries an opencode-go entry holding the dedicated scoped cliproxy key.

The deploy guard checks the base URL of the provider named by the model prefix, so it rejects a Go model whose base URL points at OpenCode directly or whose opencode-go provider block is missing. The egress allowlist gains no OpenCode inference host. apps/gateway/src/deploy.test.ts pins this behavior.

The gateway and workspace images are published to:

  • ghcr.io/marcusrbrown/infra-gateway — built from fro-bot/agent deploy/gateway.Dockerfile
  • ghcr.io/marcusrbrown/infra-workspace — built from fro-bot/agent deploy/workspace.Dockerfile

Both packages are public. The upstream Dockerfiles were audited before first publish: all secrets are runtime bind-mounts into /run/secrets/... — no ARG/ENV/COPY of secret material is baked into the image layers. The source (fro-bot/agent) is itself public. Public visibility means the droplet pulls with no authentication required.

Tags (:<ref>) are pushed for human readability. The deploy pins and verifies by digest (@sha256:<digest>), not by tag — GHCR tags are mutable, so tag-based verification would be circular. Digest pull is immutable and makes the running-image check a true identity assertion.

The build-images CI job holds packages: write permission (job-scoped only). The deploy-gateway job is package-read-only.

The deploy depends on CI and GHCR availability. If CI or GHCR is unavailable during an incident and the droplet needs a redeploy:

  1. Build and push from a workstation (which has ample RAM — this is the off-droplet build done by hand):

    Terminal window
    # Check out fro-bot/agent at the pinned ref
    REF=$(jq -r .ref apps/gateway/upstream.json)
    git clone --depth 1 --branch "$REF" https://github.com/fro-bot/agent.git /tmp/fro-bot-agent
    # Log in to GHCR
    echo "$GITHUB_TOKEN" | docker login ghcr.io -u <your-github-username> --password-stdin
    # Build and push gateway image; capture the digest
    GATEWAY_DIGEST=$(docker buildx build \
    --platform linux/amd64 \
    --file /tmp/fro-bot-agent/deploy/gateway.Dockerfile \
    --push \
    --iidfile /tmp/gateway-iid.txt \
    --metadata-file /tmp/gateway-meta.json \
    /tmp/fro-bot-agent \
    && jq -r '."containerimage.digest"' /tmp/gateway-meta.json)
    # Build and push workspace image; capture the digest
    WORKSPACE_DIGEST=$(docker buildx build \
    --platform linux/amd64 \
    --file /tmp/fro-bot-agent/deploy/workspace.Dockerfile \
    --push \
    --metadata-file /tmp/workspace-meta.json \
    /tmp/fro-bot-agent \
    && jq -r '."containerimage.digest"' /tmp/workspace-meta.json)
  2. Run the local deploy with the digests you just captured:

    Terminal window
    GATEWAY_IMAGE_DIGEST="$GATEWAY_DIGEST" \
    WORKSPACE_IMAGE_DIGEST="$WORKSPACE_DIGEST" \
    bunx @marcusrbrown/infra gateway deploy --local

Never run docker compose up --build on the droplet. The compose.override.yaml image pins make GHCR the source of truth regardless, but building on the 1vCPU/2GB droplet exhausts RAM and thrashes swap — this is the failure mode the off-droplet build flow exists to prevent. The on-droplet build fallback is intentionally removed; the workstation build above is the correct emergency path.

The deploy depends on CI + GHCR availability + pull working. This is a deliberate trade: the on-droplet build is the hazard being removed, so keeping it as a fallback would keep the hazard.

The gateway authenticates to GitHub as a public App (owned by the fro-bot account, contents: read only) to clone repos for /fro-bot add-project owner/repo. Installation tokens are scoped per-installation, so the App must be installed on every account add-project targets — that is why it is public, not private. The App ID + PEM live only in the gateway GitHub Environment and on the droplet; they never enter the repo. Naming: the Environment secrets are GH_APP_ID / GH_APP_PRIVATE_KEY (GitHub rejects GITHUB_-prefixed names); they materialize to the upstream-fixed files github-app-id / github-app-private-key, which the daemon reads as its in-container GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY. Rotation/revocation: generate a new private key in the App settings, replace GH_APP_PRIVATE_KEY in the gateway Environment, re-run the deploy (re-materializes the file), then delete the old key. To contain a compromise, revoke the App installation on affected accounts (or suspend the App) and rotate the PEM. The App identity (name, logo, docs page, install-URL default) is owned by fro-bot/agent, not this repo. See docs/runbooks/github-app-key-lifecycle.md for the full rotation/revocation runbook.

The workspace executor runs OpenCode against cliproxy.fro.bot when an authorized user @-mentions the bot. Trust model: the workspace runs on sandbox-net (internal: true — no direct external egress; all outbound forced through the mitmproxy allowlist), so a prompt-injection payload in a Discord message or cloned repo cannot reach an arbitrary destination. GATEWAY_TRIGGER_ROLE_ID bounds who can trigger a run (and thus spend); it is enforced non-empty at deploy time (REQUIRED_ENV_VARS in deploy.ts) even though the upstream daemon treats it as readOptionalSecret — this repo’s deploy is stricter (fail-closed). WORKSPACE_OPENCODE_AUTH is a dedicated scoped cliproxy key so it can be revoked independently of CI’s OPENCODE_AUTH_JSON. GATEWAY_WEBHOOK_SECRET and GATEWAY_PRESENCE_CHANNEL_ID are opt-in — set both in the gateway GitHub Environment to enable the announce/presence webhook (see Announce/presence webhook below). Honest limit: an authorized role-holder is still an untrusted-content source — containment bounds blast radius, it does not prevent a determined authorized user from misusing the agent within the allowed surface. Verify post-deploy with the release gates, and with a real authorized mention when the Discord pass runs (Discord command registration alone does not prove the workspace works); if broken, follow Rollback is not pin reversal — do not just revert upstream.json.

The deploy emits WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai. cliproxy.fro.bot is the LLM proxy the workspace routes all model traffic through; without it the @-mention loop gets no model responses. models.dev and models.opencode.ai are OpenCode model-catalog hosts, kept allowlisted but currently unused: the workspace OpenCode V2 server runs with OPENCODE_DISABLE_MODELS_FETCH=1 (upstream apps/workspace-agent/src/opencode-server-v2.ts), so it never fetches the catalog and uses its bundled snapshot. The /fro-bot add-project clone path is unaffected (it does not invoke OpenCode).

After deploying a new daemon version, run the checks below. The acceptance procedure for a new upstream ref is the release gates: they run on every new upstream ref and need no Discord action, and the Discord checks form a separate pass that runs only when it is triggered. The steps below add what the gates do not cover.

Always, in order — each must pass before the next:

  1. bunx @marcusrbrown/infra gateway status — core services (gateway, workspace, mitmproxy) must show healthy/running; on announce-enabled stacks, caddy must also appear (running or healthy — Caddy has no healthcheck, so running/n-a is normal).
  2. bunx @marcusrbrown/infra gateway logs workspace --tail 100 — confirm the workspace container started without errors (look for the Hono server listening on :9100 and :9200).
  3. Release gates — run the automated gates per the runbook. A dashboard-launched run proves workspace execution without Discord, and gate 5 covers approvals: a rejected approval lets the run continue, and the command does not run.
  4. workspace-repos volume — after a gate run on a repo, run bunx @marcusrbrown/infra gateway logs workspace --tail 50 and confirm the workspace did not re-clone the repo from scratch (it should use the existing checkout from the workspace-repos volume). On first deploy, the volume is auto-created and the repo is cloned fresh — subsequent runs reuse it.

Only when the Discord pass runs (or, for the clone smoke, after changing the GitHub App credentials):

  1. /fro-bot add-project clone smoke — a Discord slash command, not part of the gates. Run it against a repo the GitHub App is installed on; confirm the clone completes without error. This exercises the GitHub App credentials and the workspace /clone endpoint end-to-end.
  2. Authorized @fro-bot mention round-trip — DP.1 of the Discord pass, from an account holding GATEWAY_TRIGGER_ROLE_ID. It is the only proof that Discord intake and replies work; the Approve/Deny embeds and their attribution are checked in the same pass (gate 4). Discord command registration (step 10 of the droplet deploy flow) does not prove the workspace works — it only proves the daemon registered slash commands.

@mention-loop behavior:

  • A mention from an authorized user (holding GATEWAY_TRIGGER_ROLE_ID) triggers an OpenCode run in the workspace.
  • If the run needs a tool that requires approval, the gateway posts an Approve/Deny embed in the run thread. The run pauses until a button-click or the 13-minute deadline fires (fail-closed).
  • Only authorized users can approve. Unauthorized approval attempts are silently ignored.
  • If the gateway or workspace restarts while a prompt is open, the prompt is abandoned — the run does not resume.
  • Cloned repos live in the workspace-repos named volume at /workspace/repos. The volume persists repo contents across container recreation and daemon upgrades. Missing repos are re-cloned automatically on the next mention. Never run docker compose down -v — it destroys all cloned repos. Do not put secrets in repo contents — authorized runs may access the active workspace/repo surface. To fully clear a cloned repo, delete the volume and let the next mention re-clone it.
  • WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai is required for the workspace to reach the LLM proxy; the two catalog hosts stay allowlisted but are currently unused (see above). Without cliproxy.fro.bot, the mention loop gets no model responses.
  • The workspace bash approval policy is owned by deploy code (WORKSPACE_PERMISSION_POLICY in apps/gateway/src/deploy.ts) and injected into WORKSPACE_OPENCODE_CONFIG at deploy time, overwriting any permission block the secret carries. Destructive commands — file deletion (rm, rmdir), force-push/hard-reset/clean, sudo, chmod/chown, exfil-shaped curl/wget, and package installs — route through the Discord Approve/Deny gate; ordinary commands run autonomously.

The daemon’s POST /v1/announce endpoint is opt-in. When enabled, it accepts HMAC-signed requests and posts a Discord embed as the Fro Bot user to the configured presence channel. The intended caller is the fro-bot/.github control plane (separate work, not in this repo).

Set both GATEWAY_WEBHOOK_SECRET (a strong random HMAC key) and GATEWAY_PRESENCE_CHANNEL_ID (the target Discord channel ID) in the gateway GitHub Environment, then trigger a deploy. The deploy materializes both as secret files on the droplet and writes a compose.override.yaml that:

  • Adds a Caddy reverse proxy service publishing :80/:443 on gateway-net, terminating TLS for gateway.fro.bot via Let’s Encrypt auto-cert.
  • Path-scopes Caddy to /v1/announce and (when operator is enabled) /operator/* — all other paths return 404.
  • Wires the daemon’s announce *_FILE env entries and secret mounts via the override’s gateway service block.

The override is a working-tree file re-materialized on every deploy when the inputs are present. The override contents are included in the secrets checksum, so toggling announce on or off forces --force-recreate on the next deploy.

  • Auth: the daemon verifies the HMAC signature, enforces a replay cache, and applies a rate limiter. Caddy adds TLS termination.
  • No IP allowlist: GitHub Actions egress ranges are too dynamic to pin reliably. HMAC + TLS is the full auth boundary.
  • Secret materialization: GATEWAY_WEBHOOK_SECRET is written to the droplet via SSH stdin only — never via argv.
  • Path isolation: Caddy uses a handle /v1/announce { ... } block with a catch-all handle { respond 404 }. The catch-all does not shadow ACME challenge paths (Caddy uses TLS-ALPN-01 on :443 by default, which does not touch HTTP path routing).

Remove both GATEWAY_WEBHOOK_SECRET and GATEWAY_PRESENCE_CHANNEL_ID from the gateway GitHub Environment and redeploy. The compose.override.yaml is always rewritten on deploy — when announce is disabled, the override carries only the image digest pins (no Caddy/announce sections). git clean -xfd removes the Caddyfile (a working-tree file written only when announce is enabled); --remove-orphans retires the now-undeclared Caddy container; the checksum flip (override bytes changed, Caddyfile gone) triggers --force-recreate. After the deploy, verify with docker compose config on the droplet — the Caddy service must not appear.

Caddy stores TLS certs in a named Docker volume (caddy_data). Docker volumes are independent of the git working tree — git clean -xfd and normal deploys never touch them. The real cert-loss risk is an explicit docker compose down -v or docker volume prune on the gateway project, which would wipe the certs and force Let’s Encrypt re-issuance (rate-limit risk). Never run docker compose down -v or docker volume prune on the gateway stack. See the ANTI-PATTERNS section.

If GATEWAY_WEBHOOK_SECRET leaks, the blast radius is Fro-Bot-user impersonation into the presence channel. Rotate by setting a new value in the gateway GitHub Environment and redeploying. Coordinate with the fro-bot/.github caller, which holds the matching secret on its side. The deploy re-materializes the secret file via SSH stdin; the checksum flip force-recreates the daemon container.

After the first enabling deploy:

  1. bunx @marcusrbrown/infra gateway status — confirm the caddy service appears alongside gateway, workspace, mitmproxy (running or healthy — Caddy has no healthcheck, so running/n-a is normal).
  2. Send a correctly HMAC-signed test POST to https://gateway.fro.bot/v1/announce — confirm a Discord embed appears in the presence channel.
  3. Send an unsigned or replayed POST — confirm a 401/403 response.
  4. Send a request to any other path (e.g. https://gateway.fro.bot/) — confirm a 404 response.

For verifying a real control-plane go-live end-to-end (live log monitoring, success-field criteria, and distinguishing the real signed event from test probes), see docs/runbooks/gateway-announce-event-verification.md.

buildSecretFileList, getAnnounceState, buildComposeOverride, buildCaddyfile in apps/gateway/src/deploy.ts.

The operator auth gate is part of the operator listener feature. When the operator listener trio is enabled, all four auth/config secrets are required. The deploy fails before any SSH if the listener trio is set but any of the four auth/config secrets is missing or partially set.

GitHub Environment secret _FILE env var in compose override Description
GATEWAY_OPERATOR_GITHUB_CLIENT_ID GATEWAY_OPERATOR_GITHUB_CLIENT_ID_FILE GitHub OAuth App client ID
GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET_FILE GitHub OAuth App client secret
GATEWAY_OPERATOR_CSRF_SECRET GATEWAY_OPERATOR_CSRF_SECRET_FILE Strict base64url, no padding/newlines, ≥32 decoded bytes
GATEWAY_OPERATOR_ALLOWLIST GATEWAY_OPERATOR_ALLOWLIST_FILE Newline-separated numeric GitHub user IDs; fail-closed if missing/empty/malformed

All four are all-or-none with the operator listener trio. Setting one to three of the four fails the deploy before any SSH. Leaving all four unset when the listener trio is also unset is the disabled state.

These are plain env vars (not file-backed). Set them in the gateway GitHub Environment if needed; leave unset to use upstream defaults.

GitHub Environment secret Default Description
GATEWAY_OPERATOR_OAUTH_ALLOWED_RETURN_PATHS /operator Comma-separated same-origin post-auth paths
GATEWAY_OPERATOR_OAUTH_STATE_TTL_MS 600000 OAuth state TTL in milliseconds
GATEWAY_OPERATOR_OAUTH_MAX_OUTSTANDING_ATTEMPTS 5 Max concurrent outstanding OAuth attempts

These are transported as optional secrets in the workflow (not workflow_call.inputs) for consistency with the gateway deploy pipeline.

The GitHub OAuth App must be configured with:

https://dashboard.fro.bot/operator/auth/github/callback

This is a GitHub OAuth App (client ID/secret) — not the existing gateway GitHub App (GH_APP_ID). Register the callback URL in the GitHub Developer Portal before enabling operator auth. The deploy dry-run prints the expected callback URL for manual cross-check against the portal.

GATEWAY_OPERATOR_TRUSTED_PROXIES is not a GitHub Environment input. The deploy derives it from DASHBOARD_VPC_IP and writes it as a plain environment entry in the gateway service block of compose.override.yaml when the operator listener is enabled. The daemon’s socket peer on :9300 for the dashboard path is exactly the dashboard droplet’s VPC IP (Docker DNAT keeps the source), so that single address is the trusted proxy. The gateway-net Caddy address is deliberately not trusted: gateway.fro.bot/operator/* stays untrusted and returns 400. The Phase 5d rendered-config gate asserts the merged gateway env carries exactly this value before docker compose up.

Sessions are durable. Each session is a signed record in the object store under operator-web/sessions/ (grant: S3 State-Bucket IAM Actions) and lasts a fixed seven days from sign-in. A gateway restart (deploy, force-recreate, container crash) does not end sessions: the gateway restores them at startup and logs session restore finished with an auth.session.restore audit event. Logout ends only that session. To invalidate every session, rotate GATEWAY_OPERATOR_CSRF_SECRET or delete operator-web/sessions/*. If the session store grant is missing, logins return 503 until the grant is fixed and the gateway restarted.

To disable operator auth/config wiring after deployment:

  1. Remove or clear both the listener trio and the four auth/config secrets from the gateway GitHub Environment:
    • Listener trio: GATEWAY_OPERATOR_BIND_HOST, GATEWAY_OPERATOR_BIND_PORT, GATEWAY_OPERATOR_PUBLIC_ORIGIN
    • Auth/config secrets: GATEWAY_OPERATOR_GITHUB_CLIENT_ID, GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET, GATEWAY_OPERATOR_CSRF_SECRET, GATEWAY_OPERATOR_ALLOWLIST
  2. Trigger a gateway deploy: bunx @marcusrbrown/infra gateway deploy.
  3. Approve the environment gate.
  4. The deploy detects the operator listener is disabled, skips all operator auth/config wiring, and runs docker compose up --remove-orphans.
  5. The gateway restarts with the operator listener disabled and serves no operator sessions. Session records stay in the object store; to invalidate them, rotate GATEWAY_OPERATOR_CSRF_SECRET or delete operator-web/sessions/*.
  6. Verify: bunx @marcusrbrown/infra gateway status shows all services healthy.

Note: Listener trio present + auth/config secrets absent is not a valid state — the deploy fails before any SSH. The only supported rollback is to clear both the listener trio and the four auth/config secrets together.

After the first deploy with operator auth/config wiring:

  1. Gateway-side liveness probe: Probe the listener directly via the droplet-local gateway-net address:

    Terminal window
    curl -sf http://172.21.0.2:9300/operator/health

    A 200 {"ok":true,"contractVersion":"<semver>"} response confirms the operator listener is up.

    ⚠ Liveness probe warning: Do not use https://gateway.fro.bot/operator/health as a liveness probe — a global forwarded-header guard returns 400 {"error":"bad request"} for any request where X-Forwarded-Host does not match the PUBLIC_ORIGIN host (dashboard.fro.bot); the public Caddy route forwards Host: gateway.fro.bot, so it returns 400 by design; the gateway-net Caddy address is deliberately not a trusted proxy. The gateway-side liveness probe is the droplet-local direct probe above. The dashboard same-origin path is live: https://dashboard.fro.bot/operator/health → 200 {"ok":true,"contractVersion":"<semver>"} is a valid external probe. The OAuth callback URL registration uses the dashboard origin (https://dashboard.fro.bot/operator/auth/github/callback) — this is a GitHub OAuth App setting, not a live HTTP probe target.

  2. Auth gate coarse check: GET https://gateway.fro.bot/operator/ returns 400 {"error":"bad request"} by design — the forwarded-header guard rejects it because the forwarded host (gateway.fro.bot) does not match the PUBLIC_ORIGIN host (dashboard.fro.bot). Do not use gateway.fro.bot/operator/ as an auth probe. The auth-gate / OAuth-redirect check is meaningful through the live same-origin path: curl -sI https://dashboard.fro.bot/operator/auth/github/start returns 302 to GitHub OAuth. A full browser round-trip returns the operator identity JSON {"githubUserId":<operator-id>,"login":<login>} from the callback when the GitHub user is on GATEWAY_OPERATOR_ALLOWLIST.

  3. Callback URL preflight: Before triggering the OAuth flow, verify the expected callback URL string. The deploy dry-run output prints:

    Expected OAuth callback URL: https://dashboard.fro.bot/operator/auth/github/callback

    Cross-check this string against the Authorization callback URL field in the GitHub OAuth App settings (GitHub Developer Portal → OAuth Apps → your app). The portal check is manual and must be completed before enablement.

  4. Allowlist enforcement: Attempt to complete the OAuth flow with a GitHub account that is NOT in the allowlist. Confirm the response is a coarse auth-failure (non-5xx, no route or allowlist detail leaked).

  5. Session survives restart: After a successful auth, trigger a gateway restart (docker compose restart gateway on the droplet). Confirm the gateway log shows session restore finished and the same cookie still returns 200 from /operator/session without signing in again.

  6. Operator port is VPC-scoped, not public: docker compose --project-directory /opt/gateway/deploy ps shows a <vpc-ip>:9300->9300 mapping (VPC-scoped, e.g. 10.116.0.3:9300->9300). A 0.0.0.0:9300, [::]:9300, or bare 9300:9300 mapping is a defect — the publish must be bound to the gateway VPC IP only. Access is further restricted to the dashboard droplet by a DOCKER-USER iptables rule and a DigitalOcean Cloud Firewall rule (see Operator private path).

For the full operator auth lifecycle runbook (OAuth App setup, CSRF secret generation, allowlist format, secret seeding, rotation, rollback), see docs/runbooks/gateway-operator-auth-lifecycle.md.

Gateway operator push is enabled in production. The deploy accepts this VAPID quartet only as an all-or-none configuration:

Environment key Host file Container _FILE path
GATEWAY_OPERATOR_PUSH_VAPID_PUBLIC_KEY gateway-operator-push-vapid-public-key /run/secrets/gateway_operator_push_vapid_public_key
GATEWAY_OPERATOR_PUSH_VAPID_PRIVATE_KEY gateway-operator-push-vapid-private-key /run/secrets/gateway_operator_push_vapid_private_key
GATEWAY_OPERATOR_PUSH_VAPID_SUBJECT gateway-operator-push-vapid-subject /run/secrets/gateway_operator_push_vapid_subject
GATEWAY_OPERATOR_PUSH_VAPID_KEY_VERSION gateway-operator-push-vapid-key-version /run/secrets/gateway_operator_push_vapid_key_version

All four absent is disabled. Any partial set hard-fails before SSH, spawn, secret materialization, or remote writes and names every missing key. A complete set must pass strict validation: the public key is unpadded base64url decoding to exactly 65 bytes with a 0x04 first byte; the private key is unpadded base64url decoding to exactly 32 bytes; the subject is nonblank and uses mailto: or https:; and the key version is a positive integer string.

Operators never set GATEWAY_OPERATOR_PUSH_ENABLED. The deploy derives GATEWAY_OPERATOR_PUSH_ENABLED=true only from a complete valid quartet and emits no false line. The four files are materialized through the existing SSH-stdin path only; the private key never appears in argv, logs, or dashboard-bound artifacts. The compose override uses the four read-only _FILE paths above. Push is additive to the announce, operator-listener, and VPC gates; absent push configuration does not change those gates.

Verify enablement with an unauthenticated GET https://dashboard.fro.bot/operator/push/vapid-key: 401 means the route is mounted, 404 means it is not. The routes mount only after the object-store CAS self-test passes at startup, so a 401 proves the self-test passed at boot — mount state is never re-evaluated, so it is a post-deploy check, not a liveness probe. Enablement, format gates, rotation limits, and rollback: docs/runbooks/operator-push-enablement.md.

The four VAPID files and rendered compose override participate in the existing secrets checksum, so content or wiring changes select the existing recreate path. When a previously enabled deploy is followed by an absent quartet, the bounded cleanup deletes exactly the four known files under /opt/gateway/deploy/secrets/ before current secret materialization; the new override omits the VAPID mounts and derived flag, and the checksum path recreates the gateway disabled. A never-enabled absent quartet remains the ordinary no-file path.

The authenticated GET /operator/push/vapid-key route returns only {publicKey, keyVersion} when enabled. It returns 404 when push is disabled or the operator guard denies the request.

Verify names and status only: the four gateway VAPID secret/variable names remain absent, the dashboard push variable remains absent or false, and no independently forwarded gateway enable setting exists. There is no GitHub-stored derived gateway flag to inspect. Never retrieve, print, copy, or compare secret contents.

The gateway operator listener is opt-in. When enabled, it exposes GET /operator/health (and future privileged operator routes) on a gateway-net-only address. Caddy routes /operator/* traffic from the public HTTPS edge to the listener over gateway-net. The listener is also published on the gateway VPC IP (${GATEWAY_VPC_IP}:9300) so the dashboard droplet can reach it over the shared DigitalOcean VPC — see Operator private path for the full bridge topology.

Browser-visible operator origin: The ratified target for the browser-visible operator API is https://dashboard.fro.bot/operator/*. GATEWAY_OPERATOR_PUBLIC_ORIGIN must be set to https://dashboard.fro.bot when the operator listener is enabled for production use. The gateway-side Caddy /operator/* route is topology scaffolding — it proves the gateway listener and Caddy wiring work, but gateway.fro.bot/operator/* is not the production browser origin and must not be used as such. See docs/plans/2026-06-18-001-feat-dashboard-operator-same-origin-plan.md for the full decision record.

Current state: The operator listener is enabled in production. GATEWAY_OPERATOR_BIND_HOST=172.21.0.2, GATEWAY_OPERATOR_BIND_PORT=9300, and GATEWAY_OPERATOR_PUBLIC_ORIGIN=https://dashboard.fro.bot are set in the gateway GitHub Environment. Deploy run 27740787921 succeeded; operator listener liveness is confirmed via the droplet-local direct probe (curl -sf http://172.21.0.2:9300/operator/health → 200 {"ok":true,"contractVersion":"<semver>"}). The public https://gateway.fro.bot/operator/health route returns 400 by design (forwarded-header guard — X-Forwarded-Host: gateway.fro.bot does not match PUBLIC_ORIGIN host dashboard.fro.bot). Operator auth is live: all four auth/config secrets (GATEWAY_OPERATOR_GITHUB_CLIENT_ID, GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET, GATEWAY_OPERATOR_CSRF_SECRET, GATEWAY_OPERATOR_ALLOWLIST) are seeded in the gateway GitHub Environment and materialized as _FILE mounts. The dashboard Caddy /operator/* same-origin route is live: https://dashboard.fro.bot/operator/* routes through dashboard Caddy → VPC 10.116.0.3:9300 → gateway operator listener. End-to-end auth is verified: https://dashboard.fro.bot/operator/auth/github/start returns 302 to GitHub OAuth; completing the flow returns the operator identity JSON for an allowlisted user. The push client ships in the deployed dashboard image: https://dashboard.fro.bot/sw.js serves a Workbox service worker carrying push, notificationclick, and pushsubscriptionchange handlers. The authenticated UI surface itself sits behind GitHub OAuth and has not been verified from outside auth. Background: docs/plans/2026-06-18-001-feat-dashboard-operator-same-origin-plan.md.

Set all three vars in the gateway GitHub Environment and trigger a deploy:

Variable Description
GATEWAY_OPERATOR_BIND_HOST Gateway-net IPv4 address for the listener bind (e.g. 172.21.0.2). Must be a static gateway-net address — not 0.0.0.0, loopback (127.*), sandbox-net (10.*), 172.20.* (the known conflicted Docker subnet from the failed operator deploy), or IPv6.
GATEWAY_OPERATOR_BIND_PORT Port for the operator listener (e.g. 9300). Must be a positive integer in [1, 65535].
GATEWAY_OPERATOR_PUBLIC_ORIGIN HTTPS origin for the browser-visible operator API. The ratified target is https://dashboard.fro.bot. Must be HTTPS.

All three must be set together (all-or-none). Setting one or two fails the deploy before any SSH. Leaving all three unset disables the operator listener.

The deploy:

  • Validates all three values before any SSH (rejects unsafe bind hosts, invalid ports, non-HTTPS origins).
  • Requires GATEWAY_VPC_IP and DASHBOARD_VPC_IP (valid exact IPv4) before any SSH; DASHBOARD_VPC_IP is the derived trusted proxy.
  • Writes GATEWAY_OPERATOR_BIND_HOST, GATEWAY_OPERATOR_BIND_PORT, GATEWAY_OPERATOR_PUBLIC_ORIGIN, and GATEWAY_OPERATOR_TRUSTED_PROXIES (= DASHBOARD_VPC_IP) into the gateway service environment in compose.override.yaml.
  • Sets a static ipv4_address on gateway-net for the gateway service so the bind address is deterministic.
  • Adds a /operator/* Caddy route with flush_interval -1 (no response buffering, ready for future SSE).
  • Auto-cleans stale fro-bot_gateway-net — after docker compose pull and before docker compose up, inspects the remote Docker network. If it exists with the old 172.20.0.0/16 subnet, removes it so Docker can recreate it with 172.21.0.0/16. Missing or already-correct networks are skipped. Pull happens first so images are cached before any container disruption — if the pull fails, gateway/caddy containers are never removed. If the first removal fails due to active endpoints (gateway/caddy still attached), the deploy removes those two containers directly via docker rm -f fro-bot-gateway-1 and docker rm -f fro-bot-caddy-1 (tolerating missing containers so recovery works when caddy is absent from the current compose config) and retries the removal (stop alone is insufficient — stopped containers retain network endpoints). Any other removal failure, or failure of the container removal/retry, fails the deploy closed before docker compose up. GATEWAY_OPERATOR_BIND_HOST must be in the 172.21.x.x range (e.g. 172.21.0.2).
  • Probes GET ${GATEWAY_OPERATOR_PUBLIC_ORIGIN}/operator/health (i.e. https://dashboard.fro.bot/operator/health) after compose up, through the dashboard Caddy → VPC path whose peer is the trusted proxy. Success is HTTP 200 with a non-empty string contractVersion in the JSON body; the reported version is logged and not compared to any expected value. The probe is warning-only: a failed or malformed response is retried, then warned about, and never fails the deploy. The gateway-origin https://gateway.fro.bot/operator/health returns 400 by design and is not probed. The droplet-local curl -sf http://172.21.0.2:9300/operator/health remains the direct listener check.
  • The operator listener is published only on the gateway VPC IP (${GATEWAY_VPC_IP}:9300, e.g. 10.116.0.3:9300), never on 0.0.0.0 or the public eth0. A 0.0.0.0:9300 or bare 9300:9300 publish is a defect.
  • Access to the published VPC port is restricted to the dashboard droplet by two independent controls: a DOCKER-USER iptables rule (reapplied every deploy; load-bearing because Docker DNAT bypasses ufw) and a DigitalOcean Cloud Firewall rule (created by provisioning via setupOperatorFirewall(); reboot-durable provider-level control). Both must be in place before the dashboard route goes live.
  • The workspace (sandbox-net) has no path to the operator listener.
  • The only trusted proxy is DASHBOARD_VPC_IP (GATEWAY_OPERATOR_TRUSTED_PROXIES); the gateway-net Caddy address is never trusted.
  • /v1/announce and /operator/* are separate Caddy handle blocks with distinct trust boundaries.
  • Auth/session/CSRF/allowlist wiring for privileged operator routes is live; the four operator auth/config secrets must be seeded in the gateway GitHub Environment to activate the auth gate (see Enabling and docs/runbooks/gateway-operator-auth-lifecycle.md).
  • The gateway Caddy /operator/* route is topology scaffolding. The production browser operator path is https://dashboard.fro.bot/operator/* via the dashboard Caddy proxy — not gateway.fro.bot/operator/* directly.

After the first enabling deploy:

  1. bunx @marcusrbrown/infra gateway status — confirm caddy and gateway services appear healthy.
  2. curl -sf http://172.21.0.2:9300/operator/health — confirm a 200 {"ok":true,"contractVersion":"<semver>"} response (droplet-local direct probe). Note: https://gateway.fro.bot/operator/health returns 400 by design (forwarded-header guard); use the direct probe for liveness confirmation.
  3. Confirm the operator port publish is VPC-scoped: docker compose --project-directory /opt/gateway/deploy ps — the gateway service must show a <GATEWAY_VPC_IP>:9300->9300 mapping (e.g. 10.116.0.3:9300->9300). A 0.0.0.0:9300, [::]:9300, or bare 9300:9300 mapping is a defect.
  4. Confirm workspace is not on gateway-net: docker network inspect gateway-net — the workspace container must not appear.
  5. Confirm the DOCKER-USER source restriction is in place: iptables -nvL DOCKER-USER --line-numbers — a RETURN rule scoped to the VPC ingress interface, source <DASHBOARD_VPC_IP>, destination <GATEWAY_OPERATOR_BIND_HOST>, and --dport 9300 must appear before a DROP rule for the same destination and port. See docs/runbooks/gateway-operator-private-path-verification.md for the full verification procedure.

OPERATOR PRIVATE PATH (DASHBOARD SAME-ORIGIN)

Section titled “OPERATOR PRIVATE PATH (DASHBOARD SAME-ORIGIN)”

The dashboard droplet reaches the gateway operator listener over the shared DigitalOcean VPC (nyc1, uuid d95c26cc-…). The gateway VPC IP is 10.116.0.3 and the dashboard VPC IP is 10.116.0.5 (example values; the actual IPs are set via GATEWAY_VPC_IP and DASHBOARD_VPC_IP). The daemon stays bound to gateway-net 172.21.0.2:9300; the bridge is a host-side Docker port publish, not a daemon rebind.

  1. VPC-IP-scoped Docker port publish — the gateway service publishes ${GATEWAY_VPC_IP}:9300:9300 in the compose override. The daemon stays bound to 172.21.0.2:9300 on gateway-net; Docker DNAT forwards VPC-IP traffic to it. The publish is emitted whenever the operator listener is enabled; GATEWAY_VPC_IP and DASHBOARD_VPC_IP are required then.

  2. DOCKER-USER iptables rule — applied over SSH after docker compose up (Docker recreates DOCKER-USER-adjacent chains on daemon restart, so the rule must be reapplied last). The rule allows --dport 9300 from ${DASHBOARD_VPC_IP} and drops all other sources. Applied idempotently (-C || -I); exact-readback-verified (source/dport/jump/ordering). This rule is load-bearing — the daemon’s forwarded-header guard is forgeable by anyone who can reach :9300; Docker DNAT bypasses ufw. The rule is lost on reboot until the next deploy; the DO Cloud Firewall covers the reboot window.

  3. DigitalOcean Cloud Firewall — created and maintained by provisioning (provision-droplet.ts / bun run provision:gateway) via setupOperatorFirewall(), not by the deploy. Provisioning creates a firewall named gateway-operator-fw with base inbound rules (SSH 22, HTTP 80, HTTPS 443) plus inbound tcp/9300 from the dashboard droplet-id, and attaches it to the gateway droplet. The operation is idempotent: create-if-absent, add-9300-rule-if-missing, no-op if already present. Requires GATEWAY_VPC_IP and DASHBOARD_VPC_IP to be set at provisioning time. Source is the dashboard droplet-id (stable across rebuild), not the private IP. Reboot-durable provider-level control. DIGITALOCEAN_ACCESS_TOKEN is a provisioning-time concern (repo-level, used locally) — it is not required in the gateway GitHub Environment for deploy.

  4. Dashboard Caddy /operator/* route — the dashboard Caddy proxies /operator/* to {$GATEWAY_VPC_IP}:9300 over the VPC. The route is a handle block before the dashboard:3000 catch-all, with flush_interval -1 and header_up Host dashboard.fro.bot + header_up X-Forwarded-Proto https to satisfy the daemon’s forwarded-header guard.

Required env vars (whenever the operator listener is enabled)

Section titled “Required env vars (whenever the operator listener is enabled)”
Variable Where set Description
GATEWAY_VPC_IP gateway GitHub Environment + local .env Gateway droplet’s DigitalOcean VPC IP (e.g. 10.116.0.3). Required for the VPC-IP publish and DOCKER-USER rule.
DASHBOARD_VPC_IP gateway GitHub Environment + local .env Dashboard droplet’s DigitalOcean VPC IP (e.g. 10.116.0.5). Required for the DOCKER-USER source restriction and used as the derived GATEWAY_OPERATOR_TRUSTED_PROXIES value.
DIGITALOCEAN_ACCESS_TOKEN local .env only (provisioning-time) Used by provision-droplet.ts to create the gateway-operator-fw Cloud Firewall. Not a deploy secret — not required in the gateway GitHub Environment.

GATEWAY_VPC_IP is also required in the dashboard GitHub Environment so the dashboard Caddy service can expand {$GATEWAY_VPC_IP} in the Caddyfile.

First-deploy prerequisite: the dashboard droplet must already exist before running bun run provision:gateway with the firewall setup, because provisioning resolves the dashboard droplet-id to set the 9300 inbound source. Run bun run provision:dashboard first, then bun run provision:gateway with GATEWAY_VPC_IP and DASHBOARD_VPC_IP set.

Deploy the gateway bridge first (VPC publish + DOCKER-USER + DO firewall), verify the bridge is live and restricted, then deploy the dashboard /operator/* route. The dashboard route must not go live before its target is reachable and restricted — a partial state where the route is live but the bridge is absent or unrestricted is a security defect.

See docs/runbooks/gateway-operator-private-path-verification.md for the full verification procedure (positive probes, control readback, negative-control caveat, and break-glass).

The mitmproxy CA cert and private key live in the Docker named volume fro-bot_mitmproxy-certs (the compose key mitmproxy-certs prefixed with the Compose project name fro-bot). Backup and restore both check that this volume exists before doing anything else and abort naming it if it is missing, so neither can create a stray empty mitmproxy-certs volume. If the droplet is destroyed or the volume is lost, all currently-running workspaces lose trust in the gateway’s egress proxy.

Backup (operator-driven, periodic):

Terminal window
bunx @marcusrbrown/infra gateway backup --include-ca --output ./gateway-ca.tar

Move the file to a safe store (1Password, encrypted backup, etc.). The tar contains both mitmproxy-ca-cert.pem (public) and mitmproxy-ca.pem (private key). Treat the file as a secret.

Restore — droplet intact (volume lost or cert corrupted):

Terminal window
bunx @marcusrbrown/infra gateway restore --include-ca --input ./gateway-ca.tar

The CLI validates the archive locally first — it must contain exactly the two expected files, otherwise it is rejected before any remote mutation. It then confirms the fro-bot_mitmproxy-certs volume exists on the host (restore never creates it), uploads to an unguessable remote path, extracts into the named volume, restarts mitmproxy + gateway, and byte-equal compares the restored cert + key against the input archive. Mismatch aborts with a clear error.

Restore — full disaster recovery (droplet destroyed):

  1. Set GATEWAY_HOST locally and run bun run provision:gateway to create a replacement droplet.
  2. Commit and push the updated .github/known_hosts.
  3. If the host or IP changed, update GATEWAY_HOST (and GATEWAY_SSH_KEY if re-keyed) in the gateway GitHub Environment.
  4. Trigger a deploy (bunx @marcusrbrown/infra gateway deploy) and approve the environment gate. This creates /opt/gateway/deploy/ on the new droplet.
  5. bunx @marcusrbrown/infra gateway restore --include-ca --input ./gateway-ca.tar — restores the CA into the named volume.
  6. Verify: bunx @marcusrbrown/infra gateway status and bunx @marcusrbrown/infra gateway logs gateway --tail 100.
  1. Update the value in the gateway GitHub Environment.
  2. Trigger gateway deploy (or wait for the next push-to-main run).
  3. The deploy script writes the new value to the droplet, recomputes the secrets checksum, force-recreates affected containers via docker compose up -d --force-recreate <service>, polls Discord for command registration, and writes the new checksum on success.

The checksum-after-success invariant means: if compose or polling fail mid-rotation, the old checksum persists and the next deploy will force-recreate again — no silent stale-credentials state.

For the full operator-facing rotation and emergency revocation procedure (including Keychain sync, shell-history hygiene, and audit surfaces), see docs/runbooks/discord-token-lifecycle.md.

  • Never ssh-keyscan in CI — host keys are pinned in .github/known_hosts at provision time and committed. Provisioning scripts may use ssh-keyscan locally.
  • Never pass secret bytes via argv — writeRemoteFile pipes bytes through SSH stdin only. --body <value> patterns are banned.
  • Never skip validateGatewayHost — it rejects --prefixed values and characters outside the allowed alphabet. SSH treats --prefixed hostnames as flags (including -oProxyCommand=).
  • Never restart the gateway in-place to rotate the CA — workspaces lose trust in the egress proxy. Restore from backup instead.
  • Never run docker compose up --build on the droplet — building the gateway or workspace images on the 1vCPU/2GB droplet exhausts RAM and thrashes swap while the live stack is running. The deploy uses docker compose pull + up -d --no-build; the compose.override.yaml image pins make GHCR the source of truth. For emergency rebuilds, build from a workstation and push to GHCR (see Break-glass runbook).
  • Never validate WORKSPACE_OPENCODE_CONFIG with the shell-metachar guard — it is JSON (", $, \ are required) and SHELL_METACHAR_RE would reject every valid config. Validate it with JSON.parse + newline/size checks; SHELL_METACHAR_RE is only for simple values like WORKSPACE_OPENCODE_MODEL.
  • Never bind-mount config files outside /opt/gateway/deploy/secrets/ — init-certs.sh and docker-compose.yaml are upstream’s; this repo materializes secrets only.
  • Never run pollRegistration with an unbounded per-attempt timeout — each fetch is wrapped in an AbortController with perAttemptTimeoutMs (defaults to max(6000, intervalMs * 2)).
  • Never run docker compose down -v or docker volume prune on the gateway stack — this wipes named Docker volumes including caddy_data (TLS certs), mitmproxy-certs (CA key; live name fro-bot_mitmproxy-certs), and workspace-repos (all cloned repo checkouts). Normal deploys and git clean -xfd do NOT touch Docker volumes. Losing caddy_data forces Let’s Encrypt re-issuance (rate-limit risk); losing mitmproxy-certs breaks workspace egress trust (requires CA restore); losing workspace-repos destroys all cloned repos (they will be re-cloned automatically on the next mention, but the volume loss itself is irreversible).
  • Never publish the operator port on 0.0.0.0 or without a VPC-IP bind — the operator listener must be published only on ${GATEWAY_VPC_IP}:9300, never on 0.0.0.0:9300, [::]:9300, or as a bare 9300:9300 mapping. A public-internet-reachable operator port is a security defect; the DOCKER-USER rule and DO Cloud Firewall are the access controls, but they are only effective when the publish is VPC-scoped.
  • Never apply the DOCKER-USER rule before docker compose up — Docker recreates DOCKER-USER-adjacent chains on daemon restart and compose up, wiping any rule applied before the stack comes up. The rule must be reapplied after compose up completes.
  • Never create a fresh DO Cloud Firewall for the operator port outside of provisioning — DO Cloud Firewalls are default-deny/allowlist; attaching a new firewall that only allows :9300 would lock out :22/:80/:443/announce. Provisioning (setupOperatorFirewall()) creates gateway-operator-fw with all required base rules (22, 80, 443) plus the 9300 source restriction, idempotently. The deploy does not touch the DO firewall.
  • Never raise the workspace memory limit without re-measuring host available memory — measure with the workspace at its limit. At 848 MiB used the host had 490 MiB available, so the 1024 MiB limit leaves about 314 MiB; keep at least 256 MiB.
  • Never drop memswap_limit from the workspace service — without it Docker allows swap up to twice mem_limit.
  • Never grant the fro-bot-gateway runtime identity IAM self-management — its credentials write run-state objects; they must not carry iam:* permissions to manage their own policy. If a permission is missing (e.g. s3:PutObjectTagging, see S3 State-Bucket IAM Actions), fix the grant in AWS out-of-band — never widen the identity to route around a mismatched permission.
  1. bunx @marcusrbrown/infra gateway backup --include-ca — preserve the CA if you ever want to restore the gateway.
  2. doctl compute droplet delete <droplet-id>
  3. Remove the gateway entries from .github/known_hosts.
  4. Delete the gateway GitHub Environment.