Gateway Deploy Package
The gateway is the Fro Bot Discord client and workspace runner — a 3-service Docker Compose stack (gateway daemon, workspace executor, mitmproxy egress filter) deployed on a dedicated DigitalOcean droplet at gateway.fro.bot. The upstream source is fro-bot/agent, pinned in apps/gateway/upstream.json, which serves the operator contract version reported by /operator/health and must be paired with a dashboard release that declares the same contract. The workspace image’s Node, OpenCode, and Systematic versions come from that pinned upstream release.
With the operator surface enabled the daemon refuses to start unless GATEWAY_OPERATOR_TRUSTED_PROXIES is set; the deploy derives it from DASHBOARD_VPC_IP (see Operator Listener). The operator listener is enabled when GATEWAY_OPERATOR_BIND_HOST, GATEWAY_OPERATOR_BIND_PORT, and GATEWAY_OPERATOR_PUBLIC_ORIGIN are all set. Its auth config — GitHub OAuth client ID/secret, CSRF secret, and allowlist — is materialized as _FILE mounts. The operator surface provides a session/OAuth/CSRF/allowlist auth layer (the OAuth callback redirects to the validated return_to path after minting the session), an authenticated SSE run-stream, and server-side operator token retention, with metadata/repos.yaml redacted on operator surfaces. GET /operator/runs is a read-only, session-gated run index; POST /operator/runs launches runs and caps idempotencyKey at 256 characters (400 above that); POST /operator/dispatch exists; web tool-approval is supported.
The workspace runs OpenCode as uid 10001 (checkout ownership migrates one-way on first boot — see Rollback) and bakes OPENCODE_EXPERIMENTAL_DISABLE_FILEWATCHER=true. Do not override it back on. The only state that watcher refreshes is OpenCode’s per-directory cached current branch, read solely by GET /vcs and branch-mode GET /vcs/diff — neither of which the gateway mention loop calls, so the staleness is inert here. Two corrections to the upstream note: a stale value skews branch-mode diff results, not just a displayed branch name; and clearing this flag does not enable recursive working-tree watching, which is a separate opt-in (OPENCODE_EXPERIMENTAL_FILEWATCHER). If reused checkouts under /workspace/repos drift behind GitHub, that is a checkout-update policy problem — /clone returns 409 repo-exists and never fetches or switches branches — and no amount of file watching fixes it. /fro-bot dispatch needs the GitHub App to hold Actions: read and write; that permission is granted on the Fro Bot Agent App and accepted by the installations that use it.
The deploy consumes these upstream files: deploy/compose.yaml, deploy/gateway.Dockerfile, deploy/workspace.Dockerfile, deploy/mitmproxy/allowlist.py, and deploy/workspace-entrypoint.sh. The workspace has native always-on session tools. Mention runs are serialized by a per-channel run queue (one active run per Discord channel at a time), and /fro-bot force-release-lock lets an operator release a channel lock. Mention runs show a live-status message and typing indicator (GATEWAY_STATUS_MODE, defaults to live-status, no infra wiring needed). The daily_digest presence/announce event reuses the existing /v1/announce HMAC webhook ingress (GATEWAY_WEBHOOK_SECRET + GATEWAY_PRESENCE_CHANNEL_ID).
WORKSPACE_EGRESS_HOSTS is consumed by deploy/mitmproxy/allowlist.py (comma-separated exact hosts the sandboxed workspace may reach through the mitmproxy egress proxy; fail-closed if empty). The deploy emits WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai so the workspace OpenCode can reach the cliproxy endpoint. models.dev and models.opencode.ai are OpenCode model-catalog hosts: they stay allowlisted but are currently unused, because the workspace OpenCode V2 server runs with OPENCODE_DISABLE_MODELS_FETCH=1 and starts from its bundled catalog snapshot. The workspace executor exposes /healthz + /clone on :9100 and an OpenCode bearer proxy on :9200, all on the internal sandbox-net with no host ports, enabling both /fro-bot add-project repo cloning and the @fro-bot mention loop. GATEWAY_WEBHOOK_SECRET and GATEWAY_PRESENCE_CHANNEL_ID are readOptionalSecret (opt-in) in the upstream daemon — set both to enable the announce/presence webhook (see Announce/presence webhook below), or leave both unset to keep the gateway outbound-only. The deploy materializes GitHub App credentials (github-app-id, github-app-private-key) from GH_APP_ID / GH_APP_PRIVATE_KEY environment secrets.
The deploy script materializes secrets as files on the droplet (never via argv), bootstraps the mitmproxy CA on first run, brings up the Compose stack, and gates completion on Discord command registration. A secrets checksum written only after a fully successful deploy prevents silent stale-credentials states across retries.
WHERE TO LOOK
Section titled “WHERE TO LOOK”| Task | Location | Notes |
|---|---|---|
| Upstream pin | apps/gateway/upstream.json |
fro-bot/agent ref — update to upgrade |
| Deploy script | apps/gateway/src/deploy.ts |
Secrets materialization, compose up, registration poll |
| Provision droplet | apps/gateway/server/provision-droplet.ts |
One-time. Refuses re-run on existing droplet without --force |
| CLI commands | packages/cli/src/commands/gateway/ |
status, deploy, logs, backup, restore |
DEPLOY FLOW
Section titled “DEPLOY FLOW”The deploy runs in two stages: a CI build-images job builds and pushes the gateway and workspace Docker images to GHCR, then the deploy-gateway job SSHes to the droplet to pull those images and bring up the stack. The droplet never builds images — it only pulls prebuilt artifacts.
CI: build-images job
Section titled “CI: build-images job”Runs on ubuntu-latest with packages: write permission (job-scoped only; the deploy job is package-read-only). Steps:
- Reads the pinned
reffromapps/gateway/upstream.json. - Checks out
fro-bot/agentat that ref. - Logs in to
ghcr.iowithGITHUB_TOKEN. - Builds
deploy/gateway.Dockerfile(context: upstream root) and pushes toghcr.io/marcusrbrown/infra-gateway:<ref>-<github.sha>. Captures the pushed digest frombuild-push-actionoutputs. The tag is keyed on the infra commit as well as the upstream ref so repeated builds at the same upstream ref no longer overwrite each other’s tag. - Builds
deploy/workspace.Dockerfile(context: upstream root) with--build-arg OPENCODE_VERSION=<version>and pushes toghcr.io/marcusrbrown/infra-workspace:<ref>-<github.sha>. Captures the pushed digest. The version is read frompackages/runtime/src/shared/opencode-v2-pin.jsonin the checked-out upstream source (theRead workspace OpenCode V2 pinstep), so it always matches the asset digests the Dockerfile verifies. The Dockerfile’sversion-gatestage refuses any otherOPENCODE_VERSION, including a V1 harness build, so the workspace image runs OpenCode V2 at the pinned version only; the gateway build takes no OpenCode arg. - Exposes both digests as job outputs (
gateway_digest,workspace_digest).
Both build steps pin a docker/setup-buildx-action builder to a fixed moby/buildkit image, set SOURCE_DATE_EPOCH as a build-arg, and export with rewrite-timestamp=true — this makes config/history/index timestamps and in-layer tar timestamps deterministic for a given upstream ref. Bit-reproducibility of a ref is not guaranteed. The upstream Dockerfiles install Alpine packages without version pins, because Alpine’s branch index drops older versions and an exact pin would fail the build. Rebuilding the same ref can therefore pick up newer packages and produce a different digest. The deployed image is always the digest this job pushed. To check whether a ref reproduced, compare the digests of two published tags sharing an upstream ref prefix (<ref>-<sha>). Layers map to instructions via the image config history (skipping empty_layer entries), so a drifting index names the instruction that caused it; equal size with a different digest is timestamp drift, not content.
CI: scan-images job
Section titled “CI: scan-images job”Runs between build-images and the gateway environment approval gate, so results are visible on the run page when the operator decides whether to approve. Scans both images by the exact digest from build-images outputs, never by tag. Uses the official Trivy container pinned by digest (ghcr.io/aquasecurity/trivy:0.74.0@sha256:62b1e65e...) rather than aquasecurity/trivy-action, because that action had all but one of its version tags force-pushed to credential-stealing malware in March 2026 (GHSA-69fq-xp46-6x23) and is kept out of the trust path. Report-only — --exit-code 0 and job-level continue-on-error: true — so scan findings or a scanner outage never block a deploy. Table output goes to the job step summary. On main only, each step also emits SARIF (CRITICAL,HIGH, --ignore-unfixed) and uploads it with github/codeql-action/upload-sarif; alerts appear under Security → Code scanning with categories trivy-gateway and trivy-workspace. A missing or empty SARIF file produces a ::warning:: and skips the upload — it is not a clean scan. Permissions are contents: read + packages: read + security-events: write; the deploy.yaml caller job must grant the same security-events: write.
The deploy-gateway job declares needs: [build-images, scan-images, notify]. A build or push failure in build-images blocks the deploy entirely — the old stack stays live. scan-images is continue-on-error: true, so neither findings nor a scanner outage block the deploy. The ungated, main-only notify job runs after the build and scan, before the environment approval gate; the deploy condition ignores its result, so a skipped or failed notification does not block deployment. See Deploy-Gate Notifications for delivery verification and troubleshooting.
Droplet: deploy-gateway job
Section titled “Droplet: deploy-gateway job”- Validate host —
validateGatewayHostrejects--prefixed values and characters outside the allowed alphabet. SSH treats--prefixed hostnames as flags (including-oProxyCommand=); this check is mandatory before any SSH invocation. - Checksum computation — SHA-256 of all secret values (including the
compose.override.yamlcontent) is computed locally. The result is compared against/opt/gateway/.secrets-checksumon the droplet. If the checksums differ,--force-recreateis added todocker compose upso containers pick up the new secret files. If they match and--force-recreatewas not explicitly passed, Compose converges the stack without forcing recreation: unchanged running containers can be reused, so a restart is not guaranteed. In both cases the deploy continues through its health and verification gates — there is no early exit. The checksum file lives outside the deploy clone sogit clean -xfdon subsequent deploys doesn’t wipe it. - Source materialization — SSH to the droplet; clone or fetch
fro-bot/agentat the pinned ref fromupstream.json, thengit reset --hard && git clean -xfdto the pinned SHA. The upstream checkout is still needed forcompose.yaml,init-certs.sh, and the Dockerfiles compose references — but the droplet does not build from them. - Secret files — 11 required files written atomically under
/opt/gateway/deploy/secrets/(discord-token,discord-application-id,discord-guild-id,aws-access-key-id,aws-secret-access-key,s3-bucket,s3-region,github-app-id,github-app-private-key,workspace-opencode-token,workspace-opencode-auth) plus optional-shaped files (s3-endpoint,aws-session-token,discord-privileged-intents,workspace-opencode-url,gateway-binder-allowlist(always written, empty whenGATEWAY_BINDER_ALLOWLISTis unset: compose bind-mounts it withcreate_host_path: false, so a missing file stops the stack), andgateway-trigger-role-id— written as 0-byte placeholders when unset, thoughGATEWAY_TRIGGER_ROLE_IDis enforced non-empty viaREQUIRED_ENV_VARS, so the mention-loop authz gate fails closed rather than opening to all guild members). The/opt/gateway/deploy/.envcarriesOBJECT_STORE_HOSTS(computed fromS3_BUCKET/S3_REGION/S3_ENDPOINT),WORKSPACE_OPENCODE_MODEL,WORKSPACE_OPENCODE_CONFIG— the latter is JSON-validated (viaJSON.parse, not the shell-metachar guard) and$→$$-escaped so docker-compose${VAR}interpolation passes the JSON through intact —WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai(v0.52.1+ requirement; consumed bydeploy/mitmproxy/allowlist.pyto allow the sandboxed workspace to reach the cliproxy OpenCode proxy and the OpenCode model catalog through mitmproxy; fail-closed if unset), and optionallyWORKSPACE_OPENCODE_READY_TIMEOUT_MS(v0.53.1+; absent/empty uses the upstream default of 60000ms; non-numeric/zero/negative values are rejected before any SSH). Secret bytes enter via SSH stdin only — never via argv. Filenames on disk are kebab-case; the compose contract maps each to/run/secrets/<snake_case>and exposes via${NAME}_FILEenv vars. The GitHub App credentials come from theGH_APP_ID/GH_APP_PRIVATE_KEYenvironment secrets (GitHub reserves theGITHUB_prefix for secret names) and materialize to the upstream-fixedgithub-app-id/github-app-private-keyfiles. Seedocs/runbooks/discord-token-lifecycle.mdfor the full mapping table. - Compose override —
compose.override.yamlis written on every deploy. It pinsgateway.imagetoghcr.io/marcusrbrown/infra-gateway@<digest>andworkspace.imagetoghcr.io/marcusrbrown/infra-workspace@<digest>using the exact digests from thebuild-imagesjob. It mounts theworkspace-reposnamed volume at/workspace/reposon the workspace service (persists cloned repo checkouts across container recreation and daemon upgrades) and setsWORKSPACE_OPENCODE_READY_TIMEOUT_MSin the workspace container environment via Docker Compose interpolation from the.envfile. It also setsmem_limitandmemswap_limiton the workspace service to 1024 MiB, so the workspace is hard-capped with no swap (WORKSPACE_MEMORY_LIMIT_MIBinapps/gateway/src/deploy.ts, issue #1652). Sizing, in MiB asfree -mreports them, from a measured run: with the workspace using 848 the host had 490 available, so a 1024 limit leaves about 314 available, above the 256 floor. Docker Compose deep-merges the override; the upstreambuild:stanzas remain present but are inert without--build. When announce is enabled, the override also adds the Caddy service and announce secret wiring (see Announce/presence webhook). - CA bootstrap —
init-certs.shruns on the droplet (idempotent; skips if the CA already exists in thefro-bot_mitmproxy-certsnamed volume). - Pull images —
docker compose --project-directory /opt/gateway/deploy pullfetches the pinned GHCR images by digest. A missing or unpullable image errors here and does not fall back to an on-droplet build. Pull happens before any container disruption so that if the pull fails, gateway/caddy containers are never removed. - Stale gateway-net cleanup — After pulling images, the deploy inspects the remote
fro-bot_gateway-netDocker network. If it exists with the old172.20.0.0/16subnet (from a previous failed operator deploy), it is removed so Docker can recreate it with the correct172.21.0.0/16subnet. If the network is missing or already has172.21.0.0/16, this step is a no-op. If the first removal attempt fails because containers are still attached (active endpoints — gateway and caddy services), the deploy removes those two containers directly viadocker rm -f fro-bot-gateway-1anddocker rm -f fro-bot-caddy-1(tolerating missing containers so recovery works when caddy is absent from the current compose config) and retries the removal. Note:docker compose stopalone is not sufficient — stopped containers can still keep active endpoints; only removing the containers releases their network endpoints. If container removal or the second removal fails, the deploy fails closed beforedocker compose upwith an error message stating that gateway/caddy containers were removed. This step runs afterdocker compose pulland beforedocker compose up— images are already cached before any container disruption. - Compose up —
docker compose --project-directory /opt/gateway/deploy up -d --no-build --wait --wait-timeout 600 --remove-orphans(plus--force-recreatewhen the checksum changed or--force-recreatewas passed).mitmproxystarts first, then the gateway daemon.--no-buildis explicit — the droplet never builds images. The 600s wait budget outlasts the workspace healthcheckstart_period(360s), which covers the one-time checkout ownership migration on first boot (default deadline 300s,WORKSPACE_MIGRATION_DEADLINE_MS). If the migration deadline is widened for a largeworkspace-reposvolume, raise thestart_periodand this wait together. - Registration poll —
GET /applications/{app_id}/guilds/{guild_id}/commandson the Discord API. Polls Discord registration with ~90s default budget (10 attempts × (3s interval + 6s per-attempt timeout); defaults fromapps/gateway/src/deploy.ts:344-368). 429 honorsRetry-Afterand doesn’t count against the attempt budget (each 429 retry adds up to 60s; pathological all-429 ceiling ~11 min). 401/403/404 abort immediately with token-sanitized errors. 5xx retries. - Running image digest verification — for each of
gatewayandworkspace, the deploy reads the running container’sRepoDigestsand asserts they match the CI-pushed digest. This confirms the droplet is running the GHCR artifact, not a stale or locally-built image. Throws if the digest does not match — deploy fails loudly. Manual verification:Terminal window docker inspect --format '{{json .RepoDigests}}' "$(docker inspect --format '{{.Image}}' "$(docker compose --project-directory /opt/gateway/deploy ps -q gateway)")"docker inspect --format '{{json .RepoDigests}}' "$(docker inspect --format '{{.Image}}' "$(docker compose --project-directory /opt/gateway/deploy ps -q workspace)")" - Workspace memory limit readback — the deploy resolves the workspace container with
docker compose ps -q workspace(exactly one ID, else it throws), then readsHostConfig.MemoryandHostConfig.MemorySwapfromdocker inspect. Both must equal 1024 MiB in bytes; any other value (empty,0,-1, a different limit, including one set by another compose file) throws before the checksum write with the expected and actual values. The read is single-shot with no retry. The stack stays running and the old checksum persists, so the next deploy recreates the containers. Success logs one line naming the limit. - Checksum write —
/opt/gateway/.secrets-checksumis written only after compose + registration + digest verification + memory limit readback all succeed. If any step fails mid-rotation, the old checksum persists and the next deploy force-recreates again.
A deploy that adds or changes the workspace limit changes the override, and so the checksum, which recreates every gateway service (gateway, mitmproxy, Caddy, workspace). In-flight runs are interrupted and must be cancelled by hand. Checkouts survive on the fro-bot_workspace-repos volume.
DAY-2 OPERATIONS
Section titled “DAY-2 OPERATIONS”- Monitoring — GitHub Actions deploy logs for the deploy run;
gateway statusfor live service states;gateway logs <service>to stream container output. - Roll back a bad repo change that does not touch the pin — revert the commit and dispatch
Deploy Gateway. The deploy is idempotent; containers either updated or didn’t. - Roll back an upstream pin bump — see Rollback is not pin reversal. Never revert only
apps/gateway/upstream.json. - Roll back a bad secret rotation — restore the previous value in the
gatewayGitHub Environment and redeploy. The checksum-after-success invariant means a failed rotation leaves the old checksum in place, so the next deploy will force-recreate — but the credential value must be rolled back manually first. - A failed deploy does not destroy the live droplet — containers either updated or didn’t. Fix the underlying issue and retry; the deploy is safe to re-run.
- Scaling or region change — provision a new droplet, update
GATEWAY_HOSTin the GitHub Environment, commit the updated.github/known_hosts, and trigger a deploy. Decommission the old droplet after verifying the new one is healthy.
Workspace memory
Section titled “Workspace memory”The workspace limit is 1024 MiB (sizing in Deploy flow step 5). The Workspace memory block of gateway status shows the limit, current usage, max events (times usage hit the limit), OOM kills, restarts with the container start time, OOMKilled, and the OpenCode supervisor state (starting, ready, down, degraded, skipped, or unknown). The OOM kill count is since the container last started: it resets whenever the container is recreated or restarted. The block is display-only; the status line and exit code follow compose state, and a failed or timed-out probe reports the whole block as unknown. gateway status --json prints one line, {ok, services, workspace}, with the same data; the unified status --json envelope carries it as facts.workspace on the gateway entry, with a workspace-memory resource observation alongside.
opencode: skipped— the supervisor probe was deliberately not run, because the workspace has no limit or is within 64 MiB of it, so the probe cannot cause a kill. The display says which.unknownmeans the probe ran (or its outcome could not be established) and gave no usable answer.degradedworkspace — recover withcd /opt/gateway/deploy && docker compose restart workspace.- Run interrupted by an OOM kill — the run stays
runningand must be cancelled from its card. - Checkout left dirty by a kill — check it read-only with
cd /opt/gateway/deploy && docker compose exec -u 10001 workspace git -c core.fsmonitor=false -C /workspace/repos/<owner>/<repo> status --porcelain. Remove files only with operator approval, also as UID10001inside the container.
Rollback is not pin reversal
Section titled “Rollback is not pin reversal”Workspace checkout ownership migrates one-way: on first boot the workspace entrypoint hands every owner/repo checkout on the workspace-repos volume to uid 10001. An older workspace image runs git as root, which refuses to operate on those checkouts (“dubious ownership”). Reverting upstream.json and redeploying therefore leaves the old stack unable to use existing checkouts.
Rollback means restoring a pre-upgrade snapshot of the workspace-repos volume and redeploying the old gateway and workspace images together (they share the workspace bearer contract; a mismatch yields HTTP 401 / workspace-unavailable). Take the snapshot before any pin bump that crosses the uid migration. Upstream’s alternative is chowning the checkout trees back to root — see deploy/README.md in fro-bot/agent, “Workspace uid isolation” → Rollback. Never revert only the pin.
The ownership migration ran at the v0.118.2 cutover. Rolling back between releases at or after v0.118.2 is a paired pin reversal with the dashboard and needs no volume restore; the snapshot requirement applies only when a rollback crosses the migration. The target must also be at or after v0.130.1 (see Revert to a previous gateway and workspace pair).
Rolling back to a release without durable operator sessions leaves the session records in operator-web/sessions/ in place, and the older daemon cannot revoke them on logout. To invalidate them, rotate GATEWAY_OPERATOR_CSRF_SECRET or delete operator-web/sessions/*.
Revert to a previous gateway and workspace pair
Section titled “Revert to a previous gateway and workspace pair”The gateway and workspace run OpenCode V2 only. apps/gateway/upstream.json holds the current pin, and the workspace image carries the OpenCode version in that release’s opencode-v2-pin.json. There is no OpenCode generation to select or fall back to: fro-bot/agent removed the V1 path in v0.139.0, the workspace Dockerfile refuses any other OPENCODE_VERSION, and the gateway rejects a run unless the workspace /readyz reports an OpenCode V2 version. A workspace that reports no version or an older one is not ready.
The gateway and workspace roll back together, to a pair that can run V2. v0.130.1 is the floor: it is the oldest release that provisions provider credentials into each V2 spawn, and on v0.130.0 every model call fails with 401. Releases before v0.130.1 have no working V2 path and are not rollback targets. Before reverting, confirm no run is in flight: recreating the containers interrupts runs, which must then be cancelled by hand. V2 keeps no OpenCode session state across workspace restarts (it recreates its database on every spawn), so a mention thread loses OpenCode context at the revert.
To revert, use either path. Both ship the gateway and workspace together, and in both the pin and the images must describe the same upstream ref.
- Previous images (fast, byte-identical).
gateway deploy --localtakes the images only fromGATEWAY_IMAGE_DIGESTandWORKSPACE_IMAGE_DIGEST, but takes the upstream ref fromapps/gateway/upstream.jsonin the checkout it runs from, and resets the droplet’s/opt/gatewayclone (the source of the Compose files and scripts) to that ref (resolveUpstreamPinand the source-materialization phase inapps/gateway/src/deploy.ts). Nothing checks the digests against the ref, and the command has no flag for either, so the operator makes them agree. In order:- Provenance of the pair. In the last successful
deploy-gatewayrun before the bump, take the gateway and workspace digests from itsbuild-imagesoutputs (that run’s digest readback verified them on the droplet) and the tags it pushed,<ref>-<sha>:<ref>is the upstream ref,<sha>is the infra commit the run built from. - Deploy files for the same ref. Check out infra commit
<sha>, soupstream.jsonand the deploy code match the pair, and run the deploy from that checkout (the root.envis needed there). Editing onlyupstream.jsonat the current commit leaves the current deploy code, which has not been checked against the older ref. - Deploy. With
GATEWAY_IMAGE_DIGESTandWORKSPACE_IMAGE_DIGESTset to the pair, runbunx @marcusrbrown/infra gateway deploy --local. No rebuild is needed. - Verify after the deploy. The live
RepoDigestsmust equal the recorded pair (command in Droplet: deploy-gateway job step 11). The deploy’s own readback checks the digests it was given, not where they came from, so step 1 is the only provenance check.
- Provenance of the pair. In the last successful
- Revert the commit. Reverting the pin-bump commit restores
upstream.jsonto the previous ref, andbuild-imagesrebuilds that ref with theOPENCODE_VERSIONfrom its own pin file, so the pin and images agree by construction. The digests may differ from the previous pair (see CI: build-images job). A rebuild of a ref older than the current pin has not been verified against the current deploy code; prefer the recorded pair.
No workspace-repos volume restore is needed while both sides stay at or after v0.118.2 (see Rollback is not pin reversal). The previous run’s digests must serve the operator contract the pinned dashboard requires.
CLI COMMANDS
Section titled “CLI COMMANDS”| Command | Purpose |
|---|---|
bunx @marcusrbrown/infra gateway status |
SSH to droplet, run docker compose ps, show service states + ages + healthchecks, then a workspace memory block (limit, usage, OOM kills, restarts, OpenCode state). --json prints one {ok, services, workspace} line instead. |
bunx @marcusrbrown/infra gateway deploy |
Trigger the deploy workflow via gh workflow run (remote, default). --local runs the deploy script directly (requires SSH_AUTH_SOCK). --dry-run prints the plan without side effects. |
bunx @marcusrbrown/infra gateway logs <service> [--tail N] |
Stream docker compose logs from the droplet. Services: gateway, workspace, mitmproxy. --tail defaults to 50. Logs surface unredacted; operator-only via SSH boundary. CI/headless contexts require explicit --allow-ci to discourage scripted log capture. |
bunx @marcusrbrown/infra gateway backup [--output FILE] [--include-ca] |
Pull the mitmproxy CA cert + key from the fro-bot_mitmproxy-certs named volume as a tarball (errors if the volume does not exist). Local file created with mode 0600 via O_EXCL|O_CREAT (no chmod race). --include-ca is required (only CA backup is currently implemented). |
bunx @marcusrbrown/infra gateway restore --input FILE [--include-ca] |
Validate the tarball locally (must contain exactly mitmproxy-ca-cert.pem + mitmproxy-ca.pem; otherwise rejected before any remote mutation), upload to an unguessable mktemp path, extract into the named volume, restart mitmproxy + gateway, byte-equal confirm restored cert + key match the input archive. |
ONE-TIME PROVISIONING
Section titled “ONE-TIME PROVISIONING”Prerequisites:
DIGITALOCEAN_ACCESS_TOKENandGATEWAY_HOSTin.env;doctl auth initrun locally.GATEWAY_HOSTis the FQDN used for host-key pinning (e.g.gateway.fro.bot) — DNS does not need to point at the new droplet yet.- Discord application created at https://discord.com/developers/applications with bot scope; token + application ID + guild ID captured
- S3 or R2 bucket created;
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,S3_BUCKET,S3_REGIONcaptured (addS3_ENDPOINTfor R2/MinIO); IAM identity granted the actions in S3 State-Bucket IAM Actions gatewayGitHub Environment created with required reviewer set- All required secrets seeded in the
gatewayGitHub Environment (see Required Secrets below)
Run:
bun run provision:gateway(Root wrapper — loads the repo-root .env; --cwd apps/gateway would miss it.)
The script will:
- Confirm
doctlis authenticated - Reject if a
gatewaydroplet already exists (aborts;--forceto override) - Create an
s-1vcpu-2gbdroplet innyc1, taggedgateway - Append unhashed domain host keys and hashed IP host keys to
.github/known_hosts(ssh-keyscan <domain>+ssh-keyscan -H <ip>) - Print operator setup steps for finalizing the GitHub Environment
SSH auth during provisioning: when GATEWAY_SSH_KEY is set, the script materializes it to a 0600
temp key file and pins it with -i + IdentitiesOnly=yes (no ssh-agent needed; cleaned up after).
When unset, it falls back to ssh-agent.
After provisioning: commit the updated .github/known_hosts.
S3 STATE-BUCKET IAM ACTIONS
Section titled “S3 STATE-BUCKET IAM ACTIONS”The fro-bot-gateway identity (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) owns the state bucket: it is the only principal with access, and the bucket has no bucket policy. The grant is the managed policy fro-bot-gateway-s3-access, which is the source of truth for the action set below. It is scoped to the whole bucket rather than a prefix, so it covers the run-state prefix and the operator session keys under operator-web/ alike. Grant it before seeding the secrets in Required Secrets.
Object-level (resource arn:aws:s3:::<bucket>/*):
| Action | Why |
|---|---|
s3:GetObject |
Read run-state objects |
s3:PutObject |
Write run-state objects |
s3:PutObjectTagging |
Required alongside s3:PutObject — run-state writes send an x-amz-tagging header (object-type=run-state), and AWS requires both actions to accept a tagged PutObject. A missing grant rejects every createRun with AccessDenied on both the Discord @fro-bot mention path and the dashboard launch path — this is not a dashboard-only dependency. Call sites in upstream fro-bot/agent: packages/runtime/src/coordination/run-state.ts (create, transition) and packages/runtime/src/coordination/heartbeat.ts. |
s3:DeleteObject |
Remove run-state objects |
s3:GetObjectTagging |
Read an object’s tags, so an operator can confirm the object-type=run-state tag landed with the same credential |
s3:GetObjectVersion |
Read a specific object version; harmless while the bucket is unversioned |
Bucket-level (resource arn:aws:s3:::<bucket>):
| Action | Why |
|---|---|
s3:ListBucket |
List run-state and operator session keys |
s3:GetBucketLocation |
Resolve the bucket’s region |
Durable operator sessions keep one signed record per session under operator-web/sessions/ at the bucket root, and the startup self-test writes under operator-web/_self-test/. These writes are untagged. If a narrower grant ever leaves these keys out, the gateway still starts but operator logins return 503 until the grant is fixed and the gateway is restarted. Keep the Action’s S3 principal on its own bucket so nothing else can write session records.
Lifecycle configuration is operator-owned, not provisioning-owned. A bucket lifecycle rule is bucket-level AWS state, independent of the droplet — destroying and rebuilding the droplet never touches it, and it is set-once configuration rather than per-provision or per-deploy state. apps/gateway/server/provision-droplet.ts does not apply it. The fro-bot-gateway runtime identity intentionally does not hold s3:GetLifecycleConfiguration / s3:PutLifecycleConfiguration, and should not be granted them — the only candidate for holding that bucket-administration power permanently is a credential that lives on the droplet and is used by the running daemon, which is a bad trade for automating a change that happens approximately once.
ensureRunStateLifecycleRule (exported from apps/gateway/server/provision-droplet.ts) remains available as a correct, idempotent helper for an operator to run by hand with a privileged, provisioning-scoped credential — never the runtime fro-bot-gateway identity.
The rule’s canonical shape — RUN_STATE_LIFECYCLE_RULE in apps/gateway/server/provision-droplet.ts is the source of truth, referenced here rather than duplicated:
- ID:
run-state-30d-expiration - Filter:
Tag {Key: object-type, Value: run-state} - Status:
Enabled - Expiration:
Days: 30
Why the filter must stay tag-scoped, not prefix-scoped: the same <prefix> also holds durable Discord channel/repo bindings (bindings/repo.json and bindings/by-channel/<id>.json) that must never expire. A prefix-wide rule would delete them and silently unbind every repository. The tag filter — matching only objects the daemon explicitly tags object-type=run-state — is what makes a retention rule safe to have at all.
Operator procedure — read first, merge, then put. PutBucketLifecycleConfiguration REPLACES the bucket’s entire lifecycle configuration. A blind PUT silently destroys any other rules present (e.g. a NoncurrentVersionExpiration rule covering the IAM probe prefix, see the versioning caveat below). Never PUT without reading first:
# 1. Read the current configuration. NoSuchLifecycleConfiguration means there is# nothing to merge — proceed straight to writing just the run-state rule.aws s3api get-bucket-lifecycle-configuration --bucket <bucket>
# 2. Merge: keep every existing rule whose ID is not run-state-30d-expiration,# then add/replace the run-state-30d-expiration rule with the canonical shape above.
# 3. Put the merged document (never a bare/blind PUT of just the new rule).aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://merged.jsonOr invoke ensureRunStateLifecycleRule(env) directly with a privileged credential — it already performs this read-check-write-readback sequence correctly and merges rather than replaces.
IAM policy management is out-of-band; tagged-write capability is checked at deploy time. No code here manages or inspects the fro-bot-gateway identity’s IAM policy. The gateway deploy verifies the runtime identity’s tagged-write and probe-cleanup capabilities through the fail-closed preflight described below; it does not audit every grant in the tables above. For a manual capability check, use a tagged write, since an untagged write does not exercise s3:PutObjectTagging:
aws s3api put-object --bucket <bucket> --key <prefix>/_iam-probe/<uuid>.json \ --body /dev/null --tagging 'object-type=run-state'aws s3api delete-object --bucket <bucket> --key <prefix>/_iam-probe/<uuid>.jsonThe probe key must sit inside the same resource scope (<bucket>/<prefix>/*) the real run-state objects use — a probe outside a narrowly scoped grant’s prefix produces a false negative. Cleanup needs s3:DeleteObject.
The gateway deploy runs this same probe automatically as a fail-closed preflight (assertS3TaggedWriteCapability in apps/gateway/src/deploy.ts), before any remote mutation and before containers are replaced. It is skipped under --dry-run, and skipped with an explicit logged reason when S3_ENDPOINT is set, because upstream suppresses object tagging entirely for non-AWS endpoints.
Versioning caveat: on a versioning-enabled bucket, deleting the probe writes a delete marker and retains a noncurrent version, so one small object accumulates per deploy. The runtime identity cannot read bucket versioning state (no s3:GetBucketVersioning), so neither the probe nor the deploy can detect this. If the state bucket is versioned, cover the probe prefix with a NoncurrentVersionExpiration lifecycle rule rather than granting s3:DeleteObjectVersion to the runtime identity.
REQUIRED SECRETS
Section titled “REQUIRED SECRETS”| Secret | Required | Description |
|---|---|---|
GATEWAY_SSH_KEY |
✓ | Ed25519 private key for the gateway.fro.bot droplet |
DISCORD_TOKEN |
✓ | Discord bot token |
DISCORD_APPLICATION_ID |
✓ | Discord application ID |
DISCORD_GUILD_ID |
✓ | Discord guild (server) ID |
AWS_ACCESS_KEY_ID |
✓ | S3/R2 access key |
AWS_SECRET_ACCESS_KEY |
✓ | S3/R2 secret key |
S3_BUCKET |
✓ | Bucket name |
S3_REGION |
✓ | Bucket region |
GATEWAY_HOST |
✓ | FQDN or IP of the droplet |
GH_APP_ID |
✓ | GitHub App ID for /fro-bot add-project repo access (materializes to github-app-id) |
GH_APP_PRIVATE_KEY |
✓ | GitHub App private key PEM (materializes to github-app-private-key) |
WORKSPACE_OPENCODE_TOKEN |
✓ | Internal shared bearer between the gateway and the workspace :9200 OpenCode proxy (random value; fail-closed if empty) |
WORKSPACE_OPENCODE_AUTH |
✓ | OpenCode provider auth.json for the workspace — a dedicated scoped cliproxy key, not the repo’s OPENCODE_AUTH_JSON (separate revocation + blast radius) |
WORKSPACE_OPENCODE_MODEL |
✓ | OpenCode model id for the mention loop (e.g. openai/gpt-5.5). MUST be a real cliproxy /v1/models catalog id — an unlisted id (e.g. gpt-5.5-fast) 502s and OpenCode retries silently, yielding an empty run. Written to .env |
WORKSPACE_OPENCODE_CONFIG |
✓ | OpenCode provider/baseURL config JSON routing through cliproxy.fro.bot/v1; JSON-validated, $-escaped, written to .env |
GATEWAY_TRIGGER_ROLE_ID |
✓ | Discord role ID allowed to trigger the @fro-bot mention loop — required (fail-closed); empty would open LLM execution to every guild member |
S3_ENDPOINT |
— | Custom endpoint URL (R2, MinIO, etc.) |
WORKSPACE_OPENCODE_URL |
— | Override the workspace OpenCode proxy URL (default http://workspace:9200) |
OBJECT_STORE_HOSTS |
— | Comma-separated hostnames the mitmproxy egress filter allows through to S3 |
WORKSPACE_OPENCODE_READY_TIMEOUT_MS |
— | Workspace OpenCode supervisor ready timeout in ms (1–600000; default: 60000). Absent or empty uses the upstream default. Whitespace-only, non-numeric, zero, negative, non-integer, and above-max values are rejected before any SSH. Written to .env and wired into the workspace container environment via the compose override. |
DISCORD_PRIVILEGED_INTENTS |
— | Opt-in privileged intents (e.g. MessageContent); materializes to discord-privileged-intents, empty = baseline intents |
GATEWAY_WEBHOOK_SECRET |
opt-in† | HMAC signing key for the announce webhook — strong random value; materialized via SSH stdin, never argv. Set together with GATEWAY_PRESENCE_CHANNEL_ID to enable the announce endpoint; leave both unset to keep the gateway outbound-only. Setting exactly one fails the deploy before any SSH. |
GATEWAY_PRESENCE_CHANNEL_ID |
opt-in† | Discord channel ID where the daemon posts presence embeds as the Fro Bot user. Set together with GATEWAY_WEBHOOK_SECRET. |
GATEWAY_BINDER_ALLOWLIST |
— | Discord user IDs allowed to run /add-project. A gateway Environment variable (read via vars.), not a secret. One ID per line; commas and spaces also work, blank lines and full-line # comments are ignored. Each entry must be a Discord snowflake (17–20 digits); a malformed entry fails the deploy before any SSH (--dry-run included) and names the bad line. Written on every deploy to gateway-binder-allowlist, normalized to one ID per line; unset or empty writes an empty file and /add-project then refuses everyone. A change alters the secrets checksum, so the gateway is recreated. |
GATEWAY_OPERATOR_BIND_HOST |
opt-in‡ | Gateway-net IPv4 address for the operator listener bind (e.g. 172.21.0.2). Rejects 0.0.0.0, loopback, sandbox-net (10.*), 172.20.* (the known conflicted Docker subnet from the failed operator deploy), and IPv6. |
GATEWAY_OPERATOR_BIND_PORT |
opt-in‡ | Port for the operator listener (e.g. 9300). Must be a positive integer in [1, 65535]. |
GATEWAY_OPERATOR_PUBLIC_ORIGIN |
opt-in‡ | HTTPS origin for the browser-visible operator API. The ratified value is https://dashboard.fro.bot — set this for production use. Must be a bare HTTPS origin (no path, query, hash, credentials, or non-default port). |
GATEWAY_VPC_IP |
opt-in§ | Gateway droplet’s DigitalOcean VPC IP (e.g. 10.116.0.3). Required for the VPC-IP-scoped operator port publish and DOCKER-USER rule. All-or-none with DASHBOARD_VPC_IP and the operator listener trio. |
DASHBOARD_VPC_IP |
opt-in§ | Dashboard droplet’s DigitalOcean VPC IP (e.g. 10.116.0.5). Required whenever the operator listener is enabled: it is the DOCKER-USER source restriction and the derived GATEWAY_OPERATOR_TRUSTED_PROXIES value. Must be an exact IPv4 address (not unspecified, multicast, CIDR, hostname, or IPv6). Deploy fails before any SSH if absent or invalid. |
GATEWAY_OPERATOR_TRUSTED_PROXIES |
derived | Not an input. The deploy sets it in the gateway service environment of compose.override.yaml to DASHBOARD_VPC_IP (plain env entry, not a _FILE secret) whenever the operator listener is enabled; omitted otherwise. The daemon refuses to start without it. |
DIGITALOCEAN_ACCESS_TOKEN |
provisioning-only | DigitalOcean API token used by provision-droplet.ts to create the gateway-operator-fw Cloud Firewall. Not a deploy secret — not required in the gateway GitHub Environment. |
GATEWAY_IMAGE_DIGEST |
CI-injected | sha256:<digest> of the ghcr.io/marcusrbrown/infra-gateway image pushed by the build-images job. Threaded from needs.build-images.outputs.gateway_digest. Required for the deploy to pin and verify the running image. For a local/break-glass deploy, supply manually (see Break-glass runbook). |
WORKSPACE_IMAGE_DIGEST |
CI-injected | sha256:<digest> of the ghcr.io/marcusrbrown/infra-workspace image pushed by the build-images job. Threaded from needs.build-images.outputs.workspace_digest. Required for the deploy to pin and verify the running image. For a local/break-glass deploy, supply manually. |
†Both-or-neither: set both to enable the announce/presence webhook; set neither to disable. Setting exactly one is an error — the deploy fails fast with a message naming the missing input, before any SSH connection is made.
‡All-or-none: set all three to enable the operator listener; set none to disable. Setting one or two is an error — the deploy fails fast before any SSH. See Operator Listener for bind host/port/origin constraints.
§Required with the operator listener trio: GATEWAY_VPC_IP and DASHBOARD_VPC_IP must both be set whenever the listener trio is set; the deploy fails before any SSH otherwise. With the listener trio unset, both are ignored. DIGITALOCEAN_ACCESS_TOKEN is a provisioning-only concern (used locally by provision-droplet.ts to create the Cloud Firewall) — it is not part of this all-or-none group and is not required in the gateway GitHub Environment. See Operator private path for the full bridge topology.
OpenCode Go through the proxy
Section titled “OpenCode Go through the proxy”The workspace runs an OpenCode Go model only through cliproxy.fro.bot. Three values select it:
WORKSPACE_OPENCODE_MODEL=opencode-go/<id>—<id>is a bare id listed by cliproxy/v1/modelswithowned_by: opencode-go.WORKSPACE_OPENCODE_CONFIG— carriesprovider["opencode-go"].options.baseURLset tohttps://cliproxy.fro.bot/v1.WORKSPACE_OPENCODE_AUTH— carries anopencode-goentry holding the dedicated scoped cliproxy key.
The deploy guard checks the base URL of the provider named by the model prefix, so it rejects a Go model whose base URL points at OpenCode directly or whose opencode-go provider block is missing. The egress allowlist gains no OpenCode inference host. apps/gateway/src/deploy.test.ts pins this behavior.
GHCR IMAGES
Section titled “GHCR IMAGES”The gateway and workspace images are published to:
ghcr.io/marcusrbrown/infra-gateway— built fromfro-bot/agentdeploy/gateway.Dockerfileghcr.io/marcusrbrown/infra-workspace— built fromfro-bot/agentdeploy/workspace.Dockerfile
Both packages are public. The upstream Dockerfiles were audited before first publish: all secrets are runtime bind-mounts into /run/secrets/... — no ARG/ENV/COPY of secret material is baked into the image layers. The source (fro-bot/agent) is itself public. Public visibility means the droplet pulls with no authentication required.
Tags (:<ref>) are pushed for human readability. The deploy pins and verifies by digest (@sha256:<digest>), not by tag — GHCR tags are mutable, so tag-based verification would be circular. Digest pull is immutable and makes the running-image check a true identity assertion.
The build-images CI job holds packages: write permission (job-scoped only). The deploy-gateway job is package-read-only.
BREAK-GLASS RUNBOOK
Section titled “BREAK-GLASS RUNBOOK”The deploy depends on CI and GHCR availability. If CI or GHCR is unavailable during an incident and the droplet needs a redeploy:
-
Build and push from a workstation (which has ample RAM — this is the off-droplet build done by hand):
Terminal window # Check out fro-bot/agent at the pinned refREF=$(jq -r .ref apps/gateway/upstream.json)git clone --depth 1 --branch "$REF" https://github.com/fro-bot/agent.git /tmp/fro-bot-agent# Log in to GHCRecho "$GITHUB_TOKEN" | docker login ghcr.io -u <your-github-username> --password-stdin# Build and push gateway image; capture the digestGATEWAY_DIGEST=$(docker buildx build \--platform linux/amd64 \--file /tmp/fro-bot-agent/deploy/gateway.Dockerfile \--push \--iidfile /tmp/gateway-iid.txt \--metadata-file /tmp/gateway-meta.json \/tmp/fro-bot-agent \&& jq -r '."containerimage.digest"' /tmp/gateway-meta.json)# Build and push workspace image; capture the digestWORKSPACE_DIGEST=$(docker buildx build \--platform linux/amd64 \--file /tmp/fro-bot-agent/deploy/workspace.Dockerfile \--push \--metadata-file /tmp/workspace-meta.json \/tmp/fro-bot-agent \&& jq -r '."containerimage.digest"' /tmp/workspace-meta.json) -
Run the local deploy with the digests you just captured:
Terminal window GATEWAY_IMAGE_DIGEST="$GATEWAY_DIGEST" \WORKSPACE_IMAGE_DIGEST="$WORKSPACE_DIGEST" \bunx @marcusrbrown/infra gateway deploy --local
Never run docker compose up --build on the droplet. The compose.override.yaml image pins make GHCR the source of truth regardless, but building on the 1vCPU/2GB droplet exhausts RAM and thrashes swap — this is the failure mode the off-droplet build flow exists to prevent. The on-droplet build fallback is intentionally removed; the workstation build above is the correct emergency path.
The deploy depends on CI + GHCR availability + pull working. This is a deliberate trade: the on-droplet build is the hazard being removed, so keeping it as a fallback would keep the hazard.
GitHub App (/fro-bot add-project)
Section titled “GitHub App (/fro-bot add-project)”The gateway authenticates to GitHub as a public App (owned by the fro-bot account, contents: read only) to clone repos for /fro-bot add-project owner/repo. Installation tokens are scoped per-installation, so the App must be installed on every account add-project targets — that is why it is public, not private. The App ID + PEM live only in the gateway GitHub Environment and on the droplet; they never enter the repo. Naming: the Environment secrets are GH_APP_ID / GH_APP_PRIVATE_KEY (GitHub rejects GITHUB_-prefixed names); they materialize to the upstream-fixed files github-app-id / github-app-private-key, which the daemon reads as its in-container GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY. Rotation/revocation: generate a new private key in the App settings, replace GH_APP_PRIVATE_KEY in the gateway Environment, re-run the deploy (re-materializes the file), then delete the old key. To contain a compromise, revoke the App installation on affected accounts (or suspend the App) and rotate the PEM. The App identity (name, logo, docs page, install-URL default) is owned by fro-bot/agent, not this repo. See docs/runbooks/github-app-key-lifecycle.md for the full rotation/revocation runbook.
Mention loop (@fro-bot → OpenCode)
Section titled “Mention loop (@fro-bot → OpenCode)”The workspace executor runs OpenCode against cliproxy.fro.bot when an authorized user @-mentions the bot. Trust model: the workspace runs on sandbox-net (internal: true — no direct external egress; all outbound forced through the mitmproxy allowlist), so a prompt-injection payload in a Discord message or cloned repo cannot reach an arbitrary destination. GATEWAY_TRIGGER_ROLE_ID bounds who can trigger a run (and thus spend); it is enforced non-empty at deploy time (REQUIRED_ENV_VARS in deploy.ts) even though the upstream daemon treats it as readOptionalSecret — this repo’s deploy is stricter (fail-closed). WORKSPACE_OPENCODE_AUTH is a dedicated scoped cliproxy key so it can be revoked independently of CI’s OPENCODE_AUTH_JSON. GATEWAY_WEBHOOK_SECRET and GATEWAY_PRESENCE_CHANNEL_ID are opt-in — set both in the gateway GitHub Environment to enable the announce/presence webhook (see Announce/presence webhook below). Honest limit: an authorized role-holder is still an untrusted-content source — containment bounds blast radius, it does not prevent a determined authorized user from misusing the agent within the allowed surface. Verify post-deploy with the release gates, and with a real authorized mention when the Discord pass runs (Discord command registration alone does not prove the workspace works); if broken, follow Rollback is not pin reversal — do not just revert upstream.json.
The deploy emits WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.ai. cliproxy.fro.bot is the LLM proxy the workspace routes all model traffic through; without it the @-mention loop gets no model responses. models.dev and models.opencode.ai are OpenCode model-catalog hosts, kept allowlisted but currently unused: the workspace OpenCode V2 server runs with OPENCODE_DISABLE_MODELS_FETCH=1 (upstream apps/workspace-agent/src/opencode-server-v2.ts), so it never fetches the catalog and uses its bundled snapshot. The /fro-bot add-project clone path is unaffected (it does not invoke OpenCode).
Post-cutover verification ritual
Section titled “Post-cutover verification ritual”After deploying a new daemon version, run the checks below. The acceptance procedure for a new upstream ref is the release gates: they run on every new upstream ref and need no Discord action, and the Discord checks form a separate pass that runs only when it is triggered. The steps below add what the gates do not cover.
Always, in order — each must pass before the next:
bunx @marcusrbrown/infra gateway status— core services (gateway,workspace,mitmproxy) must show healthy/running; on announce-enabled stacks,caddymust also appear (running or healthy — Caddy has no healthcheck, sorunning/n-ais normal).bunx @marcusrbrown/infra gateway logs workspace --tail 100— confirm the workspace container started without errors (look for the Hono server listening on:9100and:9200).- Release gates — run the automated gates per the runbook. A dashboard-launched run proves workspace execution without Discord, and gate 5 covers approvals: a rejected approval lets the run continue, and the command does not run.
- workspace-repos volume — after a gate run on a repo, run
bunx @marcusrbrown/infra gateway logs workspace --tail 50and confirm the workspace did not re-clone the repo from scratch (it should use the existing checkout from theworkspace-reposvolume). On first deploy, the volume is auto-created and the repo is cloned fresh — subsequent runs reuse it.
Only when the Discord pass runs (or, for the clone smoke, after changing the GitHub App credentials):
/fro-bot add-projectclone smoke — a Discord slash command, not part of the gates. Run it against a repo the GitHub App is installed on; confirm the clone completes without error. This exercises the GitHub App credentials and the workspace/cloneendpoint end-to-end.- Authorized
@fro-botmention round-trip — DP.1 of the Discord pass, from an account holdingGATEWAY_TRIGGER_ROLE_ID. It is the only proof that Discord intake and replies work; the Approve/Deny embeds and their attribution are checked in the same pass (gate 4). Discord command registration (step 10 of the droplet deploy flow) does not prove the workspace works — it only proves the daemon registered slash commands.
@mention-loop behavior:
- A mention from an authorized user (holding
GATEWAY_TRIGGER_ROLE_ID) triggers an OpenCode run in the workspace. - If the run needs a tool that requires approval, the gateway posts an Approve/Deny embed in the run thread. The run pauses until a button-click or the 13-minute deadline fires (fail-closed).
- Only authorized users can approve. Unauthorized approval attempts are silently ignored.
- If the gateway or workspace restarts while a prompt is open, the prompt is abandoned — the run does not resume.
- Cloned repos live in the
workspace-reposnamed volume at/workspace/repos. The volume persists repo contents across container recreation and daemon upgrades. Missing repos are re-cloned automatically on the next mention. Never rundocker compose down -v— it destroys all cloned repos. Do not put secrets in repo contents — authorized runs may access the active workspace/repo surface. To fully clear a cloned repo, delete the volume and let the next mention re-clone it. WORKSPACE_EGRESS_HOSTS=cliproxy.fro.bot,models.dev,models.opencode.aiis required for the workspace to reach the LLM proxy; the two catalog hosts stay allowlisted but are currently unused (see above). Withoutcliproxy.fro.bot, the mention loop gets no model responses.- The workspace bash approval policy is owned by deploy code (
WORKSPACE_PERMISSION_POLICYinapps/gateway/src/deploy.ts) and injected intoWORKSPACE_OPENCODE_CONFIGat deploy time, overwriting anypermissionblock the secret carries. Destructive commands — file deletion (rm,rmdir), force-push/hard-reset/clean,sudo,chmod/chown, exfil-shapedcurl/wget, and package installs — route through the Discord Approve/Deny gate; ordinary commands run autonomously.
ANNOUNCE/PRESENCE WEBHOOK
Section titled “ANNOUNCE/PRESENCE WEBHOOK”The daemon’s POST /v1/announce endpoint is opt-in. When enabled, it accepts HMAC-signed requests and posts a Discord embed as the Fro Bot user to the configured presence channel. The intended caller is the fro-bot/.github control plane (separate work, not in this repo).
Enabling
Section titled “Enabling”Set both GATEWAY_WEBHOOK_SECRET (a strong random HMAC key) and GATEWAY_PRESENCE_CHANNEL_ID (the target Discord channel ID) in the gateway GitHub Environment, then trigger a deploy. The deploy materializes both as secret files on the droplet and writes a compose.override.yaml that:
- Adds a Caddy reverse proxy service publishing
:80/:443ongateway-net, terminating TLS forgateway.fro.botvia Let’s Encrypt auto-cert. - Path-scopes Caddy to
/v1/announceand (when operator is enabled)/operator/*— all other paths return 404. - Wires the daemon’s announce
*_FILEenv entries and secret mounts via the override’sgatewayservice block.
The override is a working-tree file re-materialized on every deploy when the inputs are present. The override contents are included in the secrets checksum, so toggling announce on or off forces --force-recreate on the next deploy.
Security posture
Section titled “Security posture”- Auth: the daemon verifies the HMAC signature, enforces a replay cache, and applies a rate limiter. Caddy adds TLS termination.
- No IP allowlist: GitHub Actions egress ranges are too dynamic to pin reliably. HMAC + TLS is the full auth boundary.
- Secret materialization:
GATEWAY_WEBHOOK_SECRETis written to the droplet via SSH stdin only — never via argv. - Path isolation: Caddy uses a
handle /v1/announce { ... }block with a catch-allhandle { respond 404 }. The catch-all does not shadow ACME challenge paths (Caddy uses TLS-ALPN-01 on:443by default, which does not touch HTTP path routing).
Rollback / disabling
Section titled “Rollback / disabling”Remove both GATEWAY_WEBHOOK_SECRET and GATEWAY_PRESENCE_CHANNEL_ID from the gateway GitHub Environment and redeploy. The compose.override.yaml is always rewritten on deploy — when announce is disabled, the override carries only the image digest pins (no Caddy/announce sections). git clean -xfd removes the Caddyfile (a working-tree file written only when announce is enabled); --remove-orphans retires the now-undeclared Caddy container; the checksum flip (override bytes changed, Caddyfile gone) triggers --force-recreate. After the deploy, verify with docker compose config on the droplet — the Caddy service must not appear.
Caddy volume guardrail
Section titled “Caddy volume guardrail”Caddy stores TLS certs in a named Docker volume (caddy_data). Docker volumes are independent of the git working tree — git clean -xfd and normal deploys never touch them. The real cert-loss risk is an explicit docker compose down -v or docker volume prune on the gateway project, which would wipe the certs and force Let’s Encrypt re-issuance (rate-limit risk). Never run docker compose down -v or docker volume prune on the gateway stack. See the ANTI-PATTERNS section.
Rotation
Section titled “Rotation”If GATEWAY_WEBHOOK_SECRET leaks, the blast radius is Fro-Bot-user impersonation into the presence channel. Rotate by setting a new value in the gateway GitHub Environment and redeploying. Coordinate with the fro-bot/.github caller, which holds the matching secret on its side. The deploy re-materializes the secret file via SSH stdin; the checksum flip force-recreates the daemon container.
Post-enable verification
Section titled “Post-enable verification”After the first enabling deploy:
bunx @marcusrbrown/infra gateway status— confirm thecaddyservice appears alongsidegateway,workspace,mitmproxy(running or healthy — Caddy has no healthcheck, sorunning/n-ais normal).- Send a correctly HMAC-signed test POST to
https://gateway.fro.bot/v1/announce— confirm a Discord embed appears in the presence channel. - Send an unsigned or replayed POST — confirm a 401/403 response.
- Send a request to any other path (e.g.
https://gateway.fro.bot/) — confirm a 404 response.
For verifying a real control-plane go-live end-to-end (live log monitoring, success-field criteria, and distinguishing the real signed event from test probes), see docs/runbooks/gateway-announce-event-verification.md.
Implementation pointers
Section titled “Implementation pointers”buildSecretFileList, getAnnounceState, buildComposeOverride, buildCaddyfile in apps/gateway/src/deploy.ts.
OPERATOR AUTH
Section titled “OPERATOR AUTH”The operator auth gate is part of the operator listener feature. When the operator listener trio is enabled, all four auth/config secrets are required. The deploy fails before any SSH if the listener trio is set but any of the four auth/config secrets is missing or partially set.
Required auth/config secrets
Section titled “Required auth/config secrets”| GitHub Environment secret | _FILE env var in compose override |
Description |
|---|---|---|
GATEWAY_OPERATOR_GITHUB_CLIENT_ID |
GATEWAY_OPERATOR_GITHUB_CLIENT_ID_FILE |
GitHub OAuth App client ID |
GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET |
GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET_FILE |
GitHub OAuth App client secret |
GATEWAY_OPERATOR_CSRF_SECRET |
GATEWAY_OPERATOR_CSRF_SECRET_FILE |
Strict base64url, no padding/newlines, ≥32 decoded bytes |
GATEWAY_OPERATOR_ALLOWLIST |
GATEWAY_OPERATOR_ALLOWLIST_FILE |
Newline-separated numeric GitHub user IDs; fail-closed if missing/empty/malformed |
All four are all-or-none with the operator listener trio. Setting one to three of the four fails the deploy before any SSH. Leaving all four unset when the listener trio is also unset is the disabled state.
Optional OAuth tuning vars
Section titled “Optional OAuth tuning vars”These are plain env vars (not file-backed). Set them in the gateway GitHub Environment if needed; leave unset to use upstream defaults.
| GitHub Environment secret | Default | Description |
|---|---|---|
GATEWAY_OPERATOR_OAUTH_ALLOWED_RETURN_PATHS |
/operator |
Comma-separated same-origin post-auth paths |
GATEWAY_OPERATOR_OAUTH_STATE_TTL_MS |
600000 |
OAuth state TTL in milliseconds |
GATEWAY_OPERATOR_OAUTH_MAX_OUTSTANDING_ATTEMPTS |
5 |
Max concurrent outstanding OAuth attempts |
These are transported as optional secrets in the workflow (not workflow_call.inputs) for consistency with the gateway deploy pipeline.
Callback URL
Section titled “Callback URL”The GitHub OAuth App must be configured with:
https://dashboard.fro.bot/operator/auth/github/callbackThis is a GitHub OAuth App (client ID/secret) — not the existing gateway GitHub App (GH_APP_ID). Register the callback URL in the GitHub Developer Portal before enabling operator auth. The deploy dry-run prints the expected callback URL for manual cross-check against the portal.
Trusted proxy (derived)
Section titled “Trusted proxy (derived)”GATEWAY_OPERATOR_TRUSTED_PROXIES is not a GitHub Environment input. The deploy derives it from DASHBOARD_VPC_IP and writes it as a plain environment entry in the gateway service block of compose.override.yaml when the operator listener is enabled. The daemon’s socket peer on :9300 for the dashboard path is exactly the dashboard droplet’s VPC IP (Docker DNAT keeps the source), so that single address is the trusted proxy. The gateway-net Caddy address is deliberately not trusted: gateway.fro.bot/operator/* stays untrusted and returns 400. The Phase 5d rendered-config gate asserts the merged gateway env carries exactly this value before docker compose up.
Session semantics
Section titled “Session semantics”Sessions are durable. Each session is a signed record in the object store under operator-web/sessions/ (grant: S3 State-Bucket IAM Actions) and lasts a fixed seven days from sign-in. A gateway restart (deploy, force-recreate, container crash) does not end sessions: the gateway restores them at startup and logs session restore finished with an auth.session.restore audit event. Logout ends only that session. To invalidate every session, rotate GATEWAY_OPERATOR_CSRF_SECRET or delete operator-web/sessions/*. If the session store grant is missing, logins return 503 until the grant is fixed and the gateway restarted.
Rollback path
Section titled “Rollback path”To disable operator auth/config wiring after deployment:
- Remove or clear both the listener trio and the four auth/config secrets from the
gatewayGitHub Environment:- Listener trio:
GATEWAY_OPERATOR_BIND_HOST,GATEWAY_OPERATOR_BIND_PORT,GATEWAY_OPERATOR_PUBLIC_ORIGIN - Auth/config secrets:
GATEWAY_OPERATOR_GITHUB_CLIENT_ID,GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET,GATEWAY_OPERATOR_CSRF_SECRET,GATEWAY_OPERATOR_ALLOWLIST
- Listener trio:
- Trigger a gateway deploy:
bunx @marcusrbrown/infra gateway deploy. - Approve the environment gate.
- The deploy detects the operator listener is disabled, skips all operator auth/config wiring, and runs
docker compose up --remove-orphans. - The gateway restarts with the operator listener disabled and serves no operator sessions. Session records stay in the object store; to invalidate them, rotate
GATEWAY_OPERATOR_CSRF_SECRETor deleteoperator-web/sessions/*. - Verify:
bunx @marcusrbrown/infra gateway statusshows all services healthy.
Note: Listener trio present + auth/config secrets absent is not a valid state — the deploy fails before any SSH. The only supported rollback is to clear both the listener trio and the four auth/config secrets together.
Post-enable verification
Section titled “Post-enable verification”After the first deploy with operator auth/config wiring:
-
Gateway-side liveness probe: Probe the listener directly via the droplet-local gateway-net address:
Terminal window curl -sf http://172.21.0.2:9300/operator/healthA
200 {"ok":true,"contractVersion":"<semver>"}response confirms the operator listener is up.⚠ Liveness probe warning: Do not use
https://gateway.fro.bot/operator/healthas a liveness probe — a global forwarded-header guard returns400 {"error":"bad request"}for any request whereX-Forwarded-Hostdoes not match thePUBLIC_ORIGINhost (dashboard.fro.bot); the public Caddy route forwardsHost: gateway.fro.bot, so it returns 400 by design; the gateway-net Caddy address is deliberately not a trusted proxy. The gateway-side liveness probe is the droplet-local direct probe above. The dashboard same-origin path is live:https://dashboard.fro.bot/operator/health→200 {"ok":true,"contractVersion":"<semver>"}is a valid external probe. The OAuth callback URL registration uses the dashboard origin (https://dashboard.fro.bot/operator/auth/github/callback) — this is a GitHub OAuth App setting, not a live HTTP probe target. -
Auth gate coarse check:
GET https://gateway.fro.bot/operator/returns400 {"error":"bad request"}by design — the forwarded-header guard rejects it because the forwarded host (gateway.fro.bot) does not match thePUBLIC_ORIGINhost (dashboard.fro.bot). Do not usegateway.fro.bot/operator/as an auth probe. The auth-gate / OAuth-redirect check is meaningful through the live same-origin path:curl -sI https://dashboard.fro.bot/operator/auth/github/startreturns302to GitHub OAuth. A full browser round-trip returns the operator identity JSON{"githubUserId":<operator-id>,"login":<login>}from the callback when the GitHub user is onGATEWAY_OPERATOR_ALLOWLIST. -
Callback URL preflight: Before triggering the OAuth flow, verify the expected callback URL string. The deploy dry-run output prints:
Expected OAuth callback URL: https://dashboard.fro.bot/operator/auth/github/callbackCross-check this string against the Authorization callback URL field in the GitHub OAuth App settings (GitHub Developer Portal → OAuth Apps → your app). The portal check is manual and must be completed before enablement.
-
Allowlist enforcement: Attempt to complete the OAuth flow with a GitHub account that is NOT in the allowlist. Confirm the response is a coarse auth-failure (non-5xx, no route or allowlist detail leaked).
-
Session survives restart: After a successful auth, trigger a gateway restart (
docker compose restart gatewayon the droplet). Confirm the gateway log showssession restore finishedand the same cookie still returns200from/operator/sessionwithout signing in again. -
Operator port is VPC-scoped, not public:
docker compose --project-directory /opt/gateway/deploy psshows a<vpc-ip>:9300->9300mapping (VPC-scoped, e.g.10.116.0.3:9300->9300). A0.0.0.0:9300,[::]:9300, or bare9300:9300mapping is a defect — the publish must be bound to the gateway VPC IP only. Access is further restricted to the dashboard droplet by a DOCKER-USER iptables rule and a DigitalOcean Cloud Firewall rule (see Operator private path).
For the full operator auth lifecycle runbook (OAuth App setup, CSRF secret generation, allowlist format, secret seeding, rotation, rollback), see docs/runbooks/gateway-operator-auth-lifecycle.md.
OPERATOR PUSH
Section titled “OPERATOR PUSH”Gateway operator push is enabled in production. The deploy accepts this VAPID quartet only as an all-or-none configuration:
| Environment key | Host file | Container _FILE path |
|---|---|---|
GATEWAY_OPERATOR_PUSH_VAPID_PUBLIC_KEY |
gateway-operator-push-vapid-public-key |
/run/secrets/gateway_operator_push_vapid_public_key |
GATEWAY_OPERATOR_PUSH_VAPID_PRIVATE_KEY |
gateway-operator-push-vapid-private-key |
/run/secrets/gateway_operator_push_vapid_private_key |
GATEWAY_OPERATOR_PUSH_VAPID_SUBJECT |
gateway-operator-push-vapid-subject |
/run/secrets/gateway_operator_push_vapid_subject |
GATEWAY_OPERATOR_PUSH_VAPID_KEY_VERSION |
gateway-operator-push-vapid-key-version |
/run/secrets/gateway_operator_push_vapid_key_version |
All four absent is disabled. Any partial set hard-fails before SSH, spawn, secret materialization, or
remote writes and names every missing key. A complete set must pass strict validation: the public key
is unpadded base64url decoding to exactly 65 bytes with a 0x04 first byte; the private key is
unpadded base64url decoding to exactly 32 bytes; the subject is nonblank and uses mailto: or
https:; and the key version is a positive integer string.
Operators never set GATEWAY_OPERATOR_PUSH_ENABLED. The deploy derives
GATEWAY_OPERATOR_PUSH_ENABLED=true only from a complete valid quartet and emits no false line. The
four files are materialized through the existing SSH-stdin path only; the private key never appears in
argv, logs, or dashboard-bound artifacts. The compose override uses the four read-only _FILE paths
above. Push is additive to the announce, operator-listener, and VPC gates; absent push configuration
does not change those gates.
Verify enablement with an unauthenticated GET https://dashboard.fro.bot/operator/push/vapid-key: 401
means the route is mounted, 404 means it is not. The routes mount only after the object-store CAS
self-test passes at startup, so a 401 proves the self-test passed at boot — mount state is never
re-evaluated, so it is a post-deploy check, not a liveness probe. Enablement, format gates, rotation
limits, and rollback: docs/runbooks/operator-push-enablement.md.
The four VAPID files and rendered compose override participate in the existing secrets checksum, so
content or wiring changes select the existing recreate path. When a previously enabled deploy is
followed by an absent quartet, the bounded cleanup deletes exactly the four known files under
/opt/gateway/deploy/secrets/ before current secret materialization; the new override omits the VAPID
mounts and derived flag, and the checksum path recreates the gateway disabled. A never-enabled absent
quartet remains the ordinary no-file path.
The authenticated GET /operator/push/vapid-key route returns only {publicKey, keyVersion} when
enabled. It returns 404 when push is disabled or the operator guard denies the request.
Disabled-state verification
Section titled “Disabled-state verification”Verify names and status only: the four gateway VAPID secret/variable names remain absent, the dashboard push variable remains absent or false, and no independently forwarded gateway enable setting exists. There is no GitHub-stored derived gateway flag to inspect. Never retrieve, print, copy, or compare secret contents.
OPERATOR LISTENER
Section titled “OPERATOR LISTENER”The gateway operator listener is opt-in. When enabled, it exposes GET /operator/health (and future privileged operator routes) on a gateway-net-only address. Caddy routes /operator/* traffic from the public HTTPS edge to the listener over gateway-net. The listener is also published on the gateway VPC IP (${GATEWAY_VPC_IP}:9300) so the dashboard droplet can reach it over the shared DigitalOcean VPC — see Operator private path for the full bridge topology.
Browser-visible operator origin: The ratified target for the browser-visible operator API is https://dashboard.fro.bot/operator/*. GATEWAY_OPERATOR_PUBLIC_ORIGIN must be set to https://dashboard.fro.bot when the operator listener is enabled for production use. The gateway-side Caddy /operator/* route is topology scaffolding — it proves the gateway listener and Caddy wiring work, but gateway.fro.bot/operator/* is not the production browser origin and must not be used as such. See docs/plans/2026-06-18-001-feat-dashboard-operator-same-origin-plan.md for the full decision record.
Current state: The operator listener is enabled in production. GATEWAY_OPERATOR_BIND_HOST=172.21.0.2, GATEWAY_OPERATOR_BIND_PORT=9300, and GATEWAY_OPERATOR_PUBLIC_ORIGIN=https://dashboard.fro.bot are set in the gateway GitHub Environment. Deploy run 27740787921 succeeded; operator listener liveness is confirmed via the droplet-local direct probe (curl -sf http://172.21.0.2:9300/operator/health → 200 {"ok":true,"contractVersion":"<semver>"}). The public https://gateway.fro.bot/operator/health route returns 400 by design (forwarded-header guard — X-Forwarded-Host: gateway.fro.bot does not match PUBLIC_ORIGIN host dashboard.fro.bot). Operator auth is live: all four auth/config secrets (GATEWAY_OPERATOR_GITHUB_CLIENT_ID, GATEWAY_OPERATOR_GITHUB_CLIENT_SECRET, GATEWAY_OPERATOR_CSRF_SECRET, GATEWAY_OPERATOR_ALLOWLIST) are seeded in the gateway GitHub Environment and materialized as _FILE mounts. The dashboard Caddy /operator/* same-origin route is live: https://dashboard.fro.bot/operator/* routes through dashboard Caddy → VPC 10.116.0.3:9300 → gateway operator listener. End-to-end auth is verified: https://dashboard.fro.bot/operator/auth/github/start returns 302 to GitHub OAuth; completing the flow returns the operator identity JSON for an allowlisted user. The push client ships in the deployed dashboard image: https://dashboard.fro.bot/sw.js serves a Workbox service worker carrying push, notificationclick, and pushsubscriptionchange handlers. The authenticated UI surface itself sits behind GitHub OAuth and has not been verified from outside auth. Background: docs/plans/2026-06-18-001-feat-dashboard-operator-same-origin-plan.md.
Enabling
Section titled “Enabling”Set all three vars in the gateway GitHub Environment and trigger a deploy:
| Variable | Description |
|---|---|
GATEWAY_OPERATOR_BIND_HOST |
Gateway-net IPv4 address for the listener bind (e.g. 172.21.0.2). Must be a static gateway-net address — not 0.0.0.0, loopback (127.*), sandbox-net (10.*), 172.20.* (the known conflicted Docker subnet from the failed operator deploy), or IPv6. |
GATEWAY_OPERATOR_BIND_PORT |
Port for the operator listener (e.g. 9300). Must be a positive integer in [1, 65535]. |
GATEWAY_OPERATOR_PUBLIC_ORIGIN |
HTTPS origin for the browser-visible operator API. The ratified target is https://dashboard.fro.bot. Must be HTTPS. |
All three must be set together (all-or-none). Setting one or two fails the deploy before any SSH. Leaving all three unset disables the operator listener.
The deploy:
- Validates all three values before any SSH (rejects unsafe bind hosts, invalid ports, non-HTTPS origins).
- Requires
GATEWAY_VPC_IPandDASHBOARD_VPC_IP(valid exact IPv4) before any SSH;DASHBOARD_VPC_IPis the derived trusted proxy. - Writes
GATEWAY_OPERATOR_BIND_HOST,GATEWAY_OPERATOR_BIND_PORT,GATEWAY_OPERATOR_PUBLIC_ORIGIN, andGATEWAY_OPERATOR_TRUSTED_PROXIES(=DASHBOARD_VPC_IP) into the gateway service environment incompose.override.yaml. - Sets a static
ipv4_addressongateway-netfor the gateway service so the bind address is deterministic. - Adds a
/operator/*Caddy route withflush_interval -1(no response buffering, ready for future SSE). - Auto-cleans stale
fro-bot_gateway-net— afterdocker compose pulland beforedocker compose up, inspects the remote Docker network. If it exists with the old172.20.0.0/16subnet, removes it so Docker can recreate it with172.21.0.0/16. Missing or already-correct networks are skipped. Pull happens first so images are cached before any container disruption — if the pull fails, gateway/caddy containers are never removed. If the first removal fails due to active endpoints (gateway/caddy still attached), the deploy removes those two containers directly viadocker rm -f fro-bot-gateway-1anddocker rm -f fro-bot-caddy-1(tolerating missing containers so recovery works when caddy is absent from the current compose config) and retries the removal (stopalone is insufficient — stopped containers retain network endpoints). Any other removal failure, or failure of the container removal/retry, fails the deploy closed beforedocker compose up.GATEWAY_OPERATOR_BIND_HOSTmust be in the172.21.x.xrange (e.g.172.21.0.2). - Probes
GET ${GATEWAY_OPERATOR_PUBLIC_ORIGIN}/operator/health(i.e.https://dashboard.fro.bot/operator/health) after compose up, through the dashboard Caddy → VPC path whose peer is the trusted proxy. Success is HTTP200with a non-empty stringcontractVersionin the JSON body; the reported version is logged and not compared to any expected value. The probe is warning-only: a failed or malformed response is retried, then warned about, and never fails the deploy. The gateway-originhttps://gateway.fro.bot/operator/healthreturns 400 by design and is not probed. The droplet-localcurl -sf http://172.21.0.2:9300/operator/healthremains the direct listener check.
Security posture
Section titled “Security posture”- The operator listener is published only on the gateway VPC IP (
${GATEWAY_VPC_IP}:9300, e.g.10.116.0.3:9300), never on0.0.0.0or the publiceth0. A0.0.0.0:9300or bare9300:9300publish is a defect. - Access to the published VPC port is restricted to the dashboard droplet by two independent controls: a DOCKER-USER iptables rule (reapplied every deploy; load-bearing because Docker DNAT bypasses ufw) and a DigitalOcean Cloud Firewall rule (created by provisioning via
setupOperatorFirewall(); reboot-durable provider-level control). Both must be in place before the dashboard route goes live. - The workspace (
sandbox-net) has no path to the operator listener. - The only trusted proxy is
DASHBOARD_VPC_IP(GATEWAY_OPERATOR_TRUSTED_PROXIES); the gateway-net Caddy address is never trusted. /v1/announceand/operator/*are separate Caddyhandleblocks with distinct trust boundaries.- Auth/session/CSRF/allowlist wiring for privileged operator routes is live; the four operator auth/config secrets must be seeded in the
gatewayGitHub Environment to activate the auth gate (see Enabling anddocs/runbooks/gateway-operator-auth-lifecycle.md). - The gateway Caddy
/operator/*route is topology scaffolding. The production browser operator path ishttps://dashboard.fro.bot/operator/*via the dashboard Caddy proxy — notgateway.fro.bot/operator/*directly.
Post-enable verification
Section titled “Post-enable verification”After the first enabling deploy:
bunx @marcusrbrown/infra gateway status— confirmcaddyandgatewayservices appear healthy.curl -sf http://172.21.0.2:9300/operator/health— confirm a200 {"ok":true,"contractVersion":"<semver>"}response (droplet-local direct probe). Note:https://gateway.fro.bot/operator/healthreturns 400 by design (forwarded-header guard); use the direct probe for liveness confirmation.- Confirm the operator port publish is VPC-scoped:
docker compose --project-directory /opt/gateway/deploy ps— thegatewayservice must show a<GATEWAY_VPC_IP>:9300->9300mapping (e.g.10.116.0.3:9300->9300). A0.0.0.0:9300,[::]:9300, or bare9300:9300mapping is a defect. - Confirm workspace is not on
gateway-net:docker network inspect gateway-net— theworkspacecontainer must not appear. - Confirm the DOCKER-USER source restriction is in place:
iptables -nvL DOCKER-USER --line-numbers— a RETURN rule scoped to the VPC ingress interface, source<DASHBOARD_VPC_IP>, destination<GATEWAY_OPERATOR_BIND_HOST>, and--dport 9300must appear before a DROP rule for the same destination and port. Seedocs/runbooks/gateway-operator-private-path-verification.mdfor the full verification procedure.
OPERATOR PRIVATE PATH (DASHBOARD SAME-ORIGIN)
Section titled “OPERATOR PRIVATE PATH (DASHBOARD SAME-ORIGIN)”The dashboard droplet reaches the gateway operator listener over the shared DigitalOcean VPC (nyc1, uuid d95c26cc-…). The gateway VPC IP is 10.116.0.3 and the dashboard VPC IP is 10.116.0.5 (example values; the actual IPs are set via GATEWAY_VPC_IP and DASHBOARD_VPC_IP). The daemon stays bound to gateway-net 172.21.0.2:9300; the bridge is a host-side Docker port publish, not a daemon rebind.
Bridge topology
Section titled “Bridge topology”-
VPC-IP-scoped Docker port publish — the gateway service publishes
${GATEWAY_VPC_IP}:9300:9300in the compose override. The daemon stays bound to172.21.0.2:9300ongateway-net; Docker DNAT forwards VPC-IP traffic to it. The publish is emitted whenever the operator listener is enabled;GATEWAY_VPC_IPandDASHBOARD_VPC_IPare required then. -
DOCKER-USER iptables rule — applied over SSH after
docker compose up(Docker recreates DOCKER-USER-adjacent chains on daemon restart, so the rule must be reapplied last). The rule allows--dport 9300from${DASHBOARD_VPC_IP}and drops all other sources. Applied idempotently (-C || -I); exact-readback-verified (source/dport/jump/ordering). This rule is load-bearing — the daemon’s forwarded-header guard is forgeable by anyone who can reach:9300; Docker DNAT bypasses ufw. The rule is lost on reboot until the next deploy; the DO Cloud Firewall covers the reboot window. -
DigitalOcean Cloud Firewall — created and maintained by provisioning (
provision-droplet.ts/bun run provision:gateway) viasetupOperatorFirewall(), not by the deploy. Provisioning creates a firewall namedgateway-operator-fwwith base inbound rules (SSH 22, HTTP 80, HTTPS 443) plus inbound tcp/9300 from the dashboard droplet-id, and attaches it to the gateway droplet. The operation is idempotent: create-if-absent, add-9300-rule-if-missing, no-op if already present. RequiresGATEWAY_VPC_IPandDASHBOARD_VPC_IPto be set at provisioning time. Source is the dashboard droplet-id (stable across rebuild), not the private IP. Reboot-durable provider-level control.DIGITALOCEAN_ACCESS_TOKENis a provisioning-time concern (repo-level, used locally) — it is not required in thegatewayGitHub Environment for deploy. -
Dashboard Caddy
/operator/*route — the dashboard Caddy proxies/operator/*to{$GATEWAY_VPC_IP}:9300over the VPC. The route is ahandleblock before thedashboard:3000catch-all, withflush_interval -1andheader_up Host dashboard.fro.bot+header_up X-Forwarded-Proto httpsto satisfy the daemon’s forwarded-header guard.
Required env vars (whenever the operator listener is enabled)
Section titled “Required env vars (whenever the operator listener is enabled)”| Variable | Where set | Description |
|---|---|---|
GATEWAY_VPC_IP |
gateway GitHub Environment + local .env |
Gateway droplet’s DigitalOcean VPC IP (e.g. 10.116.0.3). Required for the VPC-IP publish and DOCKER-USER rule. |
DASHBOARD_VPC_IP |
gateway GitHub Environment + local .env |
Dashboard droplet’s DigitalOcean VPC IP (e.g. 10.116.0.5). Required for the DOCKER-USER source restriction and used as the derived GATEWAY_OPERATOR_TRUSTED_PROXIES value. |
DIGITALOCEAN_ACCESS_TOKEN |
local .env only (provisioning-time) |
Used by provision-droplet.ts to create the gateway-operator-fw Cloud Firewall. Not a deploy secret — not required in the gateway GitHub Environment. |
GATEWAY_VPC_IP is also required in the dashboard GitHub Environment so the dashboard Caddy service can expand {$GATEWAY_VPC_IP} in the Caddyfile.
First-deploy prerequisite: the dashboard droplet must already exist before running bun run provision:gateway with the firewall setup, because provisioning resolves the dashboard droplet-id to set the 9300 inbound source. Run bun run provision:dashboard first, then bun run provision:gateway with GATEWAY_VPC_IP and DASHBOARD_VPC_IP set.
First-deploy order
Section titled “First-deploy order”Deploy the gateway bridge first (VPC publish + DOCKER-USER + DO firewall), verify the bridge is live and restricted, then deploy the dashboard /operator/* route. The dashboard route must not go live before its target is reachable and restricted — a partial state where the route is live but the bridge is absent or unrestricted is a security defect.
Verification
Section titled “Verification”See docs/runbooks/gateway-operator-private-path-verification.md for the full verification procedure (positive probes, control readback, negative-control caveat, and break-glass).
CA RESTORE PROCEDURE
Section titled “CA RESTORE PROCEDURE”The mitmproxy CA cert and private key live in the Docker named volume fro-bot_mitmproxy-certs (the compose key mitmproxy-certs prefixed with the Compose project name fro-bot). Backup and restore both check that this volume exists before doing anything else and abort naming it if it is missing, so neither can create a stray empty mitmproxy-certs volume. If the droplet is destroyed or the volume is lost, all currently-running workspaces lose trust in the gateway’s egress proxy.
Backup (operator-driven, periodic):
bunx @marcusrbrown/infra gateway backup --include-ca --output ./gateway-ca.tarMove the file to a safe store (1Password, encrypted backup, etc.). The tar contains both mitmproxy-ca-cert.pem (public) and mitmproxy-ca.pem (private key). Treat the file as a secret.
Restore — droplet intact (volume lost or cert corrupted):
bunx @marcusrbrown/infra gateway restore --include-ca --input ./gateway-ca.tarThe CLI validates the archive locally first — it must contain exactly the two expected files, otherwise it is rejected before any remote mutation. It then confirms the fro-bot_mitmproxy-certs volume exists on the host (restore never creates it), uploads to an unguessable remote path, extracts into the named volume, restarts mitmproxy + gateway, and byte-equal compares the restored cert + key against the input archive. Mismatch aborts with a clear error.
Restore — full disaster recovery (droplet destroyed):
- Set
GATEWAY_HOSTlocally and runbun run provision:gatewayto create a replacement droplet. - Commit and push the updated
.github/known_hosts. - If the host or IP changed, update
GATEWAY_HOST(andGATEWAY_SSH_KEYif re-keyed) in thegatewayGitHub Environment. - Trigger a deploy (
bunx @marcusrbrown/infra gateway deploy) and approve the environment gate. This creates/opt/gateway/deploy/on the new droplet. bunx @marcusrbrown/infra gateway restore --include-ca --input ./gateway-ca.tar— restores the CA into the named volume.- Verify:
bunx @marcusrbrown/infra gateway statusandbunx @marcusrbrown/infra gateway logs gateway --tail 100.
SECRET ROTATION
Section titled “SECRET ROTATION”- Update the value in the
gatewayGitHub Environment. - Trigger
gateway deploy(or wait for the next push-to-main run). - The deploy script writes the new value to the droplet, recomputes the secrets checksum, force-recreates affected containers via
docker compose up -d --force-recreate <service>, polls Discord for command registration, and writes the new checksum on success.
The checksum-after-success invariant means: if compose or polling fail mid-rotation, the old checksum persists and the next deploy will force-recreate again — no silent stale-credentials state.
For the full operator-facing rotation and emergency revocation procedure (including Keychain sync, shell-history hygiene, and audit surfaces), see docs/runbooks/discord-token-lifecycle.md.
ANTI-PATTERNS
Section titled “ANTI-PATTERNS”- Never
ssh-keyscanin CI — host keys are pinned in.github/known_hostsat provision time and committed. Provisioning scripts may usessh-keyscanlocally. - Never pass secret bytes via argv —
writeRemoteFilepipes bytes through SSH stdin only.--body <value>patterns are banned. - Never skip
validateGatewayHost— it rejects--prefixed values and characters outside the allowed alphabet. SSH treats--prefixed hostnames as flags (including-oProxyCommand=). - Never restart the gateway in-place to rotate the CA — workspaces lose trust in the egress proxy. Restore from backup instead.
- Never run
docker compose up --buildon the droplet — building thegatewayorworkspaceimages on the 1vCPU/2GB droplet exhausts RAM and thrashes swap while the live stack is running. The deploy usesdocker compose pull+up -d --no-build; thecompose.override.yamlimage pins make GHCR the source of truth. For emergency rebuilds, build from a workstation and push to GHCR (see Break-glass runbook). - Never validate
WORKSPACE_OPENCODE_CONFIGwith the shell-metachar guard — it is JSON (",$,\are required) andSHELL_METACHAR_REwould reject every valid config. Validate it withJSON.parse+ newline/size checks;SHELL_METACHAR_REis only for simple values likeWORKSPACE_OPENCODE_MODEL. - Never bind-mount config files outside
/opt/gateway/deploy/secrets/—init-certs.shanddocker-compose.yamlare upstream’s; this repo materializes secrets only. - Never run
pollRegistrationwith an unbounded per-attempt timeout — each fetch is wrapped in anAbortControllerwithperAttemptTimeoutMs(defaults tomax(6000, intervalMs * 2)). - Never run
docker compose down -vordocker volume pruneon the gateway stack — this wipes named Docker volumes includingcaddy_data(TLS certs),mitmproxy-certs(CA key; live namefro-bot_mitmproxy-certs), andworkspace-repos(all cloned repo checkouts). Normal deploys andgit clean -xfddo NOT touch Docker volumes. Losingcaddy_dataforces Let’s Encrypt re-issuance (rate-limit risk); losingmitmproxy-certsbreaks workspace egress trust (requires CA restore); losingworkspace-reposdestroys all cloned repos (they will be re-cloned automatically on the next mention, but the volume loss itself is irreversible). - Never publish the operator port on
0.0.0.0or without a VPC-IP bind — the operator listener must be published only on${GATEWAY_VPC_IP}:9300, never on0.0.0.0:9300,[::]:9300, or as a bare9300:9300mapping. A public-internet-reachable operator port is a security defect; the DOCKER-USER rule and DO Cloud Firewall are the access controls, but they are only effective when the publish is VPC-scoped. - Never apply the DOCKER-USER rule before
docker compose up— Docker recreates DOCKER-USER-adjacent chains on daemon restart andcompose up, wiping any rule applied before the stack comes up. The rule must be reapplied aftercompose upcompletes. - Never create a fresh DO Cloud Firewall for the operator port outside of provisioning — DO Cloud Firewalls are default-deny/allowlist; attaching a new firewall that only allows
:9300would lock out:22/:80/:443/announce. Provisioning (setupOperatorFirewall()) createsgateway-operator-fwwith all required base rules (22, 80, 443) plus the 9300 source restriction, idempotently. The deploy does not touch the DO firewall. - Never raise the workspace memory limit without re-measuring host available memory — measure with the workspace at its limit. At 848 MiB used the host had 490 MiB available, so the 1024 MiB limit leaves about 314 MiB; keep at least 256 MiB.
- Never drop
memswap_limitfrom the workspace service — without it Docker allows swap up to twicemem_limit. - Never grant the
fro-bot-gatewayruntime identity IAM self-management — its credentials write run-state objects; they must not carryiam:*permissions to manage their own policy. If a permission is missing (e.g.s3:PutObjectTagging, see S3 State-Bucket IAM Actions), fix the grant in AWS out-of-band — never widen the identity to route around a mismatched permission.
DECOMMISSIONING
Section titled “DECOMMISSIONING”bunx @marcusrbrown/infra gateway backup --include-ca— preserve the CA if you ever want to restore the gateway.doctl compute droplet delete <droplet-id>- Remove the gateway entries from
.github/known_hosts. - Delete the
gatewayGitHub Environment.