Skip to content

Gateway Operator Private Path Verification

The dashboard→gateway operator private path routes https://dashboard.fro.bot/operator/* to the gateway operator listener over the shared DigitalOcean VPC. The bridge consists of four controls: a VPC-IP-scoped Docker port publish, a DOCKER-USER iptables source restriction, a DigitalOcean Cloud Firewall rule, and the dashboard Caddy /operator/* route. This runbook covers verifying that all four controls are in place and the path is healthy.


  • The gateway operator listener is enabled: GATEWAY_OPERATOR_BIND_HOST, GATEWAY_OPERATOR_BIND_PORT, and GATEWAY_OPERATOR_PUBLIC_ORIGIN are set in the gateway GitHub Environment.
  • The VPC bridge is enabled: GATEWAY_VPC_IP and DASHBOARD_VPC_IP are set in the gateway GitHub Environment (both are required whenever the listener is enabled; DASHBOARD_VPC_IP is also the derived GATEWAY_OPERATOR_TRUSTED_PROXIES value); GATEWAY_VPC_IP is set in the dashboard GitHub Environment.
  • The gateway-operator-fw Cloud Firewall was created by provisioning (bun run provision:gateway with GATEWAY_VPC_IP and DASHBOARD_VPC_IP set). DIGITALOCEAN_ACCESS_TOKEN is a provisioning-time concern (local .env only) — it is not required in the gateway GitHub Environment for deploy.
  • SSH access to the gateway droplet via GATEWAY_SSH_KEY from the repo-root .env.
  • doctl authenticated locally (doctl auth init).

Terminal window
curl -sf https://dashboard.fro.bot/operator/health
# expect: 200 {"ok":true,"contractVersion":"<semver>"}

A 200 response confirms the full path is live: dashboard Caddy → VPC → gateway daemon. This is the primary health signal for the private path.

SSH to the gateway droplet and probe the daemon directly on the gateway-net address:

Terminal window
ssh root@gateway.fro.bot \
'curl -sf http://172.21.0.2:9300/operator/health'
# expect: 200 {"ok":true,"contractVersion":"<semver>"}

A 200 here confirms the operator listener is up on gateway-net, independent of the VPC bridge. If this fails but the same-origin probe succeeds, something is wrong with the daemon — not the bridge.


Public internet must not reach the operator port

Section titled “Public internet must not reach the operator port”
Terminal window
curl -sf --connect-timeout 5 http://gateway.fro.bot:9300/operator/health
# expect: connection refused or timeout (not 200)

The gateway’s public eth0 must not accept connections on :9300. A successful connection here means the VPC-IP publish has leaked to the public interface — a security defect requiring immediate remediation (see Break-glass below).

Caveat: A true “non-dashboard VPC source denied” packet test requires a VPC peer originating from a non-dashboard VPC IP. The deploy verifies this structurally via control readback (see below), not a live foreign-source packet — the runner is not a VPC peer.


SSH to the gateway droplet and read the DOCKER-USER chain:

Terminal window
ssh root@gateway.fro.bot \
'iptables -nvL DOCKER-USER --line-numbers'

Expected output (example IPs and VPC ingress interface eth1; substitute actual DASHBOARD_VPC_IP, GATEWAY_OPERATOR_BIND_HOST, and the interface used to reach the dashboard VPC IP, as reported by ip route get <DASHBOARD_VPC_IP>):

Chain DOCKER-USER (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 RETURN tcp -- eth1 * 10.116.0.5 172.21.0.2 tcp dpt:9300
2 0 0 DROP tcp -- * * 0.0.0.0/0 172.21.0.2 tcp dpt:9300
3 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

Verify:

  • Rule 1: RETURN, ingress interface = VPC interface (eth1), source = dashboard VPC IP (10.116.0.5), destination = gateway-net listener IP (172.21.0.2), dport = 9300.
  • Rule 2: DROP, source = any, destination = gateway-net IP, dport = 9300.
  • Rule 1 appears before Rule 2 (ordering is load-bearing).

RETURN allows the matching traffic to continue through Docker’s remaining forwarding rules; it is the target installed and verified by the deploy, rather than an unconditional ACCEPT. A wrong-but-present rule (wrong ingress interface, source IP, destination IP, dport, target, or ordering) is a defect — presence alone is not sufficient.

Note: The DOCKER-USER rule is reapplied every deploy and is lost on reboot until the next deploy. The DigitalOcean Cloud Firewall (below) is the reboot-durable provider-level control that covers the reboot window.

The gateway-operator-fw firewall is created by provisioning (bun run provision:gateway), not by the deploy. Verify it is attached and correctly configured:

Terminal window
# List firewalls attached to the gateway droplet
doctl compute firewall list --format ID,Name,DropletIDs
# Inspect the firewall (substitute <firewall-id>)
doctl compute firewall get <firewall-id>

In the InboundRules output, verify a rule exists with:

  • Protocol: tcp
  • Ports: 9300
  • Sources: droplet:<dashboard-droplet-id> (the dashboard droplet-id, not a private IP)

All base inbound rules (:22, :80, :443) must also be present — provisioning creates them alongside the 9300 rule. A missing :22 rule indicates the firewall was replaced or corrupted — a critical defect. If the 9300 rule is missing, re-run bun run provision:gateway (idempotent: adds the missing rule without touching existing rules).

SSH to the gateway droplet and confirm the publish is VPC-scoped:

Terminal window
ssh root@gateway.fro.bot \
'docker compose --project-directory /opt/gateway/deploy ps'

The gateway service must show a <GATEWAY_VPC_IP>:9300->9300 mapping (e.g. 10.116.0.3:9300->9300). A 0.0.0.0:9300, [::]:9300, or bare 9300:9300 mapping is a defect.


Symptom Likely cause Remediation
curl https://dashboard.fro.bot/operator/health → non-200 Dashboard Caddy route not live, or gateway bridge not reachable Check dashboard deploy logs; verify GATEWAY_VPC_IP is set in the dashboard env; verify the gateway bridge is live
curl http://172.21.0.2:9300/operator/health → non-200 Operator listener not running Check gateway daemon logs: bunx @marcusrbrown/infra gateway logs gateway --tail 100
curl http://gateway.fro.bot:9300 → 200 VPC publish leaked to public interface Immediate: redeploy the gateway to restore the VPC-scoped publish; verify the DOCKER-USER rule and DO firewall are in place
DOCKER-USER rule missing or wrong Rule was wiped (reboot or compose up without reapply) Redeploy the gateway — the deploy reapplies the rule after compose up
DO firewall :9300 rule missing Firewall rule was manually removed, or provisioning was not run with VPC vars set Re-run bun run provision:gateway with GATEWAY_VPC_IP and DASHBOARD_VPC_IP set (idempotent; adds the missing rule)
DO firewall :22 rule missing Firewall was replaced or corrupted Critical — restore the existing firewall rules immediately via doctl compute firewall update; SSH access may be lost

If a bad iptables rule or firewall configuration locks out the dashboard from the operator path:

  1. Do not SSH-hammer the gateway droplet — ufw rate-limits new connections (6 new connections per 30 seconds). Use external probes (curl, doctl) to diagnose.
  2. Power-cycle the gateway droplet to clear a bad iptables state (the DOCKER-USER rule is not persistent across reboots):
    Terminal window
    doctl compute droplet-action power-cycle <gateway-droplet-id>
  3. Redeploy the gateway — the deploy reapplies the DOCKER-USER rule after compose up:
    Terminal window
    bunx @marcusrbrown/infra gateway deploy
    The DO Cloud Firewall is a provisioning artifact and is not touched by the deploy. If the firewall rule is also missing, re-run bun run provision:gateway separately.
  4. Verify using the probes above before declaring the path healthy.

If the DO Cloud Firewall is misconfigured and SSH access to the gateway is lost, use the DigitalOcean console to restore the firewall rules before attempting SSH.


Login entry and end-to-end auth verification

Section titled “Login entry and end-to-end auth verification”

The operator login entry is https://dashboard.fro.bot/operator/auth/github/start. A 302 redirect to GitHub OAuth confirms the full same-origin path is live and the auth gate is wired.

Terminal window
curl -sI https://dashboard.fro.bot/operator/auth/github/start
# expect: HTTP/2 302
# location: https://github.com/login/oauth/authorize?...client_id=<GATEWAY_OPERATOR_GITHUB_CLIENT_ID>...

The location: header points to https://github.com/login/oauth/authorize with query parameters including the operator OAuth app’s client ID (GATEWAY_OPERATOR_GITHUB_CLIENT_ID), PKCE challenge, CSRF state, and scope=read:user. Do not hardcode the client ID value in probes — verify it matches the value registered in the GitHub Developer Portal (OAuth Apps → your operator app).

A full browser round-trip — completing GitHub OAuth at the start URL, callback at https://dashboard.fro.bot/operator/auth/github/callback — returns the operator identity JSON:

{"githubUserId":<operator-id>,"login":"<login>"}

This confirms the allowlist check passed (GATEWAY_OPERATOR_ALLOWLIST contains the GitHub user ID) and the full auth path is live end-to-end.

Route surface: The verification probes use three routes:

  • GET /operator/health — health probe; returns 200 {"ok":true,"contractVersion":"<semver>"}
  • GET /operator/auth/github/start — login entry; returns 302 to GitHub OAuth
  • GET /operator/auth/github/callback — OAuth callback; returns operator identity JSON on success

The live surface is larger than the three probes above — /operator/session, /operator/session/csrf, /operator/repos, the /operator/runs family, POST /operator/dispatch, and the conditionally registered /operator/push/* routes all exist. Root AGENTS.md carries the current list; treat it as the source of truth rather than duplicating it here.

Bare /operator/ returns {"error":"not-found"} (HTTP 404) by design. Bare /operator (no trailing slash) never reaches the gateway at all — it does not match Caddy’s /operator/* prefix and goes to the dashboard app, which returns 302 to login without a session and 404 with one. Use /operator/health as the health surface and /operator/auth/github/start as the login surface.