Credential Broker Deploy Package
OIDC-authenticated credential broker at broker.fro.bot. Docker Compose stack (Caddy + Bun service) on a DigitalOcean Droplet. Exchanges a GitHub Actions OIDC token for a short-lived cliproxy API key so the durable provider key never lands on the CI runner.
WHERE TO LOOK
Section titled “WHERE TO LOOK”| Task | Location | Notes |
|---|---|---|
| Docker stack | docker-compose.yaml |
Caddy + broker (Bun), digest-pinned, Caddy-side healthcheck |
| Caddy config | config/Caddyfile |
Exposes /v1/mint and /healthz only; all other paths → 404 |
| Provision droplet | server/provision-droplet.ts |
One-time. Refuses re-run on existing droplet without --force |
| Deploy script | src/deploy.ts |
Preflight → upload → secrets via stdin → compose up → healthcheck |
| OIDC verification | src/oidc.ts |
jose JWKS verify (RS256 only), replay denylist keyed (jti, iss) |
| Trust policy | src/policy.ts |
BROKER_TRUST_POLICY code-owned allowlist constant; evaluateClaims |
| Mint/revoke client | src/mint.ts |
GET-modify-write + read-back, single-flight lock, ghact- prefix |
| Sweeper | src/sweeper.ts |
TTL backstop (60s tick) + reconcile (5 min tick); startup reconcile |
| Audit events | src/audit.ts |
Structured JSON lines: mint, deny, deny-ratelimit, revoke, error |
| Host validator | src/host.ts |
validateBrokerHost — rejects --prefixed / out-of-alphabet values |
| CLI commands | packages/cli/src/commands/broker/ |
status (MCP-safe), deploy, logs (CLI-only) |
DEPLOY FLOW
Section titled “DEPLOY FLOW”- Preflight (
preflightChecks): verifiesCLIPROXY_MANAGEMENT_KEYis set, then GETs cliproxy/v0/management/api-keyswith the cliproxy management key. Aborts on 401/403 (key drift) or network failure; a 403 whose body says “IP banned” aborts with a distinct source-IP-ban message (about 30 min, key may be fine) instead of the invalid-key one. No compose change happens until this passes. - Build (
buildBundle): runsbun build src/main.ts --target bun --outfile dist/main.js(cwdapps/broker/). Produces a self-contained ~300KB bundle withjoseinlined via Web Crypto. Aborts on non-zero exit — no remote mutation happens if the build fails. - Upload:
docker-compose.yaml,config/Caddyfile, anddist/main.jsare SCP’d to/opt/broker/on the droplet (dist/main.js→/opt/broker/dist/main.js). - Secrets: the broker
.envfile (BROKER_HOST,CLIPROXY_MANAGEMENT_URL,CLIPROXY_MANAGEMENT_KEY,BROKER_AUD) is written via SSH stdin — never in argv. - Restart: a best-effort
docker image prune -afruns first (failure only warns), thendocker compose pull && docker compose up -d --force-recreate --wait --wait-timeout 90from/opt/broker/.--force-recreateis required becausedist/main.jsis bind-mounted (ro) into the broker container and loaded into memory at process start — a bundle-only change leaves the compose spec, image, and env byte-identical, so without--force-recreatecompose would see no config change and skip recreating the container, leaving the old process running the stale bundle. Named volumes (caddy_data,caddy_config) persist across recreation, so Let’s Encrypt certs are unaffected. - Health gate: GET
https://<BROKER_HOST>/healthzconfirms the stack is serving.
All SSH calls in a single deploy share one ControlPath socket (avoids UFW rate-limit at 6 connections/30s).
Critical: the broker .env on the droplet holds the cliproxy management key. The deploy always overwrites it via stdin. Never embed secret bytes in SSH argv.
dist/main.js is a deploy-time build artifact — it is gitignored and never committed.
STARTUP GATE
Section titled “STARTUP GATE”On boot, the broker attempts a reconcile sweep before accepting /v1/mint requests: it lists cliproxy api-keys and, for each ghact--prefixed key, parses its embedded expiry from the key name and revokes it only if that expiry has passed. Unexpired ghact- keys are preserved regardless of in-memory live-set membership — this is what lets an in-flight run’s key survive a restart.
List failures are retried up to three attempts, with 500ms and 1s backoffs; after exhaustion, the broker logs the failure and marks itself ready anyway to preserve availability. Individual revoke failures are logged without aborting the sweep. The production entrypoint (src/main.ts) waits for this attempt sequence before starting the periodic sweeper and binding the HTTP listener. The handler itself leaves /healthz ungated and returns 503 for /v1/mint when not ready; production does not bind the listener during startup reconciliation.
Readiness therefore does not prove expired-key cleanup succeeded: failed cleanup relies on subsequent sweeper/reconcile ticks, including the five-minute reconcile for keys absent from the post-restart live set.
DOCKER STACK
Section titled “DOCKER STACK”- Caddy: HTTPS termination, auto Let’s Encrypt.
restart: unless-stopped. Healthcheck probeshttp://broker:3000/healthzacross the compose network. - broker:
oven/bun(pinned digest, Renovate-managed). Runsbun main.jsagainst the pre-built bundle mounted at/app/main.js:ro. No source bind-mount; no node_modules on the droplet. Bun HTTP service on port 3000 (internal only).restart: unless-stopped. - Volumes:
caddy_data,caddy_config,./dist/main.js:/app/main.js:ro(bundle, read-only). - Env file:
BROKER_HOST,BROKER_AUD,CLIPROXY_MANAGEMENT_URL,CLIPROXY_MANAGEMENT_KEYinjected from host.env.
ONE-TIME PROVISIONING
Section titled “ONE-TIME PROVISIONING”Prerequisites:
brokerGitHub Environment created with required reviewer + main-only branch policy (pre-create before merge — auto-create is ungated)fro-bot-brokerEd25519 keypair generated; public key uploaded to DigitalOcean;BROKER_SSH_KEY(private key) in.envCLIPROXY_MANAGEMENT_KEY(the broker→cliproxy management key) in.envBROKER_AUD(the broker-minted OIDC audience value) in.env
Run:
bun run provision:broker(Root wrapper — loads the repo-root .env; --cwd apps/broker would miss it.)
The script will:
- Validate
doctlauth - Validate
BROKER_DOMAIN(defaults tobroker.fro.bot) viavalidateBrokerHostbefore any SSH argv - Reject if the
brokerdroplet already exists (aborts;--forceto override) - Create the droplet (
docker-20-04,s-1vcpu-1gb,nyc1), wait for running state - Wait for SSH (
waitForSsh) - Pin the broker FQDN + IP host keys into
.github/known_hosts(marker-idempotent) - Upload
docker-compose.yamlandconfig/Caddyfile - Write the
.envfile via SSH stdin (never argv) - Verify Docker is installed
- Start the compose stack
After provisioning: commit the updated .github/known_hosts before the first CI deploy.
Bootstrap ordering
Section titled “Bootstrap ordering”- Seed
BROKER_SSH_KEY,CLIPROXY_MANAGEMENT_KEY,BROKER_AUDinto.env bun run provision:broker— provisions the droplet and pins host keys- Seed
BROKER_HOSTinto.env+ thebrokerGitHub Environment, matching theBROKER_DOMAINused during provisioning (defaults tobroker.fro.bot). For a custom FQDN, setBROKER_DOMAINin.envbefore step 2; provisioning does not readBROKER_HOST. - Commit the updated
.github/known_hosts - First deploy:
bunx @marcusrbrown/infra broker deploy
REQUIRED SECRETS AND VARIABLES
Section titled “REQUIRED SECRETS AND VARIABLES”| Name | Kind | Required | Description |
|---|---|---|---|
BROKER_SSH_KEY |
secret | ✓ | Ed25519 private key for the broker droplet (fro-bot-broker keypair) |
BROKER_HOST |
secret | ✓ | FQDN of the broker droplet (e.g. broker.fro.bot) |
CLIPROXY_MANAGEMENT_KEY |
secret | ✓ | cliproxy management key — broker uses this to mint/revoke api-keys via the cliproxy management API |
BROKER_AUD |
variable | ✓ | OIDC audience value for the broker. Not a secret — it is a cross-context replay defense. Set as a broker GitHub Environment variable (not a secret). Also required in the local .env for provisioning. |
Secrets and the BROKER_AUD variable are scoped to the broker GitHub Environment. BROKER_AUD flows at both provision time (written to the droplet .env by provision-droplet.ts) and deploy time (passed as vars.BROKER_AUD from the deploy-broker.yaml workflow).
CLI COMMANDS
Section titled “CLI COMMANDS”| Command | Purpose |
|---|---|
bunx @marcusrbrown/infra broker status |
GET /healthz on the broker; reports HTTP reachability |
bunx @marcusrbrown/infra broker deploy |
Trigger the Deploy Broker workflow via gh workflow run (remote, default). --local runs apps/broker/src/deploy.ts directly. |
bunx @marcusrbrown/infra broker logs [--tail N] [--service broker] |
Stream broker service logs over SSH. Logs may contain run identities; operator-only. |
broker status is MCP-exposed (read-only). broker deploy and broker logs are CLI-only (mutating or sensitive).
DAY-2 OPERATIONS
Section titled “DAY-2 OPERATIONS”Mint/revoke lifecycle
Section titled “Mint/revoke lifecycle”Each POST /v1/mint request:
- Verifies the OIDC JWT (RS256, GitHub issuer, broker-minted audience,
exp/nbf/jtireplay check). - Evaluates claims against
BROKER_TRUST_POLICY(repository_id,repository_owner_id,job_workflow_ref,ref,ref_type,ref_protected,event_name,runner_environment,repository_visibility). - Mints a
ghact-<runId>-<expiresAtEpochMs>-<hexrand>key via cliproxy management API (GET-modify-write + read-back, single-flight lock). - Records the entry in the in-memory live set with a 30-minute TTL.
- Returns the OpenCode
auth.jsonpayload (same shape asapps/cliproxy/AGENTS.md):anthropic,openaiandopencode-goentries, each carrying the minted key. The consumer must configureprovider.opencode-go.options.baseURLto cliproxy, because OpenCode’s built-inopencode-goprovider otherwise sends requests to OpenCode’s own endpoint.
At run end (success/fail/cancel), revocation is handled exclusively by the TTL sweeper and reconcile — there is no run-end revoke endpoint. The sweeper is the mandatory backstop.
TTL sweeper and reconcile
Section titled “TTL sweeper and reconcile”- Sweep tick (every 60s): for each live entry past
expiresAt, revokes the key via cliproxy DELETE and removes it from the live set. A single revoke failure is logged and does not block the rest of the sweep. - Reconcile tick (every 5 min): lists cliproxy
api-keysand makes a time-based decision perghact--prefixed key by parsing its embedded expiry from the key name — expired keys are revoked, unexpired keys are kept regardless of live-set membership, and unparseable legacy keys are kept and logged. Recovers from a broker restart where the live set is empty but an in-flight key’s expiry proves it is still valid. Safety invariant: reconcile never deletes a non-ghact-key. - Startup reconcile: attempts cleanup on boot before
/v1/mintis unblocked; readiness proceeds after exhausted list retries even if cleanup failed (see Startup Gate above).
Reading audit events
Section titled “Reading audit events”Audit events are JSON lines written to stdout by the broker container. Each event carries:
{ "type": "broker-audit", "ts": "<ISO-8601>", "srcIp": "<ip>", "runId": "<run_id>", "jti": "<jti>", "repositoryId": "<repo_id>", "workflowRef": "<workflow_ref>", "decision": "mint|deny|deny-ratelimit|revoke|error", "reason": "<human-readable>"}Never contains token bytes, the minted key value, the OIDC bearer, or the management key. Stream with bunx @marcusrbrown/infra broker logs.
Decision values:
mint— key successfully minteddeny— OIDC verification or policy check faileddeny-ratelimit— per-repo or global rate limit exceededrevoke— key revoked by the TTL sweeper (reason: ttl-expired); there is no run-end revoke callbackerror— unexpected error during mint (cliproxy unreachable, etc.)
Management key rotation
Section titled “Management key rotation”If CLIPROXY_MANAGEMENT_KEY must be rotated:
- Update the key in the
brokerGitHub Environment secret. - Update the key in the local
.env. - Run
bunx @marcusrbrown/infra broker deploy— the deploy writes the new key to the droplet.envvia SSH stdin. - Verify with
bunx @marcusrbrown/infra broker status.
Do not edit the .env on the droplet directly — the next deploy overwrites it. Do not rotate the key in-place without updating the GitHub Environment secret first (the next CI deploy would revert to the old value).
Monitoring
Section titled “Monitoring”bunx @marcusrbrown/infra broker status— HTTP reachability via/healthz.bunx @marcusrbrown/infra broker logs— stream audit events and service output.- Unified
bunx @marcusrbrown/infra status— includes a broker row.
ANTI-PATTERNS
Section titled “ANTI-PATTERNS”- Never log token or key material — the OIDC bearer, the minted key, the management key, and raw claim payloads must never appear in logs, error responses, or audit events. The
Authorizationheader is always redacted before any logging. - Never full-array PUT against cliproxy
api-keys— always GET → append single key → PUT → GET-back and assert presence. A wholesale replace drops other consumers’ keys. The single-flight lock serializes all mint and revoke operations; never issue parallel management-API mutations. - Never retry on HTTP error from the management API — cliproxy IP-bans the caller after ~5 consecutive bad-key attempts (~30 min). Throw immediately on 401/403; bounded retry only on read-back mismatch.
- Never touch non-
ghact-keys — the sweeper and reconcile only delete keys starting withghact-. Durable keys and other consumers’ keys are never touched. - Never skip
validateBrokerHost— it rejects--prefixed values and characters outside the allowed alphabet. SSH treats--prefixed hostnames as flags (including-oProxyCommand=). - Never pass secret bytes via argv — the broker
.env(management keys,BROKER_AUD) is written via SSH stdin only.--body <value>patterns are banned. - Never use
ssh-keyscanin CI — host keys are pinned in.github/known_hostsat provision time and committed. Provisioning scripts may usessh-keyscanlocally. - Never scale the broker horizontally — the single-flight lock is valid only because there is exactly one broker instance (single droplet). Scaling out breaks the lock invariant and requires a distributed lock or CAS-capable management API.
- The mint response is a bare
auth.jsonmap. It carriesanthropic,openaiandopencode-goentries, each holding the same minted key, and no base URL. A consumer must setprovider.opencode-go.options.baseURLto cliproxy in itsOPENCODE_CONFIG; without it, OpenCode’s built-inopencode-goprovider sends the minted key to OpenCode’s own endpoint. - Pattern A capability caveat — a minted key is fungible with the durable cliproxy key for its TTL. cliproxy has no per-key capability surface, so the security property delivered is short-lived + revocable + off-runner, not capability-restricted. In-run abuse during the TTL window is a documented non-goal (requires Pattern B / egress containment, deferred).
- Trust policy:
BROKER_TRUST_POLICYinapps/broker/src/policy.tspins thefro-bot/agentharness integrate identity —repository_id1126485011,repository_owner_id80104189, andjob_workflow_reffro-bot/agent/.github/workflows/harness-integrate.yaml@refs/heads/main. The broker pinsjob_workflow_ref(the reusable integrate file that requests the token), notworkflow_ref(which is theharness-releasecaller).repository_visibilityispublic(fro-bot/agent is a public repo). Values sourced fromfro-bot/agent#1081and verified against the live GitHub API. If the integrate workflow file, ref, or repo identity changes, update the policy and redeploy. - Cross-repo integration: the consuming side (OIDC token request at audience
https://broker.fro.bot, broker call,auth.jsoninjection) lands infro-bot/agentviaharness-integrate.yaml— seefro-bot/agent#1081. The mint fails closed until this policy matches the integrate workflow’s OIDC claims. - Broker→cliproxy reachability: the broker runs on its own separate DigitalOcean droplet (not co-located with cliproxy). It reaches cliproxy over the public internet via
https://cliproxy.fro.bot. The hairpin concern (DO droplets don’t NAT-loopback) does not apply here — that only affects same-host container-to-container calls. - Pre-first-deploy checklist: create the
brokerGitHub Environment (reviewer + main-only) before merging the deploy workflow; addbroker-scoped secrets after the environment is gated; pin broker FQDN host keys in both.github/known_hostsandpackages/cli/src/resources/known_hosts(byte-identical). - First deploy: budget for a live contract cascade (host-key domain pin, SSH identity, paths-filter glob, ControlMaster). After the first deploy, author a
docs/solutions/compound doc capturing cascade waves and any mint/revoke surprises. - Secrets:
BROKER_SSH_KEY,BROKER_HOST, andCLIPROXY_MANAGEMENT_KEYare scoped to thebrokerGitHub Environment.DIGITALOCEAN_ACCESS_TOKENis repo-level. - Run provisioning via the root wrapper:
bun run provision:broker(loads the repo-root.env;bun run --cwd apps/broker provisionwould miss it).