Skip to content

VPN Deploy Package

The VPN box is a single-user WireGuard egress box on AWS Lightsail (eu-west-1, Ireland) — the first AWS-backed deployable in this repo. Native wg-quick@wg0 + systemd (no Docker). Provisioned via @aws-sdk/client-lightsail; deployed over SSH. The static IP is the durable client-facing endpoint; no domain in v1.

Task Location Notes
Deploy script apps/vpn/src/deploy.ts Server-key preservation, wg0.conf render, health gate
Provision script apps/vpn/server/provision.ts Lightsail SDK; one-time. Refuses re-run without --force
Peer model packages/cli/src/commands/vpn/peers.ts (published as @marcusrbrown/infra/vpn/peers, imported by apps/vpn/src/deploy.ts) peers.json read/write, next-IP allocation, config rendering
CLI commands packages/cli/src/commands/vpn/ status, deploy, logs, client add|list|remove
Peer config apps/vpn/config/peers.json Local-only working state (gitignored). Roster is stored in the VPN_PEERS GitHub Environment secret.
Client output apps/vpn/clients/ Gitignored — client .conf files with private keys
Config template apps/vpn/config/wg0.conf.template Documents the server stanza shape (no secrets)
IP forwarding apps/vpn/config/wg-forwarding.conf sysctl.d snippet shipped to the box on deploy

Deploy is SSH-only. No Docker, no Compose. The box runs native wg-quick@wg0 under systemd.

  1. Validate env + host — VPN_HOST and SSH context (SSH_AUTH_SOCK local or VPN_SSH_KEY in CI) are required. validateVpnHost rejects --prefixed values and characters outside the allowed alphabet before any SSH argv is constructed.
  2. Materialize SSH key (CI only) — VPN_SSH_KEY is written to a 0600 temp file via materializeIdentityFile; cleaned up after.
  3. Harden memory — after the wipe guard, before any WireGuard change: masks fwupd-refresh.timer, fwupd-refresh.service and fwupd.service, and creates a 1 GiB /swapfile only if no swap is active, then reconciles its /etc/fstab entry. Runs every deploy; a failed or unrecognised probe aborts before the box changes (see Memory hardening).
  4. Ensure server key — ssh "umask 077; test -f /etc/wireguard/server.key || (wg genkey | tee /etc/wireguard/server.key | wg pubkey > /etc/wireguard/server.pub)". Atomic; preserved on redeploy unless --force-server-key. The server PRIVATE key never leaves the box.
  5. Read back server.pub — only the public key is read back for status/diagnosis. The private key is never transmitted.
  6. Render wg0.conf server-side — the peer roster (from VPN_PEERS env var in CI, or local peers.json for local deploys) is used to render the config; the box assembles /etc/wireguard/wg0.conf referencing its local /etc/wireguard/server.key. Config bytes go through SSH stdin, never argv.
  7. Write ip_forward snippet — wg-forwarding.conf is written to /etc/sysctl.d/; sysctl --system applies it.
  8. Control wg-quick@wg0 — systemctl enable --now wg-quick@wg0 then systemctl restart wg-quick@wg0 to pick up the new config.
  9. Health gate — wg show wg0 confirms the interface is up and the expected peer count is present. Freshness, not just liveness.

The box is a Lightsail nano: ~412 MB usable RAM, no swap by default. The stock daily fwupd-refresh job (firmware metadata; useless on a cloud VM) pushed it into memory pressure, systemd-networkd dropped ens5, and the box sat off the network while Lightsail still showed it running. Masking is reversible and installs nothing; fwupd is not removed.

Swap is a 1 GiB /swapfile (~2.5x RAM, 5% of the 20 GB disk; more only prolongs thrashing before the OOM killer acts), root-owned 0600, added to /etc/fstab once. If any swap is already active the step creates nothing. The fstab entry is reconciled on every deploy while our /swapfile is active or being created, so a run interrupted after swapon self-heals; when only unrelated swap is active, the step leaves swap and fstab alone (if both are active, /swapfile wins and its entry is reconciled). The step touches nothing under /etc/wireguard and nothing network-related. Recognition and recovery: docs/runbooks/vpn-egress-box.md.

The server private key is generated once on the box and persisted at /etc/wireguard/server.key (mode 0600). Deploy preserves it on every redeploy. --force-server-key rotates it (warned). Rotating the server key invalidates all existing client configs — every peer must receive a new client config after rotation.

wg0.conf is rendered declaratively on the box from peers.json + the persisted server private key. A removed peer is genuinely revoked on the next deploy. Emergency manual edits to /etc/wireguard/wg0.conf are overwritten by the next deploy.

Prerequisites:

  • Dedicated least-privilege Lightsail IAM user created (see IAM note below); VPN_AWS_ACCESS_KEY_ID + VPN_AWS_SECRET_ACCESS_KEY in the repo-root .env. These are distinct from the gateway’s AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (S3-scoped, lacks Lightsail permissions).
  • wg-egress Ed25519 keypair generated; public key available; VPN_SSH_KEY (private key) in .env
  • vpn GitHub Environment created with required reviewer + main-only branch policy (pre-create before merge — auto-create is ungated)

Run:

Terminal window
bun run provision:vpn

(Root wrapper — loads the repo-root .env; --cwd apps/vpn would miss it.)

The script will:

  1. Validate VPN_AWS_ACCESS_KEY_ID and VPN_AWS_SECRET_ACCESS_KEY are present
  2. Reject if the wg-egress instance already exists (aborts; --force to override)
  3. Import the wg-egress Ed25519 public key into Lightsail (idempotent — skips if already present)
  4. Resolve the current Ubuntu LTS blueprint ID and smallest IPv4 bundle ID live via GetBlueprintsCommand/GetBundlesCommand (never hardcoded)
  5. Create the instance in eu-west-1a, poll until state.name === 'running'
  6. Allocate + attach the wg-egress-ip static IP
  7. Set the exact firewall ruleset: SSH 22 (tcp) + UDP 51820 — closes Lightsail’s default 80/443 (PutInstancePublicPortsCommand replaces the whole set; SSH 22 is always included to prevent lockout)
  8. Wait for SSH (waitForSsh shared helper)
  9. Install WireGuard: sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y wireguard
  10. Pin the IP host key into .github/known_hosts (IP-only in v1; fail closed if pinning fails)
  11. Print the allocated static IP — operator seeds this as VPN_HOST into .env + the vpn GitHub Environment before the first deploy

After provisioning: commit the updated .github/known_hosts before the first CI deploy.

The VPN_HOST cycle: the static IP is unknown until provisioning runs. Sequence:

  1. Seed VPN_AWS_ACCESS_KEY_ID, VPN_AWS_SECRET_ACCESS_KEY, VPN_SSH_KEY into .env
  2. bun run provision:vpn — allocates the static IP and prints it
  3. Seed the printed IP as VPN_HOST into .env + the vpn GitHub Environment
  4. Commit the updated .github/known_hosts
  5. First deploy: bunx @marcusrbrown/infra vpn deploy

VPN_AWS_* credentials are used only during provisioning. They are not required for deploy or status — those reach the box over SSH only. Optionally set VPN_AWS_REGION to override the default eu-west-1 (note: the availability zone is hardcoded to eu-west-1a — only eu-west-1 is supported).

Lightsail IAM is action-scoped, not ARN-scoped — many Lightsail actions require Resource: *. The provisioning IAM user holds the minimal Lightsail action set needed for provisioning only (instance create/get, static IP allocate/attach/get, firewall set, key pair import, blueprint/bundle list). It is a dedicated user separate from the gateway’s S3 keys. AWS credentials are bounded to the provisioning path — deploy and status never need them.

Ed25519 key import: ImportKeyPairCommand is called with the Ed25519 public key encoded as base64. Lightsail historically documented RSA keys; Ed25519 import is attempted. If Lightsail rejects Ed25519 at runtime, fall back to an RSA key for this box only and record the outcome here.

Secret Required Description
VPN_SSH_KEY ✓ Ed25519 private key for the VPN box (wg-egress keypair)
VPN_HOST ✓ Static IP of the Lightsail instance (printed by provisioning)
VPN_PEERS — Peer roster JSON ({"peers":[{"name":...,"publicKey":...,"tunnelIp":...}]}). Auto-synced by vpn client add/remove. An empty roster is valid (first deploy before any peers).

All three secrets are scoped to the vpn GitHub Environment. AWS provisioning credentials (VPN_AWS_ACCESS_KEY_ID, VPN_AWS_SECRET_ACCESS_KEY, and optional VPN_AWS_REGION) are operator-local only — they are NOT in the vpn Environment and are never used by the deploy or status paths.

Command Purpose
bunx @marcusrbrown/infra vpn status SSH to box, run wg show wg0, show interface state + server public key + peer count
bunx @marcusrbrown/infra vpn deploy Trigger the deploy workflow via gh workflow run (remote, default). --local runs the deploy script directly (requires SSH_AUTH_SOCK). --force-server-key rotates the server key (invalidates all client configs).
bunx @marcusrbrown/infra vpn logs [--tail N] Stream journalctl -u wg-quick@wg0 from the box. Logs may reveal peer IPs; operator-only via SSH boundary.
bunx @marcusrbrown/infra vpn client add <name> Generate a client keypair locally, assign the next tunnel IP, append the public key to peers.json, sync the roster to the VPN_PEERS secret, write the client .conf to apps/vpn/clients/. Run vpn deploy explicitly to apply the change.
bunx @marcusrbrown/infra vpn client list List peers from the local peers.json (name, tunnel IP, public key).
bunx @marcusrbrown/infra vpn client remove <name> Remove a peer from peers.json and sync the roster to the VPN_PEERS secret. Run vpn deploy explicitly to revoke the peer on the box.

vpn status is MCP-exposed (read-only). All other vpn commands are CLI-only (mutating or sensitive).

The peer roster is stored in the VPN_PEERS GitHub Environment secret and in a local gitignored file apps/vpn/config/peers.json. The schema: {"peers":[{"name":...,"publicKey":...,"tunnelIp":...}]} (Zod-validated). Public keys and tunnel IPs only — never private material.

vpn client add and vpn client remove write the updated roster to the local peers.json and then sync it to the VPN_PEERS secret via gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.json. If the gh sync fails (missing binary, auth, network), the command still succeeds and prints a warning with the exact remediation command. The operator runs vpn deploy explicitly to apply the change to the box. vpn client list reads the local file.

Stale-secret hazard: if the gh sync fails and CI runs a deploy before the secret is re-synced, the stale roster in VPN_PEERS is applied to the box. The wipe guard blocks an empty-roster deploy (see below), but a stale non-empty roster deploys silently. Re-sync with:

Terminal window
gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.json

In CI, the deploy reads the roster from VPN_PEERS. Locally (no env var set), it falls back to the local peers.json. A malformed VPN_PEERS value (invalid JSON or wrong schema) fails the deploy immediately — no file fallback when the env var is set.

Wipe guard: when the incoming roster has 0 peers, the deploy queries the live box peer count (wg show wg0 dump) before applying config. If the live box has peers and the incoming roster is empty, the deploy aborts with a clear error — this prevents silently disconnecting all clients when VPN_PEERS is unset or empty in CI. Set VPN_ALLOW_EMPTY_ROSTER=1 to intentionally remove all peers. First-boot (live=0, incoming=0) proceeds normally without the override.

Tunnel IPs are sequential 10.8.0.N/32 starting at .2 (.1 is reserved for the server). client remove frees the slot; client add reuses freed slots.

Client .conf files (containing the client private key) are written only to apps/vpn/clients/ — a gitignored directory. The deploy refuses to write a client config to a git-tracked path.

  • Monitoring — vpn status for live interface state; vpn logs to stream journalctl output.
  • Roll back a bad deploy — the deploy is idempotent and preserves the server key. Fix the underlying issue and retry.
  • Add a peer — vpn client add <name>. The client .conf is written to apps/vpn/clients/<name>.conf.
  • Remove a peer — vpn client remove <name>. Updates the local roster and syncs the VPN_PEERS secret. Run vpn deploy explicitly to revoke the peer on the box.
  • Rotate the server key — vpn deploy --force-server-key. All existing client configs become invalid; reissue every client config after rotation.
  • Reprovision (fresh disk) — see Reprovision recovery below.

Reprovisioning creates a fresh disk, which destroys the server private key at /etc/wireguard/server.key. The static IP is re-attached (endpoint unchanged), but all clients fail the cryptographic handshake — a silent break. vpn status surfaces the server public key so a key mismatch is diagnosable.

Two recovery paths:

  1. Backup/restore for continuity — before reprovisioning, back up /etc/wireguard/server.key from the old box (operator-manual, scp or ssh cat). After reprovisioning and first deploy, restore the key to /etc/wireguard/server.key (mode 0600) and restart wg-quick@wg0. Existing client configs continue to work.

  2. Regenerate and reissue — let the new box generate a fresh server key on first deploy. Run vpn client add for every peer to generate new client configs. Distribute the new configs to each peer.

The runbook in docs/runbooks/vpn-egress-box.md covers the full reprovision sequence.

  • Never ssh-keyscan in CI — host keys are pinned in .github/known_hosts at provision time and committed. Provisioning scripts may use ssh-keyscan locally.
  • Never pass secret bytes via argv — config bytes, peer data, and the VPN_PEERS roster go through SSH stdin or gh stdin only. --body <value> patterns are banned.
  • Never track apps/vpn/config/peers.json in git — the roster is gitignored; it lives in the VPN_PEERS secret and the local file. git rm --cached if accidentally staged.
  • Never skip validateVpnHost — it rejects --prefixed values and characters outside the allowed alphabet. SSH treats --prefixed hostnames as flags (including -oProxyCommand=).
  • Never read the server private key to the local machine or CI runner — deploy reads back only server.pub. The private key stays on the box.
  • Never hardcode blueprintId or bundleId — resolve live via GetBlueprintsCommand/GetBundlesCommand. Hardcoded IDs risk a non-existent-ID failure.
  • Never omit SSH 22 from PutInstancePublicPortsCommand — the command replaces the entire ruleset; omitting SSH 22 locks out the operator. The firewall ruleset always includes both SSH 22 (tcp) and UDP 51820.
  • Never read a failed remote probe as an answer — a non-zero exit is not proof of “no swap” or “no fstab line”. Probes print a sentinel, and deploy requires exit 0 plus a recognised sentinel, or it throws. A failed probe read as “none” creates a second swap file or a duplicate fstab line.
  • Never use Docker for the VPN runtime — native wg-quick@wg0 + systemd is the correct pattern for a single-box WireGuard setup.
  • Never use AWS credentials in the deploy or status paths — AWS creds are provisioning-only. Deploy and status reach the box over SSH; the MCP runtime needs no AWS credentials.

The box is an egress relay for a single user. WireGuard logs no traffic by default. journalctl output (peer IPs, connection events) is accessible only via vpn logs (CLI-only, SSH-gated, never MCP). The operator controls what is logged and for how long via standard systemd journal retention on the box.

  1. vpn client remove <name> for each peer (or skip if decommissioning entirely).
  2. Delete the wg-egress instance and wg-egress-ip static IP in the AWS Lightsail console.
  3. Remove the VPN IP entry from .github/known_hosts.
  4. Delete the vpn GitHub Environment.
  5. Terminate the old us-west-2 OpenVPN EC2 instance (manual AWS console step — see docs/runbooks/vpn-egress-box.md).