VPN Deploy Package
The VPN box is a single-user WireGuard egress box on AWS Lightsail (eu-west-1, Ireland) — the first AWS-backed deployable in this repo. Native wg-quick@wg0 + systemd (no Docker). Provisioned via @aws-sdk/client-lightsail; deployed over SSH. The static IP is the durable client-facing endpoint; no domain in v1.
WHERE TO LOOK
Section titled “WHERE TO LOOK”| Task | Location | Notes |
|---|---|---|
| Deploy script | apps/vpn/src/deploy.ts |
Server-key preservation, wg0.conf render, health gate |
| Provision script | apps/vpn/server/provision.ts |
Lightsail SDK; one-time. Refuses re-run without --force |
| Peer model | packages/cli/src/commands/vpn/peers.ts (published as @marcusrbrown/infra/vpn/peers, imported by apps/vpn/src/deploy.ts) |
peers.json read/write, next-IP allocation, config rendering |
| CLI commands | packages/cli/src/commands/vpn/ |
status, deploy, logs, client add|list|remove |
| Peer config | apps/vpn/config/peers.json |
Local-only working state (gitignored). Roster is stored in the VPN_PEERS GitHub Environment secret. |
| Client output | apps/vpn/clients/ |
Gitignored — client .conf files with private keys |
| Config template | apps/vpn/config/wg0.conf.template |
Documents the server stanza shape (no secrets) |
| IP forwarding | apps/vpn/config/wg-forwarding.conf |
sysctl.d snippet shipped to the box on deploy |
DEPLOY FLOW
Section titled “DEPLOY FLOW”Deploy is SSH-only. No Docker, no Compose. The box runs native wg-quick@wg0 under systemd.
- Validate env + host —
VPN_HOSTand SSH context (SSH_AUTH_SOCKlocal orVPN_SSH_KEYin CI) are required.validateVpnHostrejects--prefixed values and characters outside the allowed alphabet before any SSH argv is constructed. - Materialize SSH key (CI only) —
VPN_SSH_KEYis written to a0600temp file viamaterializeIdentityFile; cleaned up after. - Harden memory — after the wipe guard, before any WireGuard change: masks
fwupd-refresh.timer,fwupd-refresh.serviceandfwupd.service, and creates a 1 GiB/swapfileonly if no swap is active, then reconciles its/etc/fstabentry. Runs every deploy; a failed or unrecognised probe aborts before the box changes (see Memory hardening). - Ensure server key —
ssh "umask 077; test -f /etc/wireguard/server.key || (wg genkey | tee /etc/wireguard/server.key | wg pubkey > /etc/wireguard/server.pub)". Atomic; preserved on redeploy unless--force-server-key. The server PRIVATE key never leaves the box. - Read back server.pub — only the public key is read back for status/diagnosis. The private key is never transmitted.
- Render wg0.conf server-side — the peer roster (from
VPN_PEERSenv var in CI, or localpeers.jsonfor local deploys) is used to render the config; the box assembles/etc/wireguard/wg0.confreferencing its local/etc/wireguard/server.key. Config bytes go through SSH stdin, never argv. - Write ip_forward snippet —
wg-forwarding.confis written to/etc/sysctl.d/;sysctl --systemapplies it. - Control wg-quick@wg0 —
systemctl enable --now wg-quick@wg0thensystemctl restart wg-quick@wg0to pick up the new config. - Health gate —
wg show wg0confirms the interface is up and the expected peer count is present. Freshness, not just liveness.
Memory hardening
Section titled “Memory hardening”The box is a Lightsail nano: ~412 MB usable RAM, no swap by default. The stock daily fwupd-refresh job (firmware metadata; useless on a cloud VM) pushed it into memory pressure, systemd-networkd dropped ens5, and the box sat off the network while Lightsail still showed it running. Masking is reversible and installs nothing; fwupd is not removed.
Swap is a 1 GiB /swapfile (~2.5x RAM, 5% of the 20 GB disk; more only prolongs thrashing before the OOM killer acts), root-owned 0600, added to /etc/fstab once. If any swap is already active the step creates nothing. The fstab entry is reconciled on every deploy while our /swapfile is active or being created, so a run interrupted after swapon self-heals; when only unrelated swap is active, the step leaves swap and fstab alone (if both are active, /swapfile wins and its entry is reconciled). The step touches nothing under /etc/wireguard and nothing network-related. Recognition and recovery: docs/runbooks/vpn-egress-box.md.
Server-key preservation invariant
Section titled “Server-key preservation invariant”The server private key is generated once on the box and persisted at /etc/wireguard/server.key (mode 0600). Deploy preserves it on every redeploy. --force-server-key rotates it (warned). Rotating the server key invalidates all existing client configs — every peer must receive a new client config after rotation.
Server-side config render
Section titled “Server-side config render”wg0.conf is rendered declaratively on the box from peers.json + the persisted server private key. A removed peer is genuinely revoked on the next deploy. Emergency manual edits to /etc/wireguard/wg0.conf are overwritten by the next deploy.
ONE-TIME PROVISIONING
Section titled “ONE-TIME PROVISIONING”Prerequisites:
- Dedicated least-privilege Lightsail IAM user created (see IAM note below);
VPN_AWS_ACCESS_KEY_ID+VPN_AWS_SECRET_ACCESS_KEYin the repo-root.env. These are distinct from the gateway’sAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY(S3-scoped, lacks Lightsail permissions). wg-egressEd25519 keypair generated; public key available;VPN_SSH_KEY(private key) in.envvpnGitHub Environment created with required reviewer + main-only branch policy (pre-create before merge — auto-create is ungated)
Run:
bun run provision:vpn(Root wrapper — loads the repo-root .env; --cwd apps/vpn would miss it.)
The script will:
- Validate
VPN_AWS_ACCESS_KEY_IDandVPN_AWS_SECRET_ACCESS_KEYare present - Reject if the
wg-egressinstance already exists (aborts;--forceto override) - Import the
wg-egressEd25519 public key into Lightsail (idempotent — skips if already present) - Resolve the current Ubuntu LTS blueprint ID and smallest IPv4 bundle ID live via
GetBlueprintsCommand/GetBundlesCommand(never hardcoded) - Create the instance in
eu-west-1a, poll untilstate.name === 'running' - Allocate + attach the
wg-egress-ipstatic IP - Set the exact firewall ruleset: SSH 22 (tcp) + UDP 51820 — closes Lightsail’s default 80/443 (
PutInstancePublicPortsCommandreplaces the whole set; SSH 22 is always included to prevent lockout) - Wait for SSH (
waitForSshshared helper) - Install WireGuard:
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y wireguard - Pin the IP host key into
.github/known_hosts(IP-only in v1; fail closed if pinning fails) - Print the allocated static IP — operator seeds this as
VPN_HOSTinto.env+ thevpnGitHub Environment before the first deploy
After provisioning: commit the updated .github/known_hosts before the first CI deploy.
Bootstrap ordering
Section titled “Bootstrap ordering”The VPN_HOST cycle: the static IP is unknown until provisioning runs. Sequence:
- Seed
VPN_AWS_ACCESS_KEY_ID,VPN_AWS_SECRET_ACCESS_KEY,VPN_SSH_KEYinto.env bun run provision:vpn— allocates the static IP and prints it- Seed the printed IP as
VPN_HOSTinto.env+ thevpnGitHub Environment - Commit the updated
.github/known_hosts - First deploy:
bunx @marcusrbrown/infra vpn deploy
VPN_AWS_* credentials are used only during provisioning. They are not required for deploy or status — those reach the box over SSH only. Optionally set VPN_AWS_REGION to override the default eu-west-1 (note: the availability zone is hardcoded to eu-west-1a — only eu-west-1 is supported).
IAM note
Section titled “IAM note”Lightsail IAM is action-scoped, not ARN-scoped — many Lightsail actions require Resource: *. The provisioning IAM user holds the minimal Lightsail action set needed for provisioning only (instance create/get, static IP allocate/attach/get, firewall set, key pair import, blueprint/bundle list). It is a dedicated user separate from the gateway’s S3 keys. AWS credentials are bounded to the provisioning path — deploy and status never need them.
Ed25519 key import: ImportKeyPairCommand is called with the Ed25519 public key encoded as base64. Lightsail historically documented RSA keys; Ed25519 import is attempted. If Lightsail rejects Ed25519 at runtime, fall back to an RSA key for this box only and record the outcome here.
REQUIRED SECRETS
Section titled “REQUIRED SECRETS”| Secret | Required | Description |
|---|---|---|
VPN_SSH_KEY |
✓ | Ed25519 private key for the VPN box (wg-egress keypair) |
VPN_HOST |
✓ | Static IP of the Lightsail instance (printed by provisioning) |
VPN_PEERS |
— | Peer roster JSON ({"peers":[{"name":...,"publicKey":...,"tunnelIp":...}]}). Auto-synced by vpn client add/remove. An empty roster is valid (first deploy before any peers). |
All three secrets are scoped to the vpn GitHub Environment. AWS provisioning credentials (VPN_AWS_ACCESS_KEY_ID, VPN_AWS_SECRET_ACCESS_KEY, and optional VPN_AWS_REGION) are operator-local only — they are NOT in the vpn Environment and are never used by the deploy or status paths.
CLI COMMANDS
Section titled “CLI COMMANDS”| Command | Purpose |
|---|---|
bunx @marcusrbrown/infra vpn status |
SSH to box, run wg show wg0, show interface state + server public key + peer count |
bunx @marcusrbrown/infra vpn deploy |
Trigger the deploy workflow via gh workflow run (remote, default). --local runs the deploy script directly (requires SSH_AUTH_SOCK). --force-server-key rotates the server key (invalidates all client configs). |
bunx @marcusrbrown/infra vpn logs [--tail N] |
Stream journalctl -u wg-quick@wg0 from the box. Logs may reveal peer IPs; operator-only via SSH boundary. |
bunx @marcusrbrown/infra vpn client add <name> |
Generate a client keypair locally, assign the next tunnel IP, append the public key to peers.json, sync the roster to the VPN_PEERS secret, write the client .conf to apps/vpn/clients/. Run vpn deploy explicitly to apply the change. |
bunx @marcusrbrown/infra vpn client list |
List peers from the local peers.json (name, tunnel IP, public key). |
bunx @marcusrbrown/infra vpn client remove <name> |
Remove a peer from peers.json and sync the roster to the VPN_PEERS secret. Run vpn deploy explicitly to revoke the peer on the box. |
vpn status is MCP-exposed (read-only). All other vpn commands are CLI-only (mutating or sensitive).
PEER MODEL
Section titled “PEER MODEL”The peer roster is stored in the VPN_PEERS GitHub Environment secret and in a local gitignored file apps/vpn/config/peers.json. The schema: {"peers":[{"name":...,"publicKey":...,"tunnelIp":...}]} (Zod-validated). Public keys and tunnel IPs only — never private material.
vpn client add and vpn client remove write the updated roster to the local peers.json and then sync it to the VPN_PEERS secret via gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.json. If the gh sync fails (missing binary, auth, network), the command still succeeds and prints a warning with the exact remediation command. The operator runs vpn deploy explicitly to apply the change to the box. vpn client list reads the local file.
Stale-secret hazard: if the gh sync fails and CI runs a deploy before the secret is re-synced, the stale roster in VPN_PEERS is applied to the box. The wipe guard blocks an empty-roster deploy (see below), but a stale non-empty roster deploys silently. Re-sync with:
gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.jsonIn CI, the deploy reads the roster from VPN_PEERS. Locally (no env var set), it falls back to the local peers.json. A malformed VPN_PEERS value (invalid JSON or wrong schema) fails the deploy immediately — no file fallback when the env var is set.
Wipe guard: when the incoming roster has 0 peers, the deploy queries the live box peer count (wg show wg0 dump) before applying config. If the live box has peers and the incoming roster is empty, the deploy aborts with a clear error — this prevents silently disconnecting all clients when VPN_PEERS is unset or empty in CI. Set VPN_ALLOW_EMPTY_ROSTER=1 to intentionally remove all peers. First-boot (live=0, incoming=0) proceeds normally without the override.
Tunnel IPs are sequential 10.8.0.N/32 starting at .2 (.1 is reserved for the server). client remove frees the slot; client add reuses freed slots.
Client .conf files (containing the client private key) are written only to apps/vpn/clients/ — a gitignored directory. The deploy refuses to write a client config to a git-tracked path.
DAY-2 OPERATIONS
Section titled “DAY-2 OPERATIONS”- Monitoring —
vpn statusfor live interface state;vpn logsto stream journalctl output. - Roll back a bad deploy — the deploy is idempotent and preserves the server key. Fix the underlying issue and retry.
- Add a peer —
vpn client add <name>. The client.confis written toapps/vpn/clients/<name>.conf. - Remove a peer —
vpn client remove <name>. Updates the local roster and syncs theVPN_PEERSsecret. Runvpn deployexplicitly to revoke the peer on the box. - Rotate the server key —
vpn deploy --force-server-key. All existing client configs become invalid; reissue every client config after rotation. - Reprovision (fresh disk) — see Reprovision recovery below.
REPROVISION RECOVERY
Section titled “REPROVISION RECOVERY”Reprovisioning creates a fresh disk, which destroys the server private key at /etc/wireguard/server.key. The static IP is re-attached (endpoint unchanged), but all clients fail the cryptographic handshake — a silent break. vpn status surfaces the server public key so a key mismatch is diagnosable.
Two recovery paths:
-
Backup/restore for continuity — before reprovisioning, back up
/etc/wireguard/server.keyfrom the old box (operator-manual,scporssh cat). After reprovisioning and first deploy, restore the key to/etc/wireguard/server.key(mode0600) and restartwg-quick@wg0. Existing client configs continue to work. -
Regenerate and reissue — let the new box generate a fresh server key on first deploy. Run
vpn client addfor every peer to generate new client configs. Distribute the new configs to each peer.
The runbook in docs/runbooks/vpn-egress-box.md covers the full reprovision sequence.
ANTI-PATTERNS
Section titled “ANTI-PATTERNS”- Never
ssh-keyscanin CI — host keys are pinned in.github/known_hostsat provision time and committed. Provisioning scripts may usessh-keyscanlocally. - Never pass secret bytes via argv — config bytes, peer data, and the
VPN_PEERSroster go through SSH stdin orghstdin only.--body <value>patterns are banned. - Never track
apps/vpn/config/peers.jsonin git — the roster is gitignored; it lives in theVPN_PEERSsecret and the local file.git rm --cachedif accidentally staged. - Never skip
validateVpnHost— it rejects--prefixed values and characters outside the allowed alphabet. SSH treats--prefixed hostnames as flags (including-oProxyCommand=). - Never read the server private key to the local machine or CI runner — deploy reads back only
server.pub. The private key stays on the box. - Never hardcode
blueprintIdorbundleId— resolve live viaGetBlueprintsCommand/GetBundlesCommand. Hardcoded IDs risk a non-existent-ID failure. - Never omit SSH 22 from
PutInstancePublicPortsCommand— the command replaces the entire ruleset; omitting SSH 22 locks out the operator. The firewall ruleset always includes both SSH 22 (tcp) and UDP 51820. - Never read a failed remote probe as an answer — a non-zero exit is not proof of “no swap” or “no fstab line”. Probes print a sentinel, and deploy requires exit 0 plus a recognised sentinel, or it throws. A failed probe read as “none” creates a second swap file or a duplicate fstab line.
- Never use Docker for the VPN runtime — native
wg-quick@wg0+ systemd is the correct pattern for a single-box WireGuard setup. - Never use AWS credentials in the deploy or status paths — AWS creds are provisioning-only. Deploy and status reach the box over SSH; the MCP runtime needs no AWS credentials.
PRIVACY POSTURE
Section titled “PRIVACY POSTURE”The box is an egress relay for a single user. WireGuard logs no traffic by default. journalctl output (peer IPs, connection events) is accessible only via vpn logs (CLI-only, SSH-gated, never MCP). The operator controls what is logged and for how long via standard systemd journal retention on the box.
DECOMMISSIONING
Section titled “DECOMMISSIONING”vpn client remove <name>for each peer (or skip if decommissioning entirely).- Delete the
wg-egressinstance andwg-egress-ipstatic IP in the AWS Lightsail console. - Remove the VPN IP entry from
.github/known_hosts. - Delete the
vpnGitHub Environment. - Terminate the old
us-west-2OpenVPN EC2 instance (manual AWS console step — seedocs/runbooks/vpn-egress-box.md).