VPN Egress Box
The VPN box is a WireGuard egress box on AWS Lightsail (eu-west-1, Ireland), managed as a first-class deployable in this repo. This runbook covers operator bootstrap, reprovision recovery, client onboarding, and old-EC2 teardown. For the deploy flow and anti-patterns, see apps/vpn/AGENTS.md.
Why Ireland (eu-west-1)
Section titled “Why Ireland (eu-west-1)”Ireland is the closest AWS region that provides a non-georestricted egress point for the operator’s use cases. It is the lowest-latency option that satisfies the operator’s routing requirements without relying on a US-based exit. The region choice is durable: the static IP is the client-facing endpoint, and changing regions requires reprovisioning (new static IP, new client configs). Lightsail Ireland is also the cheapest reliable option (~$5/mo flat rate for the smallest IPv4 bundle).
Prerequisites
Section titled “Prerequisites”Before running provisioning:
-
Create a dedicated least-privilege Lightsail IAM user in the AWS console. Attach a policy granting only the Lightsail actions needed for provisioning (instance create/get, static IP allocate/attach/get, firewall set, key pair import, blueprint/bundle list). This user is separate from the gateway’s S3 keys. Lightsail IAM is action-scoped — many actions require
Resource: *; scope by action, not ARN. -
Generate the
wg-egressEd25519 keypair locally:Terminal window ssh-keygen -t ed25519 -f ~/.ssh/wg-egress -C wg-egress -N ''The public key (
~/.ssh/wg-egress.pub) is imported into Lightsail by the provisioner. The private key isVPN_SSH_KEY. -
Create the
vpnGitHub Environment in the repo settings with:- Required reviewer set
- Main-only branch policy (deployment branches:
mainonly)
Pre-create before merging the VPN feature branch — GitHub auto-creates environments ungated on first workflow reference.
-
Seed the repo-root
.envwith:Terminal window VPN_AWS_ACCESS_KEY_ID=<provisioning-iam-access-key>VPN_AWS_SECRET_ACCESS_KEY=<provisioning-iam-secret-key># VPN_AWS_REGION=eu-west-1 # optional; defaults to eu-west-1VPN_SSH_KEY=<contents of ~/.ssh/wg-egress>VPN_AWS_*credentials are provisioning-only and distinct from the gateway’sAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY(S3-scoped). They are not seeded into thevpnGitHub Environment.
Operator Bootstrap
Section titled “Operator Bootstrap”The VPN_HOST cycle: the static IP is unknown until provisioning runs. Follow this sequence exactly.
Step 1: Provision the box
Section titled “Step 1: Provision the box”bun run provision:vpnThe script:
- Imports the
wg-egressEd25519 public key into Lightsail - Resolves the current Ubuntu LTS blueprint and smallest IPv4 bundle live
- Creates the instance in
eu-west-1a - Allocates + attaches the
wg-egress-ipstatic IP - Sets the exact firewall: SSH 22 (tcp) + UDP 51820 (udp) — closes Lightsail’s default 80/443
- Waits for SSH
- Installs WireGuard (
apt-get install -y wireguard) - Pins the IP host key into
.github/known_hosts - Prints the allocated static IP
Step 2: Seed VPN_HOST
Section titled “Step 2: Seed VPN_HOST”Take the printed static IP and seed it:
# In the repo-root .env:VPN_HOST=<printed-static-ip>
# In the vpn GitHub Environment (via gh CLI or the GitHub UI):printf '%s' '<printed-static-ip>' | gh secret set --env vpn VPN_HOSTprintf '%s' "$(cat ~/.ssh/wg-egress)" | gh secret set --env vpn VPN_SSH_KEYStep 3: Commit .github/known_hosts
Section titled “Step 3: Commit .github/known_hosts”git add .github/known_hostsgit commit -m "chore: pin VPN box IP host key"git pushStep 4: First deploy
Section titled “Step 4: First deploy”bunx @marcusrbrown/infra vpn deployApprove the vpn environment gate in the GitHub Actions UI. The first deploy:
- Generates the server keypair on the box (atomic,
0600) - Renders and writes
/etc/wireguard/wg0.confserver-side - Enables and starts
wg-quick@wg0 - Health-gates on
wg show wg0
Step 5: Verify
Section titled “Step 5: Verify”bunx @marcusrbrown/infra vpn statusConfirm the interface is up and the server public key is displayed.
Client Onboarding
Section titled “Client Onboarding”Add a new WireGuard peer:
bunx @marcusrbrown/infra vpn client add <name>This:
- Generates a client keypair locally
- Assigns the next available tunnel IP (
10.8.0.N/32, sequential from.2) - Appends the client public key + tunnel IP to the local
apps/vpn/config/peers.json(gitignored) - Syncs the updated roster to the
VPN_PEERSGitHub Environment secret viagh secret set VPN_PEERS --env vpn - Writes the client
.conf(with client private key) toapps/vpn/clients/<name>.conf
If the gh sync fails (missing binary, auth, or network), the command still succeeds and prints a warning with the exact remediation command:
gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.jsonStale-secret hazard: if the sync fails and CI runs a deploy before the secret is re-synced, the stale roster in VPN_PEERS is applied to the box. The wipe guard (see below) blocks an empty-roster deploy, but a stale non-empty roster deploys silently. Re-sync with the command above.
The client .conf path is printed. Distribute it to the peer device securely (never via unencrypted channels). The apps/vpn/clients/ directory is gitignored — client private keys never enter the repo.
To list current peers:
bunx @marcusrbrown/infra vpn client listTo remove a peer:
bunx @marcusrbrown/infra vpn client remove <name>The peer is removed from the local peers.json and the VPN_PEERS secret is synced. Run vpn deploy explicitly to revoke the peer on the box. The client config on the peer device becomes invalid after the next deploy completes.
Wipe Guard
Section titled “Wipe Guard”The deploy includes a wipe guard that prevents silently disconnecting all clients when the incoming peer roster is empty. If VPN_PEERS is unset or empty in CI and the live box has peers, the deploy aborts with:
Wipe guard: the incoming roster is empty (0 peers) but the live box has N peer(s).Deploying an empty roster would disconnect all clients.To fix: re-sync the roster secret with: gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.jsonTo intentionally remove all peers, set VPN_ALLOW_EMPTY_ROSTER=1.What it protects against: an empty or missing VPN_PEERS secret in CI would deploy a 0-peer config to a box with live peers, silently disconnecting every client. The health gate would pass (expected 0 == actual 0), making the failure invisible.
When to override: set VPN_ALLOW_EMPTY_ROSTER=1 in the deploy environment only when intentionally removing all peers. First-boot bootstrap (live=0, incoming=0) proceeds normally without the override.
Remediation for a blocked deploy:
# Re-sync the local roster to the VPN_PEERS secret:gh secret set VPN_PEERS --env vpn --repo marcusrbrown/infra < apps/vpn/config/peers.json# Then re-trigger the deploy.Box Unreachable: Memory Pressure
Section titled “Box Unreachable: Memory Pressure”The box is a Lightsail nano (~412 MB RAM). Under memory pressure systemd-networkd can lose the interface and the box stays off the network while still “running”.
Recognise it:
-
vpn statusandvpn deployfail with SSH connect timeouts, while the Lightsail console shows the instance as running and the static IP still attached. -
Reboot the instance from the Lightsail console, then SSH in and check the previous boot:
Terminal window journalctl -b -1 | grep -E 'Under memory pressure|Could not set route|ens5: Failed|Out of memory'“Under memory pressure” followed by
ens5: Could not set route: Connection timed outandens5: Failedis this failure.journalctl -b -1 -k | grep -i 'killed process'shows the OOM kills.
Recover: the reboot restores networking. Run vpn deploy — it masks fwupd-refresh.timer/fwupd.service and creates a 1 GiB swap file if no swap is active, so the trigger does not return. Verify with swapon --show and systemctl is-enabled fwupd-refresh.timer (expect masked) over SSH.
Cause and fix (2026-11-12): the stock daily fwupd-refresh job pushed the box into memory pressure; the kernel had OOM-killed fwupd repeatedly before that. The deploy now runs a memory-hardening step on every run (see apps/vpn/AGENTS.md). If the box goes unreachable again with those in place, something else is consuming memory — check journalctl -b -1 -k for the process the OOM killer named.
Reprovision Recovery
Section titled “Reprovision Recovery”Reprovisioning creates a fresh disk, which destroys the server private key at /etc/wireguard/server.key. The static IP is re-attached (endpoint unchanged), but all clients fail the cryptographic handshake — a silent break. vpn status surfaces the server public key so a key mismatch is diagnosable (the new public key differs from what clients have).
Option A: Backup/restore for continuity
Section titled “Option A: Backup/restore for continuity”Before reprovisioning, back up the server private key from the old box:
ssh root@<VPN_HOST> 'cat /etc/wireguard/server.key' > /tmp/server.key.bakchmod 600 /tmp/server.key.bakStore the backup securely (1Password, encrypted storage). After reprovisioning and the first deploy:
# Restore the server keycat /tmp/server.key.bak | ssh root@<VPN_HOST> 'cat > /etc/wireguard/server.key && chmod 600 /etc/wireguard/server.key'# Regenerate server.pub from the restored keyssh root@<VPN_HOST> 'wg pubkey < /etc/wireguard/server.key > /etc/wireguard/server.pub'# Restart the servicessh root@<VPN_HOST> 'systemctl restart wg-quick@wg0'Verify with vpn status — the server public key must match the pre-reprovision value. Existing client configs continue to work.
Option B: Regenerate and reissue
Section titled “Option B: Regenerate and reissue”Let the new box generate a fresh server key on first deploy. Then reissue every client config:
# For each peer:bunx @marcusrbrown/infra vpn client remove <name>bunx @marcusrbrown/infra vpn client add <name>Distribute the new client configs to each peer device. The old configs are permanently invalid.
Secret Rotation
Section titled “Secret Rotation”Rotate VPN_SSH_KEY
Section titled “Rotate VPN_SSH_KEY”- Generate a new
wg-egressEd25519 keypair. - Add the new public key to the box’s
~/.ssh/authorized_keys(while the old key still works). - Update
VPN_SSH_KEYin thevpnGitHub Environment. - Trigger a deploy to verify the new key works.
- Remove the old public key from the box’s
authorized_keys.
Rotate the WireGuard server key
Section titled “Rotate the WireGuard server key”bunx @marcusrbrown/infra vpn deploy --force-server-keyThis regenerates the server keypair on the box. All existing client configs become invalid — reissue every client config after rotation (see Client Onboarding).
Old us-west-2 OpenVPN EC2 Teardown
Section titled “Old us-west-2 OpenVPN EC2 Teardown”The old OpenVPN EC2 instance in us-west-2 is unused and no longer free-tier. Terminate it manually in the AWS console:
- Open the EC2 console in
us-west-2. - Locate the old OpenVPN instance.
- Select it → Instance state → Terminate instance.
- Confirm termination.
If the instance has an associated Elastic IP, release it after termination (Elastic IPs incur charges when unattached).
This is a manual, irreversible step. Verify no active clients depend on the old instance before terminating.
Related
Section titled “Related”apps/vpn/AGENTS.md— deploy flow, server-key invariants, provisioning, anti-patternsapps/vpn/README.md— quick-start, secrets table, CLI referencedocs/solutions/workflow-issues/gateway-first-deploy-cascade-2026-05-20.md— SSH key trailing-newline and ControlMaster lessons (apply to VPN deploy)docs/solutions/workflow-issues/umami-first-deploy-cascade-2026-05-29.md— GitHub Environment pre-create lesson (applies tovpnenvironment)