Dashboard Released Image Rollback
The dashboard image is pinned by tag and digest in apps/dashboard/docker-compose.yaml. Renovate
opens PRs to bump the pin when a new ghcr.io/fro-bot/dashboard release is published; merging
triggers the Deploy Dashboard workflow, which pulls the new digest and ships it to the droplet.
This runbook covers rolling back to a prior image digest when a release is bad. For the deploy flow
and anti-patterns, see apps/dashboard/AGENTS.md.
Contract Changes Roll Back Paired
Section titled “Contract Changes Roll Back Paired”If the release being rolled back changed the operator contract, roll back together with the gateway: see Rollback and Rollback is not pin reversal. Never roll back only the dashboard across a contract change.
Why This Exists
Section titled “Why This Exists”The compose pin is the single source of truth for which image version runs on the droplet. Rolling back means reverting the compose pin commit to restore the prior digest, then deploying. There is no separate override file or runtime flag — the compose file is the contract.
A recent prior repository@digest may already be cached on the droplet and can speed rollback if the
registry is unavailable, but the reviewed Compose pin plus the normal deploy remains the source of
truth and authorization. Cache presence alone is not permission to change runtime state, and the next
deploy attempt may prune the cached image.
When to Use
Section titled “When to Use”- A newly deployed image causes errors, crashes, or health-check failures on
dashboard.fro.bot. https://dashboard.fro.bot/api/healthzreturns non-200 after a Renovate bump deploy.infra dashboard statusshows thedashboardcontainer unhealthy or restarting.infra dashboard logs dashboardreveals a regression introduced by the new release.
Procedure
Section titled “Procedure”Step 1: Identify the prior digest
Section titled “Step 1: Identify the prior digest”Find the commit that last changed the image pin in apps/dashboard/docker-compose.yaml:
git log --oneline -- apps/dashboard/docker-compose.yamlIdentify the commit immediately before the bad bump. Inspect it to confirm the prior digest:
git show <prior-commit>:apps/dashboard/docker-compose.yaml | grep 'image:.*dashboard'Note the full image reference, e.g.:
ghcr.io/fro-bot/dashboard:2026.06.14@sha256:abc123…
Step 2: Revert the compose pin commit
Section titled “Step 2: Revert the compose pin commit”Create a revert commit targeting the bad bump:
git revert <bad-bump-commit> --no-editVerify the compose file now contains the prior digest:
grep 'image:.*dashboard' apps/dashboard/docker-compose.yamlConfirm it matches the digest from Step 1.
Step 3: Push and open a PR
Section titled “Step 3: Push and open a PR”git push origin HEADOpen a PR against main. Title it clearly, e.g.:
revert: roll back dashboard image to <prior-tag> (regression in <bad-tag>)
Step 4: Merge and deploy
Section titled “Step 4: Merge and deploy”Merge the PR. The Deploy Dashboard workflow triggers automatically on merge to main.
Approve the dashboard environment gate in the GitHub Actions UI. The deploy pulls the reverted
digest and ships it to the droplet through the single locked SSH transaction. The transaction
validates the host, records baseline evidence, always prunes unused images, requires 6 GiB free after
prune, verifies the exact staged image set from cache before pulling when possible; otherwise it pulls
and verifies the complete set (or accepts a complete exact cache fallback after a failed pull),
requires 6 GiB free again, publishes active files with Compose last, verifies dashboard digest and
health, and only then unlocks. The remote transaction has a fixed 900-second deadline and the caller
watchdog has a 960-second deadline; timeout failures report their safe code and last stage. Advisory
probes and audit write-back happen after unlock.
Step 5: Verify
Section titled “Step 5: Verify”bunx @marcusrbrown/infra dashboard statusConfirm both dashboard and caddy services are healthy.
Probe the public endpoint:
curl -sf https://dashboard.fro.bot/api/healthzExpect HTTP 200. If Caddy ACME renewal is in progress, retry after 60 seconds.
Check logs for errors:
bunx @marcusrbrown/infra dashboard logs dashboard --tail 50Edge Cases
Section titled “Edge Cases”The bad release is the first known digest
Section titled “The bad release is the first known digest”If the bad release is the first time the dashboard was deployed (no prior digest in git history), there is no compose pin to revert to. Options:
-
Pin a specific prior tag manually. Browse
ghcr.io/fro-bot/dashboardreleases (GitHub Packages UI orgh api /orgs/fro-bot/packages/container/dashboard/versions) to find a known-good release. Update theimage:line inapps/dashboard/docker-compose.yamlwith the tag and digest of that release, commit, PR, merge, and deploy. -
Wait for a fixed upstream release. If the upstream
fro-bot/dashboardrepo is actively fixing the regression, wait for a new release and let Renovate open the bump PR normally.
The revert conflicts
Section titled “The revert conflicts”If the compose file has been modified since the bad bump (e.g., a config tweak landed on top of it),
git revert <bad-bump-commit> --no-edit will stop with a merge conflict. Two options:
Option A — staged revert, resolve manually:
git revert <bad-bump-commit> --no-commitThis stages the revert but leaves conflicts in the working tree. Open
apps/dashboard/docker-compose.yaml, resolve the conflict markers so the image: line contains the
prior digest (from Step 1), then:
git add apps/dashboard/docker-compose.yamlgit revert --continueOption B — direct edit (faster during an active incident):
If resolving the conflict is slower than the situation allows, edit the image: line directly:
# Set the image line to the known-good tag@digest identified in Step 1$EDITOR apps/dashboard/docker-compose.yamlCommit with a clear message:
git add apps/dashboard/docker-compose.yamlgit commit -m "fix: roll back dashboard image to <prior-tag> (regression in <bad-tag>)"Then continue from Step 3. This is acceptable when it is clearer or faster than resolving the revert conflict — the compose file is the contract; how the commit is authored is secondary.
A newer release exists and is known-good
Section titled “A newer release exists and is known-good”If a newer release has already shipped and is known-good, prefer bumping forward to that release rather than reverting backward. Update the compose pin to the newer digest, commit, PR, merge, and deploy. This avoids re-introducing the regression window.
Deployment stops before or during rollback
Section titled “Deployment stops before or during rollback”Resolve the reported condition before rerunning. Every failure reports a stable lowercase-hyphen code
and the last completed stage; remote stderr is not surfaced. The deterministic failure classes are
lock contention, prune failure, post-prune low headroom, acquisition/cache mismatch, post-acquisition
low headroom, unsafe path, payload malformed, transaction timeout, and active publication/runtime
failure. A prune failure always stops the deployment;
docker image prune -af may let Docker reclaim image data, but the deployment never directly deletes
containerd storage or files. It never prunes containers or volumes or uses Compose teardown for
recovery.
Stopped containers pin their images. If obsolete stopped containers keep free space below 6 GiB, inspect them and remove only the specifically obsolete containers manually, then rerun. There is no automatic override flag. After a successful rollback, the superseded image is left locally as one temporary rollback generation because pruning occurs only before acquisition; a later deploy attempt may prune it. The deploy does not perform automatic rollback if publication or runtime convergence fails; inspect the reported stage and actual state before retrying.
Related
Section titled “Related”apps/dashboard/AGENTS.md— deploy flow, secret rotation, container hardening, anti-patternsapps/dashboard/README.md— quick-start, secrets table, CLI referencedocs/solutions/workflow-issues/gateway-first-deploy-cascade-2026-05-20.md— SSH key trailing-newline and host-key handling background