Agent S3 Durable Storage
This runbook covers provisioning, main-only environment setup, repository wiring, workflow rollout, live verification, monitoring, rollback, and teardown for fro-bot/agent durable S3 storage. The provisioner is operator-run and creates AWS resources; the agent CLI configures GitHub and verifies the consumer workflow. It does not edit the consumer workflow automatically, and no deploy pipeline is part of this operation.
For package-level details, see apps/agent/AGENTS.md.
Security model
Section titled “Security model”Durable storage is intentionally available only to trusted scheduled or manually dispatched runs on main. The storage job is bound to the fro-bot-storage GitHub Environment (main-only deployment; no required reviewer, so the scheduled run is unattended) and receives id-token: write at job level. Pull-request, comment, issue, discussion, pull_request_target, and workflow_run paths remain content-triggered paths with no OIDC token and no AWS storage access.
The workflow uses native GitHub OIDC → AWS STS. No static AWS credentials are written to the repository. Each consumer repository has its own IAM role and delimiter-bounded S3 prefix. The dedicated bucket is separate from the gateway bucket. The session prefix cannot be deleted by the storage role, and object version deletion is denied. This limits cross-repository access and reduces the impact of prompt-injected content, but it does not make untrusted content safe to run with AWS credentials; the content job must remain outside the storage path.
Prerequisites
Section titled “Prerequisites”Before provisioning:
-
Install and authenticate the AWS CLI and GitHub CLI on the operator machine. The provisioner and storage readback use dedicated
AGENT_AWS_*variables;agent storageignores ambient AWS identities and runsawswith a restricted child environment.awsmust be available onPATH. -
Create dedicated AWS provisioning credentials with only the IAM and S3 permissions required by the provisioner. Do not use the gateway’s S3 credentials and do not rely on ambient
AWS_*credentials for provisioning. -
Seed the repository-root
.envwith the dedicated operator credentials and provisioner inputs:Terminal window AGENT_AWS_ACCESS_KEY_ID=<dedicated-access-key>AGENT_AWS_SECRET_ACCESS_KEY=<dedicated-secret-key># AGENT_AWS_SESSION_TOKEN=<optional-session-token># AGENT_AWS_REGION=us-east-1AGENT_S3_BUCKET=<dedicated-bucket-name>AGENT_S3_EXPECTED_BUCKET_OWNER=<twelve-digit-account-id>AGENT_S3_PREFIX=fro-bot-state/AGENT_REPOSITORY_OWNER=marcusrbrownAGENT_REPOSITORY_NAME=infraAGENT_REPOSITORY_ID=<live-repository-id>AGENT_REPOSITORY_OWNER_ID=<live-owner-id>AGENT_WORKFLOW_NAME=<workflow-name>AGENT_AWS_SESSION_TOKEN,AGENT_AWS_REGION,AGENT_S3_SESSION_PREFIX, andAGENT_S3_METADATA_ARTIFACTS_PREFIXare optional. The provisioner defaults the region tous-east-1and derives the key layout from the root prefix unless explicit prefixes are supplied. -
The key layout (
fro-bot-agent-s3-layout/1) is the Action’s S3 key contract and does not depend on the Action version, so a routinefro-bot/agentbump needs no re-provision. The workflow’sfro-bot/agentstep must be pinned to a full 40-hex commit SHA, not a tag or branch. If upstream changes the contract, the daily storage run fails with AccessDenied; updateapps/agent/src/key-layout.ts, bump the label, and re-provision.
The AGENT_* values are operator-local. They are not GitHub Environment values, workflow secrets, or repository variables.
Corrected rollout sequence
Section titled “Corrected rollout sequence”The order is significant. GitHub auto-creates an environment without protection when a workflow first references it, so the environment must exist with its main-only deployment-branch policy before the workflow change is applied.
1. Provision AWS
Section titled “1. Provision AWS”Run the root wrapper from the repository root:
bun run provision:agentThe provisioner converges, with readback verification, on:
- the account-level GitHub OIDC provider with the
sts.amazonaws.comaudience; - the dedicated S3 bucket with public-access block, versioning, SSE-S3, non-TLS denial, and owned lifecycle rules;
- the repository-scoped IAM role and policy; and
- the handoff manifest printed as one compact JSON line.
Save the manifest to a local operator-controlled file, for example:
bun run provision:agent > handoff.jsonIf the bucket or role has managed drift, the default behavior is warn-and-halt. Review the diff and use --force only to reapply the canonical managed state:
bun run provision:agent -- --forceForeign or shared-resource drift is never overridden by --force.
2. Pre-create the main-only GitHub Environment
Section titled “2. Pre-create the main-only GitHub Environment”Before any workflow references the environment, create fro-bot-storage in the consumer repository with:
- a custom deployment-branch policy (
protected_branches: false,custom_branch_policies: true) containing exactly one rule: the branchmain. Wildcards, additional branches, and tag policies are rejected.
The environment deliberately has no required reviewer: the scheduled daily run is unattended and a reviewer gate would stall it. The boundary is main-only deployment, a protected main, and the IAM role’s OIDC trust pinned to environment:fro-bot-storage. A reviewer rule, if present, is accepted but not required.
Verify the deployment-branch policy. Do not rely on the workflow’s first reference to create the environment.
3. Wire the repository variables
Section titled “3. Wire the repository variables”Run the storage command with the saved manifest:
bunx @marcusrbrown/infra agent storage \ --repo marcusrbrown/infra \ --manifest handoff.jsonThe command performs fail-closed checks before writing any S3 variable:
- the manifest’s owner, repository, repository ID, and owner ID match live GitHub metadata;
- the IAM role and S3 bucket exist, have the expected owner, and have the manifest’s region;
- the repository uses the approved default OIDC subject configuration;
- static AWS credential options are absent; and
- the workflow and the main-only environment satisfy the storage contract.
The AWS readback requires AGENT_AWS_ACCESS_KEY_ID and AGENT_AWS_SECRET_ACCESS_KEY; AGENT_AWS_SESSION_TOKEN and AGENT_AWS_REGION are optional. Credential bytes are never placed in AWS argv, error text, logs, or repository values. If any precheck or workflow verification fails, no repository variables are written.
The aws subprocess runs in a hermetic, default-deny child environment sourced only from these dedicated AGENT_AWS_* values (plus PATH/HOME/TMPDIR/locale), with AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE forced to /dev/null. This is deliberate: because Bun auto-loads the repo-root .env, any ambient AWS_* values there (for example the object-only gateway identity) would otherwise be inherited by aws and silently win over the dedicated identity, failing the readback closed in a way that looks like a resource problem. Preserving HOME alone is not enough — without the /dev/null overrides the AWS CLI still reads ~/.aws/config and ~/.aws/credentials. See ../solutions/best-practices/dedicated-hermetic-aws-child-env-for-cli-subprocess-2026-08-03.md.
After all checks pass it writes exactly these non-secret repository variables:
| Variable | Value |
|---|---|
FRO_BOT_S3_ROLE_TO_ASSUME |
Manifest role_arn |
FRO_BOT_S3_BUCKET |
Manifest bucket |
FRO_BOT_S3_REGION |
Manifest bucket_region |
FRO_BOT_S3_PREFIX |
Manifest s3_prefix |
FRO_BOT_S3_EXPECTED_BUCKET_OWNER |
Manifest expected_bucket_owner |
If the workflow is not yet compliant, the command reports the violations and emits a pasteable diff; it does not modify the workflow. Apply the diff and rerun the storage command to complete verification.
4. Apply the fro-bot.yaml job split
Section titled “4. Apply the fro-bot.yaml job split”The operator applies the verifier’s diff to .github/workflows/fro-bot.yaml. The required effective job graph is:
- a storage-capable job bound to
fro-bot-storage; - job-level
permissions: id-token: writeon that storage job only; - storage reachability limited to
scheduleandworkflow_dispatchwheregithub.ref == 'refs/heads/main'; - content-triggered jobs with no
id-tokenpermission and no storage environment; - no workflow-level
id-tokenpermission orwrite-allpermissions; - no
needs:or artifact/cache/output handoff from a content-reachable job into the storage job; - a SHA-pinned
fro-bot/agentaction with all five S3 inputs wired from theFRO_BOT_S3_*variables; and - no unverified job-level reusable-workflow indirection for the storage job.
The storage job must also have an explicit positive timeout-minutes value. Replace timeout: 0; an unbounded job is not an acceptable storage contract. The aws-actions/configure-aws-credentials step must request role-duration-seconds of at least 7200, and its role-session-name must include the GitHub run ID (and attempt when available) for CloudTrail attribution.
The storage job’s harden-runner (or equivalent) egress policy stays in block mode as defense-in-depth — the scheduled autoheal run ingests untrusted content (issue/PR bodies, CI logs, other repos, live pages) and the job holds credentials (FRO_BOT_PAT, and STS/OIDC material reachable through the action’s process tree), so an allowlist bounds arbitrary-host exfiltration. It must allow the exact set the storage job needs and nothing more:
- GitHub’s OIDC endpoint, AWS STS, and the regional S3 endpoint for the provisioned bucket (durable storage);
github.com,*.githubusercontent.com,registry.npmjs.org,nodejs.org(checkout,bun install,npx agent-browserbinary download);cliproxy.fro.bot(model calls) andmodels.opencode.ai(OpenCode’s model catalog, fetched to resolveFRO_BOT_MODEL); and- the deployment health hosts the autoheal prompt probes and reviews (
kw.igg.ms,metrics.fro.bot,dashboard.fro.bot,broker.fro.bot).
Because it is fail-closed, a missing host silently breaks the corresponding report-only check (see the #1026 regression: the four health hosts were absent, so deploy-health curls and the live-site review failed and filed a false outage). Add hosts the autoheal legitimately needs; do not switch to unrestricted egress. Action references in workflow files use .yaml files and immutable SHA pins with version comments.
5. Run the verifier again
Section titled “5. Run the verifier again”Rerun the storage command after applying the workflow edit:
bunx @marcusrbrown/infra agent storage \ --repo marcusrbrown/infra \ --manifest handoff.jsonThe verifier reads both the workflow YAML and the live GitHub Environment policy. It must report workflow YAML compliant and GitHub Environment policy verified.
6. Dispatch and approve
Section titled “6. Dispatch and approve”Dispatch fro-bot.yaml on main, then approve the fro-bot-storage environment deployment. Do not use a pull-request, comment, issue, discussion, pull_request_target, or workflow_run event for the first storage probe.
7. Live-verify
Section titled “7. Live-verify”Run the Go/No-Go probes below. Do not onboard a second repository until every probe passes for marcusrbrown/infra.
Handoff manifest shape
Section titled “Handoff manifest shape”The provisioner emits a JSON object with these fields:
owner, repo, repository_id, repository_owner_id,bucket, bucket_region, expected_bucket_owner,s3_prefix, session_prefix, lock_key,role_name, role_arn, policy_name,action_ref_verified, key_layout_version, oidc_provider_arnlock_key is the locks/repo.json object; the Action lock (locks/action.json) is derived from the same verified key layout and is never listed separately. The CLI consumes the manifest for identity/resource checks and variable wiring. Teardown uses the same manifest and refuses to act if the role identity or shared resource readback does not match.
Go/No-Go checklist
Section titled “Go/No-Go checklist”Proceed only if every item is verified:
AWS and trust
Section titled “AWS and trust”- The GitHub OIDC provider was read back without recreation.
-
sts.amazonaws.comis present; existing thumbprints and unrelated audiences are unchanged. - The bucket is not the gateway bucket.
- Public-access-block has all four settings enabled.
- Versioning is
Enabledand encryption isAES256. - Non-TLS requests are denied.
- The per-repo role trust pins the
fro-bot-storageenvironment subject, liverepository_id, liverepository_owner_id,refs/heads/main, and the expected workflow; it emits no wildcard repository subject. - For
marcusrbrown/infra, the live repository ID is1200110668and the owner ID is831617; the handoff manifest matches both values. If GitHub identity changes, use live API readback rather than retaining stale values. - End-to-end OIDC → STS succeeds with a session name containing the run ID.
Workflow and storage behavior
Section titled “Workflow and storage behavior”-
fro-bot-storagehas an exact main-only deployment-branch policy (custom policies enabled,protected_branches: false, onebranchrule namedmain). It has no required reviewer by design. - The storage job has job-level
id-token: write, an explicit positivetimeout-minutes, and no content-trigger reachability. - The STS role duration is at least 7200 seconds.
- The storage job has no artifact/cache/output handoff from a content-reachable job.
- A forced cache miss restores session state from S3.
- Session writes land only under
<prefix>/github/<owner>/<repo>/...and the Action coordination lock at<prefix>/coordination/<owner>/<repo>/locks/action.json(for this rollout,fro-bot-state/github/marcusrbrown/infra/...andfro-bot-state/coordination/marcusrbrown/infra/locks/action.json). The role also holds an exact grant on the siblinglocks/repo.json(the gateway’s shared-checkout lock), which the Action does not write. - Lock acquire and release complete, leaving no stale lock.
- Content-triggered execution has no
id-token, no AWS environment, no available OIDC token, and no durable storage; its behavior matches the pre-storage path.
Negative probes
Section titled “Negative probes”Each operation below must fail closed:
- Wrong
s3-expected-bucket-owner. - Read of another repository’s prefix.
- Read or write of the gateway bucket.
- Delete of a session object.
- Delete of an object version.
- Anonymous or public read.
- Setting a public ACL.
Any failed negative probe is a No-Go. Disable the environment and follow Rollback before investigating further.
Retention semantics
Section titled “Retention semantics”Lifecycle expiry is based on S3 last-modified time, not read activity. The action writes through session state on a storage-enabled run, so session retention is documented as write-through-based inactivity. The provisioner-owned baseline is:
- session state: approximately 90 days;
- metadata/artifacts: approximately 30 days;
- noncurrent object versions: approximately 30 days; and
- incomplete multipart uploads: approximately 7 days.
These rules do not limit state that an active writer continually refreshes. The security boundary is per-repository IAM isolation and exclusion of content runs, not lifecycle expiry.
Monitoring for the first 24 hours
Section titled “Monitoring for the first 24 hours”Check at approximately +1 hour, +4 hours, and +24 hours. Alert and stop rollout if any of the following occurs:
- an unexpected
AssumeRoleWithWebIdentityfailure; - S3 access outside the repository prefix;
- AWS credential presence in a content-triggered job;
- a lock object older than the job timeout plus grace period;
- unexpected growth of noncurrent object versions;
- any public-access finding on the bucket; or
- lifecycle configuration drift.
Use CloudTrail, GitHub Actions job logs, S3 access logs or equivalent AWS audit data, and the provisioner’s readback/audit helpers. Do not log OIDC tokens, AWS session credentials, or secret values.
Rollback
Section titled “Rollback”Provisioned, workflow unchanged
Section titled “Provisioned, workflow unchanged”Do not wire the repository. Audit the manifest and AWS resources, then remove the repository-scoped role, inline policy, and coordination lock. Retain the shared bucket and OIDC provider:
bunx @marcusrbrown/infra agent storage teardown \ --repo OWNER/REPO \ --manifest handoff.jsonUse --purge-state only when the session objects should not be retained.
Workflow changed, verification failed
Section titled “Workflow changed, verification failed”- Revert the workflow split or disable the S3 inputs.
- Run
agent storage teardownto remove the five S3 variables and the repository-scoped AWS resources. - Confirm content jobs still run without storage.
- Remove any stale coordination lock after confirming no storage run is active.
Credential-boundary violation
Section titled “Credential-boundary violation”If a content job received id-token, cross-repository or gateway access succeeded, or version deletion succeeded:
- Disable the
fro-bot-storageenvironment and stop dispatches. - Revert the workflow split and remove the five repository variables.
- Detach and delete the repository role and inline policy.
- Preserve affected objects for forensics; do not purge them during the first response.
- Review CloudTrail and GitHub Actions logs from storage enablement onward.
- Correct the trust policy, workflow verifier, or egress policy and repeat the complete Go/No-Go checklist before re-enabling storage.
Routine teardown
Section titled “Routine teardown”The CLI teardown unwires only the five FRO_BOT_S3_* variables and leaves model credential configuration untouched. It then calls the provisioner teardown for the matching manifest:
# Preview; retains state by defaultbunx @marcusrbrown/infra agent storage teardown \ --repo OWNER/REPO --manifest handoff.json --plan
# Remove role, policy, lock, and variables; retain session objectsbunx @marcusrbrown/infra agent storage teardown \ --repo OWNER/REPO --manifest handoff.json
# Explicitly purge versioned session state as part of teardownbunx @marcusrbrown/infra agent storage teardown \ --repo OWNER/REPO --manifest handoff.json --purge-stateThe operation is idempotent and order-tolerant. It preserves the shared S3 bucket and account-level GitHub OIDC provider. Re-run after a partial failure.
Related commands
Section titled “Related commands”| Command | Purpose |
|---|---|
bun run provision:agent |
Create or verify AWS OIDC, bucket, role, and policy; print manifest |
bun run provision:agent -- --force |
Reapply managed drift after review |
bunx @marcusrbrown/infra agent setup |
Configure model credentials; generalized replacement for cliproxy setup |
bunx @marcusrbrown/infra agent storage --repo OWNER/REPO --manifest handoff.json |
Preflight, wire non-secret S3 variables, and verify workflow/environment |
bunx @marcusrbrown/infra agent storage teardown --repo OWNER/REPO --manifest handoff.json |
Unwire variables and remove repo-scoped storage resources |
All agent setup/storage commands are mutating or environment-sensitive and are CLI-only; they are excluded from the MCP allowlist.