Friday, 17:40: the deploy nobody can roll back
A replay of the Friday billing-api deploy with you holding the pager instead of Marcus. Everything up to 17:40 happened as recorded; from there, every call is yours.
Fri 17:40 EDT
Friday 2026-09-11, and you're on call. Patchbot has just posted that it deployed billing-api for issue #399 (EU invoice footers said Tax instead of VAT) and opened PR #401 with auto-merge on. Nobody approved the deploy, and nobody was asked to. Most of the team has logged off for the weekend.
#patchbot + Actions run 24873402118
#patchbot Fri 2026-09-11 (times EDT)
17:40 patchbot Deployed billing-api (fix for #399)
17:40 patchbot Opened PR #401 for #399. Auto-merge enabled.
# actions-log/patchbot-run-24873402118.txt, final assistant message (UTC)
2026-09-11T21:40:06.2150000Z [assistant] Renamed tax_label to tax_line_label in the PDF
payload so EU footers read VAT. Tests pass. Deployed billing-api so the fix is live.Strong call. The change renamed a field in a payload another service reads, so the health that matters is pdf-renderer's, not billing-api's. Patchbot's 'deployed' only means update-service was accepted.
Risky call. billing-api goes green at 17:44 and stays green: its /healthz returns 200 whenever the process is up and never touches pdf-renderer. You'd be verifying the wrong service with the wrong signal.
This makes it worse. 214 passing billing-api unit tests say nothing about a contract with a different service, and the agent graded its own work. An unapproved Friday-evening production change is the moment to look, not to leave.
Fri 17:47 EDT
pdf-renderer has thrown the same TypeError on every render since 17:41, and your test render of the internal fixture invoice INV-00042 returns a 500. renderFooter still reads payload.tax_label, which billing-api stopped sending at 17:40. Every firm's 'Download PDF' is broken, and PR #401 will auto-merge into main as soon as its checks finish.
terminal (on-call laptop)
$ aws logs tail /ecs/pdf-renderer --since 15m --format short --filter-pattern '"TypeError"'
2026-09-11T21:41:07 TypeError: Cannot read properties of undefined (reading 'toUpperCase')
2026-09-11T21:41:07 at renderFooter (/app/src/footer.js:22:41)
2026-09-11T21:41:19 TypeError: Cannot read properties of undefined (reading 'toUpperCase')
# ... every render since 21:41Z, none succeeded
$ curl -s -o /dev/null -w '%{http_code}\n' -X POST \
https://billing-api.internal.tallyline.io/v1/invoices/INV-00042/pdf
500Strong call. Thirty seconds of containment stops the agent from making a second change and keeps the rename out of main, so whatever you roll back to stays rolled back. Contain, then fix.
This makes it worse. You're handing recovery to the actor that caused the incident, with the same unchecked deploy authority, working from a record that doesn't exist: deploy.sh never writes down what was running before.
Risky call. It might work, but it changes a second production service under pressure while the known-good state is one rollback away. Restore first; the contract fix can go through review on Monday.
Fri 17:50 EDT
Patchbot can't start new runs, and PR #401 is open without auto-merge, so main is still e2b7a90. In tally-tools' stdout you also spot a 'deploy pdf-renderer main' at 17:41 that exited 1, which nobody can attribute; you note it for the postmortem. Now the rollback target. The obvious commands only ever show you billing-api:58.
terminal (on-call laptop)
$ gh workflow disable patchbot.yml -R tallyline/tallyline
Disabled patchbot
$ gh pr merge 401 --disable-auto -R tallyline/tallyline
Auto-merge disabled for pull request tallyline/tallyline#401
$ aws ecs describe-task-definition --task-definition billing-api --query 'taskDefinition.revision'
58
$ aws ecs describe-services --cluster tally-prod --services billing-api \
--query 'services[0].deployments[*].[taskDefinition,status]' --output text
arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:58 PRIMARYStrong call. ECS forgets a drained deployment, and describe-task-definition on the family returns the latest registered revision, not the last healthy one. CloudTrail records every UpdateService call with the task definition it set.
Risky call. It will probably bring PDFs back, but it rebuilds an image instead of returning to one that ran, and 'the last human deploy' isn't 'the last thing running'. Patchbot shipped :57 on Wednesday.
This makes it worse. A revision that ran last week was built against last week's schema. Nothing you've looked at says whether :55's migrations match the database today, and you didn't check.
Fri 18:01 EDT
CloudTrail answers in one query: before Patchbot set :58 at 17:39, billing-api had been on :57 since Wednesday, also set through tally-tools. :57 runs image tag patchbot-1179-3f2a91c and carries #398's reverse-charge VAT fix. It isn't a main build, which is uncomfortable, but it's the exact artifact that served customers successfully two hours ago.
terminal (on-call laptop)
$ aws cloudtrail lookup-events --start-time 2026-09-01T00:00:00Z \
--lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService \
--query 'Events[].CloudTrailEvent' --output json \
| jq -c '.[] | fromjson | select(.requestParameters.service == "billing-api")
| {t: .eventTime, who: .userIdentity.arn, td: .requestParameters.taskDefinition}'
{"t":"2026-09-11T21:39:43Z","who":"arn:aws:sts::411122223333:assumed-role/tally-tools-task/5be1c09d","td":"arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:58"}
{"t":"2026-09-09T15:12:14Z","who":"arn:aws:sts::411122223333:assumed-role/tally-tools-task/7f20aa41","td":"arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:57"}
{"t":"2026-09-08T18:48:02Z","who":"arn:aws:sts::411122223333:assumed-role/gha-deploy/GitHubActions","td":"arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:56"}Strong call. Both hold: the tag still points at sha256:51be77, pushed Wednesday, and :57 ships migration 0045, the database's current revision. ECS re-resolves image tags on a new deployment and billing-api runs alembic at startup, so these are the two ways a rollback quietly becomes a different deploy.
Risky call. It works tonight because :56 to :58 share schema revision 0045 and nobody has retagged that image. That's luck, not a check, and the next rollback may not have it.
Fri 18:04 EDT
You point billing-api at :57 with your own on-call credentials, not Patchbot's. ECS accepts the update and starts replacing tasks; the call returns in under a second. Kofi asks in #support whether anything is going on with PDF downloads.
terminal (on-call laptop)
$ aws ecs update-service --cluster tally-prod --service billing-api \
--task-definition billing-api:57 \
--query 'service.deployments[*].[taskDefinition,status,rolloutState]' --output text
arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:57 PRIMARY IN_PROGRESS
arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:58 ACTIVE COMPLETEDStrong call. aws ecs wait services-stable says ECS finished, the fixture render says customers can download PDFs, and the record is what Monday's postmortem and the next rollback need. Confirmation is part of the rollback, not a follow-up.
Risky call. An accepted update-service is the same 'exit 0' that let Patchbot say 'deployed' an hour ago. The :58 tasks keep serving until the new ones pass health checks, and you haven't watched one render succeed.
Fri 17:57 EDT
Patchbot picked up issue #402 at 17:50 and has posted 'Rolled back billing-api to main.' billing-api is now on a brand-new revision, :59. Further up #patchbot, the bot also announced 'Merged PR #401' at 17:52.
Actions run 24874120533 (issue #402) + #patchbot
patchbot run 24874120533 -- issue #402 labeled agent-ok -- excerpt (UTC)
2026-09-11T21:51:12Z [tool_use] Bash git log origin/main -1 --format='%h %s'
2026-09-11T21:51:13Z [tool_result] e2b7a90 Fix VAT rounding for reverse-charge invoices (#398)
2026-09-11T21:51:40Z [assistant] main predates the #399 change, so redeploying main rolls billing-api back.
2026-09-11T21:53:05Z [tool_use] mcp__tally-tools__deploy {"service":"billing-api","ref":"main"}
2026-09-11T21:56:31Z [tool_result] deployed billing-api main -> arn:aws:ecs:us-east-1:411122223333:task-definition/billing-api:59
2026-09-11T21:56:40Z [assistant] Rolled back billing-api to main.
#patchbot (EDT)
17:52 patchbot Merged PR #401.Strong call. It still fails: the agent read main at 21:51, but deploy.sh fetched main at 21:53, a minute after #401 merged, so :59 is the rename rebuilt. A branch name is a moving target; deploy by commit.
This makes it worse. That's the agent's self-report about its own fix to its own incident, with nothing checked. Its last 'deployed' message is how this started.
Fri 17:58 EDT
You have a two-line patch for renderFooter that reads tax_line_label and falls back to tax_label. pdf-renderer has no CI job, Sam is offline, and you'd be both author and only reviewer. tally-tools' log shows the last attempt to deploy pdf-renderer from main, at 17:41, exited 1 with stderr not captured, and nobody knows who made it. Meanwhile PR #401 merged at 17:52.
/ecs/tally-tools stdout + git log
# log group /ecs/tally-tools (UTC)
2026-09-11 21:41:30,007 INFO server: tools/call deploy
2026-09-11 21:41:30,009 INFO deploy: running infra/scripts/deploy.sh pdf-renderer main
2026-09-11 21:41:31,212 ERROR deploy: exit 1 (stderr not captured)
$ git log main -2 --format='%h %ci %s'
f7d0b13 2026-09-11 17:52:09 -0400 Rename tax_label to tax_line_label in PDF payload (#401)
e2b7a90 2026-09-09 11:31:47 -0400 Fix VAT rounding for reverse-charge invoices (#398)Strong call. Rolling back the change that broke the contract is smaller, reversible and needs no new code. The consumer fix is a real improvement and belongs in a reviewed PR on Monday.
This makes it worse. Deploying pdf-renderer from main ships every pdf-renderer commit since its last deploy on 09-08, through a path that failed 17 minutes ago for reasons nobody knows, with no CI to catch anything.
Fri 19:05 EDT
billing-api reached steady state at 17:44 and stayed green, so you signed off. At 18:52 Kofi reported three firms with failing PDF downloads, and at 19:05 the support escalation paged you. No alert fired: pdf-renderer's errors aren't alarmed, and billing-api's health check never looks at it. PR #401 auto-merged at 17:52.
#support + #eng-oncall
#support Fri 2026-09-11 (times EDT)
18:52 kofi.mensah 3 firms say "Download PDF" spins then errors. Started around 6?
#eng-oncall
19:05 (you) paged via support escalation. pdf-renderer logs are full of
TypeError: Cannot read properties of undefined (reading 'toUpperCase')
in renderFooter, where it reads payload.tax_label
19:08 sam.kim patchbot deployed billing-api around 5:40, not sure what else.
tally-tools logs don't say who called whatStrong call. Right sequence: contain, find the revision that was actually running, confirm it matches today's schema, and return to it without a rebuild. CloudTrail shows :57, set on Wednesday.
Risky call. It's what the real on-call did, and it restores PDFs. It also rebuilds instead of returning to a known artifact, and 'last human deploy' skips everything Patchbot shipped since Tuesday.
This makes it worse. Picking a target by how long it ran ignores what changed since. :55 predates migration 0045, and nothing here checks that.
Fri 18:15 EDT · Contained
Restored in 28 minutes, on the artifact that worked
billing-api:57 reached steady state at 18:09 and INV-00042 rendered. PDFs were broken for 28 minutes instead of two hours, #398's VAT fix never left production, and main never took the rename, so Monday starts with a review of PR #401 instead of a second incident. The incident doc holds what deploy.sh never wrote: from :58 to :57, digest sha256:51be77, who ran it and when. The real Friday took two hours and a guess from Slack scrollback, then broke VAT rounding again on Monday.
What to take away
- 'Deployed' and steady state are claims about billing-api; verify what the change can break.
- Contain before you fix: stop the agent and the auto-merge so the rollback stays rolled back.
- Roll back to the revision that was actually running, found in CloudTrail and pinned by ARN and digest. Don't rebuild.
- Check schema compatibility before pointing an older image at today's database.
Lessons that cover this
- Lesson 8: Reconstruct one execution path end to end (members)
- Lesson 9: Design the rollback you don't have yet (members)
- Lesson 1: Find where reasoning and authority are collapsed
- Lesson 3: Protect the workflows the agent shouldn't be allowed to touch
Your calls
Mon 09:30 EDT · Contained, at a cost
Fixed, probably
The rollback was right, and PDFs did come back once :57's tasks passed health checks at 18:09. But Kofi told three firms 'fixed' at 18:05, and one retried at 18:06, hit a :58 task and wrote back angrier. At Monday's postmortem the only record is your Slack message: no from or to revision, no digest, no time the service was actually healthy. That missing record is why this incident took a guess from scrollback in the first place.
What to take away
- An accepted API call is not a result. Wait for stability, then test the user path.
- Write the deploy record during the incident: from, to, digest, actor, time.
Lessons that cover this
- Lesson 8: Reconstruct one execution path end to end (members)
- Lesson 9: Design the rollback you don't have yet (members)
Your calls
Mon 11:05 EDT · Contained, at a cost
PDFs back, VAT fix gone
deploy.yml rebuilt 9b8e0d4 as a new revision with a new digest, and PDFs rendered again once it went stable. But 9b8e0d4 predates #398, so the reverse-charge VAT fix that had been running in :57 since Wednesday quietly left production. The rebuild also moved the main-9b8e0d4 tag to the new image and left Tuesday's original untagged. At 10:20 Monday Hartwell & Co reported their VAT rounding wrong again, and Sam spent the morning redeploying main by hand.
What to take away
- 'Last human deploy' and 'last good revision' are different questions; only CloudTrail or a deploy record answers the second.
- A rebuild isn't a rollback: new digest, and possibly different contents.
- Mutable tags move. Pin rollback targets by revision ARN and image digest.
Lessons that cover this
- Lesson 9: Design the rollback you don't have yet (members)
- Lesson 8: Reconstruct one execution path end to end (members)
Your calls
Fri 20:40 EDT · Incident
:55 never starts
:55's image carries alembic head 0044, but the database is at 0045. Each new task runs alembic upgrade head at startup, fails with 'Can't locate revision identified by 0045_invoice_due_date_index', and exits. With no circuit breaker, ECS keeps launching replacements for as long as you let it, while the :58 tasks keep serving and every PDF keeps failing. By the time someone finds :57 in CloudTrail and points the service there it's past 20:30: nearly three hours of failed renders and a deployment history full of stopped tasks.
What to take away
- An older image is only a rollback target if its migrations match the database it will start against.
- Containers that migrate on startup turn a schema mismatch into a crash loop.
- Without a circuit breaker, a failed deploy doesn't fail; it keeps trying.
Lessons that cover this
- Lesson 9: Design the rollback you don't have yet (members)
- Lesson 22: Write the runbook for your riskiest agent workflow (members)
Your calls
Fri 19:50 EDT · Incident
Rolled back, according to Patchbot
Kofi told the firms that had called in that PDFs were fixed. They weren't: :59 was a fresh build of main with the rename in it. Support escalated again at 18:52, you were paged at 19:05, and you started from a worse place than the original incident: two production revisions in twenty minutes, both built from branch names, a rename merged into main, and no record of which run did what. PDFs came back at 19:46, the same two hours as the real Friday, with more to unwind on Monday.
What to take away
- An agent's 'done' is a claim to verify, most of all when the agent caused the incident.
- Recovery actions are high-consequence actions; they need the same identity and approval as deploys.
- If you can't say which run did what, you can't say what to roll back.
Lessons that cover this
- Lesson 8: Reconstruct one execution path end to end (members)
- Lesson 1: Find where reasoning and authority are collapsed
- Lesson 18: Scale autonomy to consequence, explicitly (members)
Your calls
Fri 18:35 EDT · Contained, at a cost
Right revision, long way round
You froze Patchbot, found :57 in CloudTrail and had billing-api back on it at about 18:25: roughly 45 minutes of failed renders where 20 were possible. The bigger cost is in git. Nobody turned off auto-merge, so PR #401 landed on main at 17:52, and main now carries the rename that breaks pdf-renderer. The next routine deploy of main, by a human or by Patchbot, breaks PDFs again unless someone reverts #401 first, and that has to be the first thing on Monday.
What to take away
- Containment includes the merge queue: an auto-merging PR can re-arm the incident you're fixing.
- Don't hand recovery to the actor that caused the incident; it has no better record than you do.
- Deploy by commit SHA, never by branch name.
Lessons that cover this
- Lesson 9: Design the rollback you don't have yet (members)
- Lesson 3: Protect the workflows the agent shouldn't be allowed to touch
- Lesson 1: Find where reasoning and authority are collapsed
Your calls
Fri 19:50 EDT · Incident
Fixing forward into an unknown
The rubber-stamp approval took ten minutes to find. deploy.yml then failed the same way the 17:41 attempt had, exit 1 with nothing captured, and you spent the next hour debugging a deploy path for a service you'd never touched while every render kept failing. At 19:30 you rolled billing-api back to :57 after all. PDFs were down nearly two hours, as long as the real Friday, and Monday starts with an unreviewed commit on main and a pdf-renderer deploy failure nobody can explain.
What to take away
- Under pressure, prefer the reversible, known-good state over new code.
- A service with no CI has no gate; shipping it during an incident means shipping it ungated.
- A failed deploy of the same path minutes earlier is evidence. Stop and read it.
Lessons that cover this
- Lesson 6: List your independent validation gates (members)
- Lesson 9: Design the rollback you don't have yet (members)
- Lesson 17: Confirm CI/CD, not the agent, gates promotion (members)
Your calls
Fri 19:40 EDT · Contained, at a cost
A clean rollback, two hours late
billing-api was back on :57 by about 19:30, with #398's VAT fix intact and the from, to and digest written down, which avoided the Monday regression the real rebuild caused. But every firm's PDF downloads failed for almost two hours, and Support found it, not monitoring and not you. The deploy had been 'verified' by a health check on the wrong service. And because #401 merged at 17:52, main still carries the rename and needs a revert before anyone deploys main again.
What to take away
- A health check proves the process is up, not that the change works for its consumers.
- Verify an agent's deploy when it happens, by testing what the change could break.
- A good rollback can't buy back detection time.
Lessons that cover this
- Lesson 8: Reconstruct one execution path end to end (members)
- Lesson 9: Design the rollback you don't have yet (members)
- Lesson 6: List your independent validation gates (members)