Find where reasoning and authority are collapsed
Produce a ranked authority map of every place Patchbot can act with no identity, policy, or CI check between its decision and the action.
Today at Tallyline
It's your first morning at Tallyline. Priya wants Patchbot rolled out to every team this quarter; Dana wants it switched off. Before anyone argues, you need one artifact both of them will accept: a map of everywhere Patchbot's reasoning and its authority are the same thing. Sam has sent you the three files that define how Patchbot runs today.
The prompt
Paste this into a new chat in Claude or ChatGPT, then paste the starter files below it.
Act as a security architect brought in after a routine pull request from an AI coding agent turned out to have quietly touched deployment permissions nobody remembered granting it -- now leadership wants a full map of how much autonomous authority the agent holds everywhere else, before it happens again.
Context: I'm going to paste in a directory listing, a CI/CD config, and any IAM or deployment policy files from my repository. In a governed setup, a model may recommend a tool call, but identity, policy, network, and deployment systems decide whether it actually executes -- reasoning and authority are supposed to be separate layers, not collapsed into one.
Do the following, in order:
1. List every place in what I paste where an agent could execute an action directly, with no separate identity check, policy check, or CI gate sitting between the model's decision and the action.
2. For each one, name the specific service, IAM principal, log source, and deployment gate that should sit in between, even if it doesn't exist yet.
3. Rank the list by blast radius if that action were taken incorrectly -- what's the worst plausible outcome.
Constraints:
- Don't recommend removing agent capability wholesale; the goal is inserting a control point, not blocking use of the agent.
- Cite the actual file, script, or config line each gap lives in, not a general category.
- If nothing I paste shows an existing control, say so plainly instead of inferring one that might exist elsewhere.
Output format: a table with columns Location, Current control (if any), Missing control, Owner -- followed by one paragraph naming the single highest-blast-radius gap.Starter files
.github/workflows/patchbot.yml
name: patchbot
on:
issues:
types: [labeled]
permissions: write-all
jobs:
run:
if: github.event.label.name == 'agent-ok'
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} # IAM user patchbot-ci
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: us-east-1
TALLY_TOOLS_URL: https://tally-tools.internal.tallyline.io/mcp
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.PATCHBOT_PAT }} # Sam's personal token, repo + workflow scopes
- name: Run Patchbot
run: |
npx @anthropic-ai/claude-code -p "$(cat .patchbot/task-prompt.md) Issue: ${{ github.event.issue.title }}" \
--dangerously-skip-permissions
- name: Ship it
run: |
gh pr create --fill --head patchbot/${{ github.event.issue.number }}
gh pr merge --auto --squash
env:
GH_TOKEN: ${{ secrets.PATCHBOT_PAT }}infra/terraform/iam_patchbot.tf
resource "aws_iam_user" "patchbot_ci" {
name = "patchbot-ci"
}
resource "aws_iam_access_key" "patchbot_ci" {
user = aws_iam_user.patchbot_ci.name
}
# TEMPORARY until we scope this down -- sk, 2026-01-14
resource "aws_iam_user_policy_attachment" "patchbot_admin" {
user = aws_iam_user.patchbot_ci.name
policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}tools/tally_tools/server.py (excerpt)
@mcp.tool()
def run_sql(query: str) -> list[dict]:
"""Run any SQL against the production database."""
with psycopg.connect(os.environ["PROD_DSN_APP_ADMIN"]) as conn:
return conn.execute(query).fetchall()
@mcp.tool()
def deploy(service: str, ref: str) -> str:
"""Deploy a service to production at the given git ref."""
return subprocess.check_output(["infra/scripts/deploy.sh", service, ref]).decode()
@mcp.tool()
def rotate_secret(name: str) -> str:
"""Rotate a secret in AWS Secrets Manager."""
return boto3.client("secretsmanager").rotate_secret(SecretId=name)["ARN"]The lab
- Open a fresh chat in Claude or ChatGPT. Paste the Day 1 prompt exactly as written, then paste all three starter files below it, each under its file name.
- Read the table it returns before you read the summary paragraph. For every row, find the exact line in the starter files it cites. Strike any row that cites a line you can't find -- an authority map with invented evidence is worse than none.
- Check the blast-radius ranking against Tallyline's real risks: production Postgres with customer payout details, the prod ECS cluster, and the ability to change its own permissions. Reorder anything ranked by how alarming it sounds rather than what it can actually reach.
- Copy the cleaned table into a doc titled 'Patchbot authority map -- day 1'. Add an Owner column value for every row using Tallyline's people: Marcus for AWS and deploys, Dana for secrets and SOC 2 evidence, Sam for the workflow and tally-tools.
- Write one sentence at the top naming the single highest-blast-radius gap. That sentence is what you'll bring to Priya and Dana.
You're done when
- Every row in your map cites a real line from a starter file.
- run_sql (or the AdministratorAccess key that makes everything else possible) is ranked first, with a reason tied to customer data.
- Every row has a named owner from Tallyline, not 'the team'.
- No row says 'remove Patchbot' -- every fix is a control point inserted, not capability deleted.
Answer key -- open it after you've done the lab
A strong answer finds these in the starter files:
- --dangerously-skip-permissions in patchbot.yml: the model's decision to run any command is the execution, with no tool-level permission check at all.
- gh pr merge --auto with Sam's PAT, which is on the branch-protection bypass list: Patchbot can approve its own work into main. No independent gate sits between generation and merge.
- AdministratorAccess on a long-lived key for patchbot-ci: anything the agent decides to do in AWS, including granting itself more, is allowed. No policy layer exists between reasoning and the cloud account.
- run_sql connects as app_admin to production read-write with arbitrary SQL: this is the highest blast radius -- customer payout data and schema changes, no approval, no migration record (the ALTER TABLE near-miss already proves it).
- deploy calls deploy.sh directly, bypassing deploy.yml and CI entirely; rotate_secret can lock humans out of production secrets. Neither has a policy check or a log tying the call back to an issue.
- permissions: write-all on the workflow token means even without the PAT, the job could rewrite workflows and releases.
Your turn
Run the same prompt against your own repository: your agent's workflow file, its IAM or service-account policy, and its tool definitions. If you can't find one of those three, that missing file is your first row.
Stretch
For the top-ranked gap, sketch the control in one paragraph: which existing service (an IAM permission boundary, a policy engine, a CI-required check) becomes the thing that says yes or no, and what log line proves it did.
Why it's built this way
The handbook's architecture rule: a model may recommend a tool call, but the tool contract, identity, policy engine, network path, and target system determine whether it executes. Mapping the control to a named service and log source is what makes the architecture reviewable and testable — rather than dependent on prompt wording.
Video transcript
Day one. Before you can make an AI agent safe, you have to find every place where the agent thinking about an action and the agent being allowed to take it are the same step. Today we map that, at a company called Tallyline.
Here's the architecture rule the whole course is built on.
A model is allowed to recommend an action. That's its job.
But identity, policy, the pipeline, and the target system decide whether it actually happens.
When those two collapse into one step, whatever the model decides simply executes. That's what we're hunting for.
This is the workflow that runs Patchbot. Three lines matter.
Skip permissions means every command the model chooses runs, with no tool level check.
It checks out with a personal access token that's on the branch protection bypass list.
So when it auto merges its own pull request, nothing independent stands between writing the change and shipping it.
Now the tools.
Run SQL takes any query and runs it as the admin user, read and write, against production. That's where customer payout details live.
And deploy calls the deploy script directly. It never goes through the pipeline at all.
Here's what you'll do in the lab.
Paste the day one prompt and the three starter files into Claude or ChatGPT.
Check every row of the table against an actual line in the files, and strike anything it invented.
Rank by what each gap can actually reach, not by how scary it sounds.
Then give every row a real owner at Tallyline.
You're done when four things are true.
Every row cites a real line.
Run SQL, or the admin key that makes everything else possible, ranks first.
Every row has a named owner.
And every fix adds a control point instead of taking the agent away. Tomorrow, we write the instructions file Patchbot should have had from day one.
Membership · $49/mo
Days 1–3 are free. Membership unlocks all 30 lessons -- every video, lab, starter file and answer key -- plus live workshops as they're scheduled.
Join — $49/mo