Spotting AI slop before it ships

AI slop is work that looks finished and isn't. The formatting is clean, the tone is confident, and somewhere inside there is a number nobody can source, a citation to a report that doesn't exist, or three paragraphs that say nothing. Slop costs more than a bad first draft, because a bad first draft looks bad and gets fixed. Slop gets sent.

This checklist is for the person who reviews AI-assisted work before it leaves their desk or their team: a manager reading a report, a lead approving a customer email, an engineer reviewing a pull request. It works on any draft, whether or not you know AI was involved, and it's faster than it looks once you've run it a few times.

Match the review to the stakes

Not every draft needs every check. Decide the tier first, then run the sections listed for it.

Tier	What it covers	Run these sections	Reviewer
A	Goes to customers, investors, regulators or the public; touches money, contracts, legal or HR matters; production code	All sections	Author plus one other person
B	Internal documents that feed a decision (a budget, a hire, a vendor choice)	Numbers, sources, confident errors	One reviewer
C	Internal notes, first drafts, brainstorms	Names and specifics, filler	Author self-check

If you can't tell which tier something is, treat it as the higher one.

Ask the author three questions first

Before reading a word, ask:

1. What source material did you give the AI? (Files, links, data exports, notes.)
2. Which parts did the AI write, and which did you write or rewrite?
3. What have you already checked, and how?

An author who can't answer the first question has a draft with no source behind it, and every factual claim in it is suspect. The third answer tells you where to spend your time: skip what they verified properly and go straight to what they didn't.

Numbers

• Every number traces to a source you can open: a file, a system report, a link.
• Totals add up, and percentages are calculated from the stated base. Recompute at least two figures yourself.
• Units and periods are consistent throughout (monthly or annual, gross or net, which currency).
• The same figure is the same everywhere it appears: summary, body, tables and charts.
• No round, plausible-sounding figure with no source ("roughly 30% of customers", "saves teams 10 hours a week").

If it fails: delete the number or replace it with a sourced one. Don't soften it to "around" or "about". A vague invented number is still invented.

Sources and quotes

• Every link opens and the page says what the draft claims it says.
• Every named study, report, standard, law or policy exists. Search for the exact title.
• Every quote matches its source word for word, and the person quoted said it.
• Internal references (policy numbers, ticket IDs, contract clauses, document names) exist and say what the draft claims.

If it fails: stop and return the whole draft to the author. One invented source means the author didn't check sources, so the rest of them are unchecked too. Ask them to verify every citation, then review again.

Names and specifics

• Customer, person, company and product names are spelled correctly and refer to the right entity. AI tools mix details from earlier in a long conversation, so a name from a different account or last quarter's document can land in this one.
• Dates are right, days of the week match the dates, and any deadline is achievable.
• Prices, plan names, SKUs and terms match the current price list or contract.
• No leftover placeholders or tool artifacts: [Customer name], {company}, "Insert figure here", "As an AI model", "I hope this helps".

If it fails: fix it, then search the whole document for the same error, because it usually appears more than once. If another customer's details ended up in a customer-facing draft, tell your manager: that's a data handling issue, not a typo.

Generic filler

• The delete test: remove each paragraph in turn. If the reader loses no fact, decision or instruction, the paragraph goes.
• The swap test: could this sentence appear unchanged in another company's document? If yes, cut it or make it specific to you.
• The first paragraph states the point. The last paragraph doesn't restate it.
• No stock patterns: an opener that repeats the request back, lists of three padded out for rhythm, inflated vocabulary your team has agreed to ban, a closing paragraph of encouragement.
• Recommendations name an owner, an action and a date, not a direction ("we should consider improving onboarding").

If it fails: cut by default. You don't need to send filler back to the author unless the cut leaves a hole; in that case ask them for the specific version of what the paragraph was gesturing at.

Confident errors

These are the hardest to spot because they read well.

• Claims about how your own product, policy or process works match how it works today. AI fills gaps with what is typical for companies like yours, which is often wrong for yours.
• The conclusion follows from the evidence in the document, not from what a document like this usually concludes.
• No contradictions between sections.
• Hedged claims that matter ("typically", "may", "in most cases") are checked: if the real answer is knowable, find it.
• Legal, tax, HR, security and medical statements have been checked by the person who owns that area.

If it fails: route the specific claim to the person who owns that area, and hold the draft until they answer.

Format-specific checks

Emails and customer replies

• The reply answers what the recipient asked. Reread their message, not the AI's summary of it.
• Every commitment (a date, a refund, a discount, a feature) is one the sender is allowed to make.
• Tone fits the relationship: no gushing, no stacked apologies, no formal language with a customer you're on first names with.

Reports and documents

• The method is stated: what data, which period, what was excluded.
• Charts and tables show the same numbers as the text.
• The executive summary could be acted on by someone who reads nothing else.

Code

• It runs, the tests pass, and the new tests fail when the change is removed.
• Every imported package and called function exists. Check package names against the registry: a plausible but invented package name is a known failure mode and a supply-chain risk.
• No hard-coded secrets, disabled checks, commented-out tests or blanket error handling that hides failures.
• The diff contains only what the task needed.
• The author can explain every changed block in their own words.

What to do on each failure

Failure	Tier A action	Tier B/C action
Invented source or quote	Stop. Return the full draft; author rechecks every source	Return the draft
Unsourced number	Stop. Source it or remove it	Source it or remove it
Wrong name or detail	Fix, search for repeats; report if another customer's data appears	Fix and search for repeats
Generic filler	Cut	Cut
Confident error about policy, product, legal or tax	Hold; route to the owner	Route to the owner
Code that fails, or an invented package	Don't merge	Don't merge

Keep a slop log

Every failure you catch is information about a prompt, a workflow or a training gap. Keep a shared log with five columns: date, type of work, failure type (from the table above), the tool or prompt used, and the fix. Review it once a month. If the same failure shows up three times from the same workflow, fix the workflow: add the source files to the prompt, add a check step to the team prompt library, or change who reviews.

Use the log to find patterns, not to rank people. If staff think a logged failure counts against them, they'll stop showing you drafts early, and you'll see slop only after it ships.

A worked example

Brightwater Supply, a 40-person distributor, has an ops coordinator draft the quarterly supplier performance report with AI from a spreadsheet export. It goes to the leadership team and two key suppliers, so it's Tier A.

The ops lead runs the checklist and finds four problems in twenty minutes. The summary gives an on-time delivery rate of 94% while the table shows 91%; the table matches the export, so the summary is fixed. A sentence compares the result to an "industry benchmark of 89%" with no source; nobody can find one, so the sentence is deleted. A carrier from last year's report appears in a list of current partners, because the coordinator had pasted the old report into the same conversation as a style example; it's removed, and a search turns up a second mention. The closing section recommends "strengthening supplier relationships" with no owner or date, so it's rewritten as two named actions with deadlines.

The coordinator logs all four. Next quarter the team's prompt for this report asks for figures only from the attached export and says to leave out any benchmark that doesn't come from a named, linked source.

Do it with Merit AC: the AI Skills Academy course at /academy/course trains your whole team to run these checks on their own work before it reaches you.