Guide Independent of the product

AI system design patterns

A field guide to how AI systems are designed: twelve archetypes, six complex agent patterns that combine them, and the categories of software they're built from. It doesn't assume you use Merit AC.

1. AI system archetypes: diagrams, when to use, prompts to try

Most AI systems in production combine the twelve patterns below. Each one has a description, a diagram of how it's wired, when to use it, when to skip it, and a prompt you can try. Voice and other multimodal front ends aren't listed separately because they're an input/output layer on top of these shapes.

Chatbot / single-turn assistantLinear pipeline Retrieval-augmented generation (RAG)Linear pipeline Coding assistant / copilotLinear pipeline, human-gated Agentic loop (tool-use agent)Autonomous loop Multi-agent / orchestratorHub, spokes, and merge Router / classifier dispatchBranching dispatch Ensemble / self-consistencyFan-out, fan-in Evaluator / LLM-as-judgeGenerate-and-judge loop Memory-augmented agentLoop with persistent memory Fine-tuned / specialized modelLinear pipeline, offline Workflow automation (LLM-in-pipeline)Linear pipeline, fixed by code Computer-use / browser agentAutonomous loop

Chatbot / single-turn assistant

Linear pipeline

User message

LLM+ prior turns, if any

Response

The base case everything else on this page builds on: one message in, one response out. The model doesn't "remember" anything between requests; conversation history is replayed into each new call. A long-running chat is N independent calls, each carrying the whole transcript so far.

Use it when

Quick Q&A, drafting, or brainstorming that doesn't depend on private or fast-changing information.
The full context the model needs fits comfortably in one prompt.

Skip it when

The answer depends on facts the model wasn't trained on, or that changed since training.
The task requires taking an action outside the conversation, beyond producing text.

Try this prompt

Explain the tradeoffs between two build tools for a small Node.js project, then recommend one and say what would change your recommendation.

Retrieval-augmented generation (RAG)

Linear pipeline

Query

Retrievevector / keyword search

Augment prompt

LLM

Grounded response

Used when the model needs facts it wasn't trained on: a company's own documents, a support-ticket history, anything that changes after the model's training cutoff. The retrieval step searches a knowledge base for passages relevant to the query and stuffs them into the prompt before the model ever sees the question, so the answer is grounded in current source text instead of the model's memorized (and possibly stale or wrong) recall.

Use it when

Answers must be grounded in a specific, private, or frequently-changing corpus.
The corpus is too large to paste into a prompt directly.

Skip it when

The knowledge base is small enough to include in the prompt outright. Retrieval adds a failure point for no benefit.
The question is general knowledge the model already answers reliably.

Try this prompt

Using only the attached policy documents, answer: what is our refund window for enterprise customers, and quote the exact clause it comes from.

Coding assistant / copilot

Linear pipeline, human-gated

Code contextopen file, cursor

LLM suggestion

Human review

Accept / reject

Inline completions and in-editor chat both follow this shape: the model proposes, a person disposes. That human-review step separates a copilot from an agent (below). Every suggestion is a draft a person looks at before it lands.

Use it when

Well-scoped, repetitive edits where a person can review every change before it lands.
The reviewer has enough context to catch a wrong suggestion.

Skip it when

The task needs many sequential steps with no natural point to pause and review.
Nobody is going to read the diff before accepting it, so the control exists on paper only.

Try this prompt

Suggest a fix for this failing test. Show the diff and explain the change. I'll review it before applying anything.

Agentic loop (tool-use agent)

Autonomous loop

Plan next step

Call a tool

Observe result

Repeats until the model decides the task is done, then returns a final answer.

The defining difference from a chatbot or copilot: the model decides which tool to call, when to call it, and when to stop. Each loop iteration feeds the previous tool's output back in as new context, so the model can react to what it learned at each step.

Use it when

The task needs multiple tool calls and the right sequence isn't known ahead of time.
Intermediate mistakes are cheap to notice and correct within the loop.

Skip it when

The steps are already fixed and known. A workflow pipeline gets the same result with less variance and cost.
A wrong intermediate action would be costly and nothing reviews it before it executes.

Try this prompt

Find and fix the bug causing the checkout test to fail. You have read/write access to the repo and a test runner. Work until the suite passes.

Multi-agent / orchestrator

Hub, spokes, and merge

Orchestratordecomposes the task

Research agent

Code agent

Review agent

Synthesized answer

One agent receives the overall task, breaks it into sub-tasks, and dispatches each to a specialist agent (in parallel or in sequence), then collects and synthesizes their results into one answer. Each spoke is usually itself an agentic loop (above), so this pattern is built on top of the agent archetype.

Use it when

Sub-tasks are independent and benefit from different context, tools, or specialization.
The synthesis step can catch a spoke that got it wrong.

Skip it when

A single agent could do the whole task in one pass. Splitting it multiplies cost.
There's no synthesis logic, only a spoke that happens to run last.

Try this prompt

Research this API's rate limits, then have a separate pass write and test a client wrapper around it, and a third pass review the wrapper for edge cases.

Router / classifier dispatch

Branching dispatch

Classifier

Handler A

Handler B

Handler C

Exactly one branch executes, chosen by the classifier. Unlike the orchestrator above, the others never run.

A lightweight first call reads the request and decides which single, purpose-built handler should take it. A support ticket gets tagged as billing/bug/general and sent to the matching template, prompt, or a different model entirely. The classifier only picks the lane; the handler does the work.

Use it when

Requests fall into a handful of known categories that each deserve a different, purpose-built handler.
Getting the category wrong occasionally is an acceptable cost.

Skip it when

The categories overlap heavily, or one general handler already does fine. A router only adds a misclassification failure mode.
A wrong route is expensive and there's no fallback for a low-confidence classification.

Try this prompt

Read this support ticket and decide: is it a refund request, a bug report, or a general question? Route it to the matching template and fill it in.

Ensemble / self-consistency

Fan-out, fan-in

Same prompt

Run 1

Run 2

Run 3

Vote / synthesize

The same prompt runs several times, sometimes with slightly different phrasing or a nonzero temperature, and the final answer is picked by majority vote or by synthesizing across the runs. This trades extra cost for reliability on tasks where a single pass is inconsistent.

Use it when

The task has a knowable right answer and single runs disagree with each other often enough to matter.
The extra latency and cost of running N times is affordable for how often this gets called.

Skip it when

The task is open-ended or creative. There's no "most common good essay" to vote toward.
A single well-checked run is already reliable enough; the extra runs only add cost.

Try this prompt

Solve this math word problem five times independently, then tell me which final answer appears most often and why the others might have diverged.

Evaluator / LLM-as-judge

Generate-and-judge loop

Generate response

LLM judge scores it

Revise if it fails

Repeats until the judge accepts the output, or a retry cap is hit.

A second model call (sometimes the same model, sometimes a different one) scores or critiques the first call's output against a rubric, and a failing score triggers a revision. Used both as an offline evaluation harness over many examples and as a live gate before an answer ships.

Use it when

You need a scalable, consistent way to score or filter a large volume of model output against a rubric.
The rubric is specific enough that a judge model can apply it consistently.

Skip it when

The judge is the only check on something high-stakes. It can be fooled by confident, well-formatted output the same way a person skimming quickly can.
The rubric is vague enough that the judge's own bias decides the outcome.

Try this prompt

Grade this draft against the rubric below on a 1-5 scale for each criterion, and say specifically what would move it from a 3 to a 5.

Memory-augmented agent

Loop with persistent memory

Recall relevant memory

LLM + recalled context

Write memory update

Each new session recalls what earlier ones wrote, so context persists without replaying the whole history.

Unlike RAG's static knowledge base, this store is written to as well as read from, usually by the same agent, so preferences and facts learned in one session are available in the next one without a person re-explaining them. The loop shape is the same as the tool-use agent above, with memory read/write treated as one more tool.

Use it when

An assistant should carry context, preferences, or facts across separate conversations or sessions.
What's worth remembering stays true across conversations.

Skip it when

Everything relevant fits in the current conversation anyway. Persistent memory adds a staleness and privacy surface for no benefit.
The stored memory can't be reviewed or corrected by the person it's about.

Try this prompt

Remember that I prefer terse code review comments with a suggested fix attached. Use that preference on every review you do for me from now on.

Fine-tuned / specialized model

Linear pipeline, offline

Base model

+ domain training data

Fine-tuned checkpoint

Narrower, faster inference

The training step happens once, offline, ahead of time. Unlike every other archetype here, there's no live pipeline at inference time beyond a normal model call. The payoff is a model that's cheaper, faster, and more consistent at one narrow, repeated task than prompting a large general model for the same thing over and over.

Use it when

One narrow task, run at high volume, where consistency and latency/cost matter more than flexibility.
Enough labeled examples exist (or can be produced) to make training worthwhile.

Skip it when

The task changes often. A fine-tune goes stale the same way any snapshot does, only less visibly.
Volume is too low to justify collecting examples and retraining when the task shifts.

Try this prompt

Fine-tune a small model on 500 labeled support tickets, then compare its classification accuracy against prompting a general model with the same examples as few-shot context.

Workflow automation (LLM-in-pipeline)

Linear pipeline, fixed by code

Ingest

ClassifyLLM call

ExtractLLM call

Write to system

Control flow is fixed by code ahead of time, and the LLM fills one or more defined steps without deciding what happens next. This is the right shape when the sequence of steps is known and stable. It's more predictable and easier to debug than an agent, at the cost of not adapting when a case doesn't fit the pipeline it was built for.

Use it when

The sequence of steps is already known and stable, and you want predictable, debuggable behavior.
Each step's inputs and outputs are well-defined enough to test independently.

Skip it when

Inputs vary enough that a fixed pipeline would mishandle common cases. That's a sign the task needs an agent's judgment.
The steps themselves need to change based on what an earlier step found.

Try this prompt

Write a pipeline: read each incoming email, classify it as billing, support, or sales with one model call, extract the account ID with a second call, then insert both into the ticket queue.

Computer-use / browser agent

Autonomous loop

Read screenscreenshot / DOM

LLM picks an action

Action executesclick, type, scroll

Repeats until the on-screen task is complete.

The same loop shape as the tool-use agent above, except the "tools" are mouse and keyboard actions on a screen instead of API calls. Each iteration re-reads the current screen state before deciding the next action, since a click or page load can change what's on screen in ways the model has to notice.

Use it when

There's no API for the system you need to operate, only a UI.
The UI is stable enough between runs that a screen-reading loop can reliably find what it needs.

Skip it when

An API exists. Call it directly; it's faster, cheaper, and far less brittle than driving the UI.
The UI changes often enough (a moved button, a new dialog) that runs would derail unpredictably.

Try this prompt

Open the billing dashboard, find last month's invoice total, and paste it into cell B2 of this spreadsheet.

2. Complex agent patterns built from the basics

These six combine the archetypes above. A debate setup is two agentic loops plus a judge; a planner-executor is a loop wrapped around another loop. Use one only when a single archetype isn't enough. Each adds complexity (more calls, more places to debug, more cost) that has to pay for itself.

Hierarchical planner-executorIterative goal decomposition Reflection / self-critique agentGenerate-critique-revise loop Debate / adversarial agentsAdversarial fan-out, judged Tree-of-thought explorationBranch, evaluate, prune Swarm / blackboard agentsDecentralized shared state Long-horizon goal-stack agentPersistent loop with a goal stack

Hierarchical planner-executor

Iterative goal decomposition

Planner picks next subgoal

Executor completes it

Planner reassesses

Repeats, pulling one subgoal at a time, until the planner judges the overall goal met.

A planner agent breaks a large goal into an ordered list of subgoals and hands them off one at a time to an executor, which may itself be a full agentic loop from §6. Unlike the flat multi-agent orchestrator above, the planner revisits and re-plans after every subgoal, so the remaining work can change mid-task as the executor learns things the planner didn't anticipate.

Use it when

The goal is too large or ambiguous to decompose correctly in one shot up front.
Later subgoals depend on what earlier ones discover.

Skip it when

The full decomposition is knowable ahead of time. A flat multi-agent dispatch gets there with less overhead.
Re-planning this often would add latency without changing the outcome.

Try this prompt

Break "migrate our billing service off the deprecated API" into an ordered list of subgoals, complete the first one, then reassess whether the remaining list still makes sense before continuing.

Reflection / self-critique agent

Generate-critique-revise loop

Draft answer

Self-critique

Revise

Repeats until the agent's own critique passes, or a retry cap is hit.

The same model plays both roles: it drafts an answer, switches into a critical mode to critique its own draft against the task's requirements, then revises based on that critique. The difference from the evaluator/LLM-as-judge archetype in §6: no independent second model checks the work, so a blind spot in the model's judgment shows up in both the draft and the critique of it.

Use it when

A revised draft is reliably better than a first-and-only draft, and the model can critique its own category of mistakes.
An independent judge model isn't available or is too costly to run on every request.

Skip it when

The task has a failure mode the model systematically can't see in its own output. Self-critique won't catch what the model doesn't know it's getting wrong.
A single well-prompted pass is already reliable enough that the extra round trip isn't worth the latency.

Try this prompt

Draft a response to this customer complaint, then critique your own draft against our tone guidelines, then rewrite it based on that critique.

Debate / adversarial agents

Adversarial fan-out, judged

Question

Agent: argue for

Agent: argue against

Judge decides

Two agents argue opposing positions on the same question to stress-test a claim, with one of them trying to break it. A judge (a third model call, or a person) weighs the two arguments and decides. Ensemble/self-consistency in §6 runs the same prompt several times looking for agreement; this pattern runs opposing prompts looking for disagreement.

Use it when

A claim needs to survive pushback before it's trusted.
The judge step can weigh two arguments on their merits instead of picking whichever sounds more confident.

Skip it when

The question doesn't have two sides. Forcing a debate on a factual lookup adds noise.
Nothing in the system can judge the two arguments fairly.

Try this prompt

Have one pass argue we should ship this feature behind a flag next week, and another argue we should delay a sprint, then judge which argument holds up better and why.

Tree-of-thought exploration

Branch, evaluate, prune

Problem

Branch A

Branch B

Branch C

Weak branches are pruned after evaluation; the survivors are explored further, recursively.

The model generates several possible next steps in parallel, evaluates how promising each looks, discards the weak ones, and keeps exploring only the survivors, recursively, like a search tree. The router in §6 picks one branch up front by classification and never looks back; here, several branches are tried before any are cut.

Use it when

The problem has several plausible approaches and you can't tell which will work without trying them.
A wrong early commitment would be expensive to recover from.

Skip it when

There's usually one obvious approach. Exploring alternatives that were never going to work burns tokens.
The branching factor makes this too expensive for how often the task runs.

Try this prompt

Propose three different approaches to fix this flaky test, sketch each one out one level deep, then rank them and continue only with the most promising one.

Swarm / blackboard agents

Decentralized shared state

Task

Agent 1

Agent 2

Agent 3

Shared blackboard state

No central dispatcher. Agents read and write the shared state independently and asynchronously.

Several lightweight agents work on pieces of the same problem with no controller assigning work. Each reads a shared state (the "blackboard"), contributes what it can given what's there now, and writes its result back for the others to build on. Coordination emerges from the shared state, the opposite design bet from the orchestrator archetype in §6.

Use it when

The problem breaks naturally into many small, loosely coupled contributions.
The system should keep making progress even if individual agents fail or run slow, since there's no single coordination point to bottleneck on.

Skip it when

The work has a clear dependency order. Emergent coordination adds complexity a simple pipeline wouldn't need.
Debuggability matters more than throughput. A blackboard system is harder to reason about after the fact than an orchestrator's explicit dispatch log.

Try this prompt

Set up three independent passes that all read and append to the same shared notes document about this codebase (one cataloguing modules, one flagging dead code, one noting test coverage gaps), with no pass waiting on another.

Long-horizon goal-stack agent

Persistent loop with a goal stack

Check goal stack

Work top sub-goal

Push / pop stack

Runs across many sessions. The goal stack persists, so progress survives a restart.

For objectives too large for one sitting: the agent keeps an explicit stack (or queue) of sub-goals, works the top one, and can push new sub-goals it discovers mid-task before popping back to where it left off. The stack itself is checkpointed, so a restart resumes where it left off. That is the key difference from the memory-augmented agent in §6, which persists learned facts and preferences, not in-progress task state.

Use it when

The objective can't finish in one continuous run because it spans restarts, rate limits, or multiple days.
Sub-goals discovered mid-task need to be handled without losing the thread of the original objective.

Skip it when

The task reliably finishes in one run. The checkpointing machinery is pure overhead for something that never needs to survive a restart.
Losing progress on a restart is acceptable. A simpler stateless loop is easier to reason about.

Try this prompt

Start migrating this monorepo to the new build system. Keep a goal stack of remaining packages, work through them one at a time, and if you find a package with an unexpected circular dependency, push a sub-goal to resolve that before returning to the stack.

3. ML/AI software categories

The patterns above describe how a system is wired. This table covers what it's built from. Most production stacks combine several rows: a RAG pipeline alone typically touches a model hub, a vector database, and an orchestration framework before it answers a question.

Category	What it does	Examples
Model hubs & hosting	Where trained models get published, versioned, downloaded, and often run directly.	Hugging Face, Ollama's model library, Replicate
Training frameworks	The libraries model training code is written against, for defining, running, and differentiating neural networks.	PyTorch, TensorFlow, JAX
Classical ML	Machine learning without deep learning (regression, decision trees, clustering). Still the right tool for many tabular-data problems.	scikit-learn, XGBoost, LightGBM
Agent / orchestration frameworks	Code scaffolding for chaining prompts, tools, and retrieval into the archetypes above.	LangChain, LlamaIndex, Haystack
Vector databases	Store embeddings and run similarity search over them: the retrieval half of RAG (above).	Pinecone, Weaviate, Chroma, pgvector
Experiment tracking / MLOps	Log training runs, compare metrics across them, and manage a model's lifecycle from experiment to deployed checkpoint.	MLflow, Weights & Biases
Inference serving	Run a trained model efficiently at request time: batching, scaling, and managing GPU memory so it isn't wasted per request.	vLLM, NVIDIA Triton, Hugging Face Text Generation Inference

This table leaves out pricing, versions, and endorsements because those go stale fast. For maintained, sourced entries with pricing and a verification date, see the models & tools directory.