Multimodal architecture: vision and documents with Claude
Converting an image or PDF to text before it reaches the model throws away information a vision-capable model can use directly: table structure, a chart's shape, where something sits on a page. Treating every image as a free drop-in ignores a per-request token cost that scales with what you send. The design question is which of those two failures a given input risks.
1. Images and PDFs as first-class input
A vision-capable model doesn't need a document converted to plain text. Give it the image or PDF directly and it reasons over what's on the page, layout included. A text-extraction step before the model always discards something. A table's rows and columns collapse into a token sequence, and the spatial link between a label and its value is no longer explicit. A chart becomes, at best, whatever caption someone wrote. Sending the image or PDF page keeps that structure available to the model, so you don't depend on an extraction step having preserved or correctly reconstructed it.
2. What vision is good at, and what it isn't
Vision is strong at reading structured or semi-structured visual content: a chart's approximate shape and labeled values, a screenshot of a UI element, a form's layout and filled-in fields, a photo of a whiteboard. That's the case for sending an image directly instead of routing through a separate extraction tool. It's not reliable at exact, pixel-level measurement. Reading a precise value off a dense, low-resolution chart where the point sits between gridlines, or transcribing very small or degraded text with zero tolerance for error, depends on visual interpretation. If the underlying numbers exist in a more exact form elsewhere, interpretation is no substitute.
So when the underlying data exists and precision matters (a table's actual values, a chart's source numbers), send the data. Use vision where no exact underlying data exists, or where the layout itself is part of what needs to be understood.
3. PDFs specifically: layout is signal, not noise
A naive text-extraction library routinely scrambles a multi-column PDF, interleaving the two columns in an order that suits the extractor's left-to-right scan and makes no sense to a person or a model. Table cell boundaries, which tell you a value's row and column, vanish into a flat token stream. Sending the PDF directly lets the model use the rendered layout: tables read as tables, columns in the right order, headers and footers recognized as such instead of injected mid-paragraph.
That's the argument for treating PDF input as its own capability instead of "extract the text, then treat it like any other text." Dense tables, multi-column layouts, and forms are the cases naive extraction handles worst and the cases where reading the original layout gains the most.
4. The real token cost of an image
An image costs tokens, scaling with its resolution and dimensions, the same way a large block of text does. A request with several full-resolution images, or images at more resolution than the content needs, pays in context budget and processing time for detail the task doesn't use. Context engineering argues the same discipline for tool results: an image included because it might be relevant competes for space and attention with everything else in the request. Weigh that cost deliberately. Attaching a file feels lightweight from the caller's side, but it isn't free.
Resize or crop images to the resolution the task needs before sending. Crop a full-page screenshot when only one region matters; downscale a high-resolution scan when a smaller version reads just as reliably. Don't default to whatever resolution the source was captured at.
5. Where OCR-first still wins
Sending documents to vision doesn't replace a dedicated OCR pipeline in every case. Two cases still favor an OCR step first. One is very high volume, where the marginal cost of vision on every document, multiplied across the volume, clearly exceeds a cheaper purpose-built OCR pass followed by a much smaller text-only model call. The other is a hard, auditable requirement for exact character-level transcription, where a dedicated OCR engine's per-character confidence scores give a verifiable signal a vision model doesn't provide in the same explicit form.
Decide by which failure the pipeline can least afford. If it can tolerate occasional interpretive imprecision in exchange for understanding layout directly, use vision. If it's built around strict character-level accuracy, or runs at a volume where per-document cost dominates, an OCR-first design still makes sense, with the model reasoning over the resulting text.
6. Combining vision with tool use
Vision composes directly with tool use. A model reading a scanned form can extract a customer ID from the image and call a tool to look up the account, or read a chart in a screenshot and call a tool to fetch the dataset it was likely built from. This is the same agentic loop building agents with Claude describes for text. The loop mechanics don't change when the input is visual; the image is one more piece of context the model reasons over when deciding the next step.
7. Worked example: a scanned-invoice pipeline
An accounts-payable pipeline processes a few hundred scanned vendor invoices a day, from dozens of vendors, each with its own layout and no shared template. A naive OCR-first design (general OCR pass, then flat text into an extraction step) performs unevenly across vendors because the layouts vary so much. A vendor whose totals sit in a table, label in one column and value in the next, is the worst case for layout-blind extraction: the row-column relationship that says which value belongs to which label is gone before the model sees it.
The fix sends the invoice image directly to a vision-capable extraction call, with a schema (see structured data extraction) defining the fields to pull. The model uses the visual layout (a total in the row labeled "Total," a line-item table with clearly bounded columns) to resolve the cases the layout-blind pipeline got wrong. Images are resized to a resolution that keeps text legible without sending oversized source scans, which keeps per-invoice token cost bounded as volume grows. A few vendors whose invoices are dense, small-print, high-volume forms with strict character-level accuracy requirements stay on a dedicated OCR path, where that guarantee matters more than layout understanding.
See structured data extraction for the schema and validation layer vision-based extraction still needs downstream, and context engineering for treating every input, images included, as something that must earn its place in the context budget.