OCR pipeline vs VLM?

Pipelines and Vision Language Models (VLMs) are two common ways to OCR a document. But many people don’t understand the trade-offs that the different class of approaches have.

An OCR pipeline uses specialized models for each job: detecting text, recognizing it, and understanding the layout. A VLM takes the whole page in a single call to generate the final OCR output. Let’s talk details and how to pick between them.

Pipelines

Pipelines are fast, inexpensive, and can run on most devices (even a phone). For “regular” documents they work fantastic.

Because they are a sequence of models you can swap and tune different pieces individually. If text recognition works well but table extraction does not, you can replace the table component without changing the rest of the system. You can add specialized handling for handwriting, formulas, or text inside figures.

The tradeoff is that pipelines don’t see the whole page at once, so complicated layouts require more pipeline steps. If you need reading order, that’s another step. If you need tables structured, or form fields, that’s also means more step.

And with pipelines each step can have an error, and those will get carried through the whole system. If a text detector misses a line there’s not way for it to recover and it can fail very badly.

VLMs

VLMs use the whole page as context in a single pass, which solves both the problems with pipelines (cascading errors, and not seeing the page at once). They handle complicated layouts much more robustly because it sees the full page picture together.

However to have 1 model that does all the tasks of a pipeline in 1 pass, you need a much larger model. This means slower and more expensive. The types of failures are often different too. A VLM may invent content when a scan is cut off mid word or table, or add boilerplate content to an empty page.

There are some great approaches to reducing latency with VLMs significantly, but it requires some fine tuning. More on that later in the series.

Choosing between them

If your documents are regular and you need to process millions of pages cheaply, start with a pipeline. If the hard part is understanding unusual layouts, tables, figures, or relationships between parts of the page, start with a VLM.

A pipeline is usually good when you need super low latency, low cost and have fairly simple, regular documents. A VLM is usually good when you have complex layouts or output requirements.

You can also combine the two. A VLM might identify paragraphs, tables, figures, and formulas, then send each region to a specialized model. This is often useful for images and figures (identify images with bounding boxes, crop, send that piece onto a second step).

For a deeper dive into VLMs, check out this talk (also by my co-teacher Joe Barrow like the prior one)

Whichever approach you choose, test it on actual documents you have. We’ll talk about citations next to help with evaluating the whole system.

Go deeper

• PDF to Production: Build reliable products and workflows that use PDFs and documents in a full-day workshop with 30 days of support and 700+ of OCR credits