how to choose an ocr model
What OCR model should I use for X type of document is the most common question I get by far. The question assumes that you only need to know the type of document to pick the right OCR model, and that's not true.
People start with just how to convert it all to text, which is a fine super-early starting point, but it'll lead you to a wrong model choice. You need to start with what the task is, what that needs, and work backwards from there.
For example, if you are building question answering over medical documents, bounding boxes for citations are often critical for a doctor to validate any claims before prescribing a treatment. In other cases you may be making every law in the U.S. searchable where a markdown conversion works great.
But even when the words are properly extracted many documents have a structure that is important, like in engineering diagrams, charts, or redlines. All these details matter and you need to choose your approach across 6 core pieces of functionality.
- Reading order
- Tables and form fields
- Blank fields and checkboxes
- Coordinates for highlighting evidence
- Relationships between labels and values
- Field types, such as dates, names, and amounts
See examples of how OCR handles tables, forms, and complex layouts →
Once you have narrowed the selection down, take 25 pages from documents you need (including difficult ones) and run them through different models that fit those requirements and look for issues.
Did it preserve the important information? Is the reading order correct? Did it connect labels to the right values? Are coordinates accurate enough to use? Did it invent anything?
The errors need to be weighted by impact on the user or your final goal. Missing a dot at the end of a sentence and missing a dot in the middle of a number are worlds apart even though what was missed was the same. They’re different because of how much worse an incorrect data value is compared to missing sentence punctuation when we look at downstream impact.
For more info, there’s longer guide and talk by my co-teacher Joe Barrow (document researcher at THE document company (Adobe)) for more detail here
Read the longer guide to choosing an OCR model →
Next, I’ll cover tradeoffs between document pipelines and vision-language models (VLMs) and when you should choose one over the other.
Go deeper
- PDF to Production: Build reliable products and workflows that use PDFs and documents in a full-day workshop with 30 days of support and 700+ of OCR credits
