two kinds of PDF evals

Hey,

Evals on OCR outputs are usually much harder than on product agentic systems for two main reasons. This is a huge part of our upcoming course because of how critical it is, and we want to share as much knowledge publicly as possible. So let’s talk about the challenges, and then a process on how to tackle them.

First, figuring out what the input (document) is the hard part is harder to understand. Is it the table that seems clean but does a slight change in formatting part way through? Or is it the massive background watermark that says “DRAFT” that’s very easy to read but not part of the normal prose (but still critical)? Or is it the signature or checkbox on a form? It’s much harder to decompose into the difficult parts than a human text based question.

Second, errors are incredibly hard to spot. Even in this fairly straightforward case of a Berkshire Hathaway table an OCR model got everything correct except for misreading one number in the middle of the page. Very painful and error prone to manually compare every number on just this one page.

So what do we do about it? Well, you do have to look and annotate data but there’s a process that helps you get the most out of it.

1. Gather documents that represent a spread of what you have to deal with (tables, handwriting, charts, etc.). You do not need full documents, you can pull out relevant pages to have a condensed dataset.
2. Run these through several VLMs to create bounding boxes for errors. This will highlight most likely places by seeing where different models disagree.
3. Create an annotation app that lets you draw bounding boxes. You want to tie an error description, to a bounding box on the document, to lines in the model output seamlessly (the image above is actually from one of my annotation apps that I use to create the OCR gallery)
4. Add fast objective checks as you encounter errors so you do not have to re-annotate issues you’ve run into before when you try a new model. For example, two objective checks I made based this document:1. Sum every value in the column and verify it equals this totals row
5. Check this value against a separate different document to make sure it matches. Often values exist in multiple places in legal and financial reports especially.

There’s usually many of these objective checks where you can extract values then do a check. And you can often get creative, like you can extract an EIN number and verify that it’s a legitimate EIN through public records. If you’re dealing with legal paperwork for startups that can be done across many documents for fast checking without making your eyes bleed (too much).

These are called formative evaluations. We discuss both formative and summative evaluation in depth in our course. I hope to see you there!

Go deeper

• PDF to Production: Build reliable products and workflows that use PDFs and documents in a full-day workshop with 30 days of support and 700+ of OCR credits