how I test document systems

Most of the document work I publish starts in private annotation and test apps on my computer. This is a behind-the-scenes look at how I evaluate new document systems.

I need all this because model outputs are hard to compare and there's tons of new models and choices coming out every week that all have small tradeoffs. Two OCR systems can both look pretty good while one screws up chart labels and the other breaks nested table formatting.

There are three parts to it.

1. How I pick what to read to stay up to date
2. How I annotate and do error analysis on OCR models
3. Agentic Q&A lab for end-to-end annotation and error analysis

How I pick what to read to stay up to date

Trying to following everybody on X, linkedin, watch all the github/huggingface orgs, and watch all the different blogs manually is madness. There's too much duplication and noise.

So I built an ingestion pipeline that watches very specific sources across all those channels (and more). This works because it is a very aggressively curated source list based on my taste.

From there I have a scheduled Codex task refreshes it every morning and gives me a short summary of each item. I like this approach because I can start my day reading a summary and asking questions with the sources in context. And then one I have a good high level overview (and the coffee kicks in) I start reading the source articles directly.

A lot of what I write and post about starts here, and then gets triaged into the OCR evals or agent Q&A lab workflows for me to test myself before I share an opinion.

OCR annotation and error analysis

The most important foundation to understanding how system works in practice is looking at inputs and outputs and thinking about what went wrong and why.

I think everyone should build their own data viewers and annotation apps. I preached this before agents could create them for you, and I believe it doubly so now.

It lets me put the same page through lots of models, and I can easily add or change these models out. I can see the raw and rendered output for each model beside the original page.

Then I can annotate errors by tying 3 things together:

1. A note saying what is wrong
2. A bounding box showing where it happened on the PDF
3. The exact lines in the model output

Agentic Q&A lab for end-to-end annotation and error analysis

Sometimes OCR on a document is all that's needed, but often you want to ask an agent questions. For that OCR is one step, but retrieval and the agent harness also play a role and need to be evalled.

So I have a lab that asks questions over a set of documents, and answers that question with citations. I can change out the OCR approach, the retrieval approach, or the agent harness separately to see how the whole system is effective.

In this case I need the OCR annotation capabilities from the prior model, but also be able to see the full agent trace and tool calls and annotate those.

I need to know why something failed. Did the OCR fail to turn the PDF into text correctly? Was the data OCR'd properly but the agent didn't find it? Did the agent have all the right information and still answer incorrectly? This annotation app keeps the answer, citation, and trace together so I can debug and drill into any part.

Isaac

Go deeper

• PDF to Production: Build reliable products and workflows that use PDFs and documents in a full-day workshop with 30 days of support and 700+ of OCR credits