make document answers easy to check
After OCRing text usually you want to do something with the information. Maybe answer questions, make a research report, draft a legal case, or make a medical overview of a patient.
But the OCR process might have errors! And even if OCR was perfect there could be an issue in retrieval or in LLM/agent generation steps. How can we trust the final output?
In most cases if a human cannot trust the output, it’s not useful. And if checking the output to get that trust takes close to the amount of time it would have taken to compile the info yourself, then it’s still not useful.
A citation on a document usually links at least back to the page, but a bounding box or region containing the relevant information is becoming more common because of how much faster you can check for accuracy.
In the example above, the answer is $1,097. The citation points to the table containing the underwriting fee, and the orange box highlights the relevant region on the page. This is much more useful than linking to page 1 and asking the reader to search the whole page themselves.
While there are always exceptions, in general I think every domain specific application or product needs citations. The more precise the citation, the easier it is to verify but the more likely there’s an error. A citation can contain the right text but point to the wrong location. It can point to the right location but omit part of a table row. Or it can highlight a large region that contains the answer along with unrelated values, making the citation technically correct but not very useful.
But these citations are not just for the user, the developer needs to do this same validation to do error analysis and evals and debugging! A citation lets you determine whether the problem came from OCR, retrieval, question answering, or the citation itself. This is important because often a wrong answer gets attributed to LLM hallucination when really the agent is faithfully reporting what it was given and it’s an upstream OCR failure.
As a start, you can use gemini flash to create bounding boxes on top of your existing model outputs and create an annotation app, but you’ll want to work it into your workflow more holistically pretty quickly. Joe and I will be covering the importance of seamless evals (including citations) and how to do it in detail in our upcoming 1 day PDF to Production full-day intensive.
Next, I’ll cover how to do the evaluations, find errors, and turn them into tests to make your system robust and controllable in production.
Go deeper
- PDF to Production: Build reliable products and workflows that use PDFs and documents in a full-day workshop with 30 days of support and 700+ of OCR credits
