Which OCR Solution Works Best for Your Documents?

In today's newsletter:

• This inference engine runs LLMs 4x faster than llama.cpp:
• Which OCR solution works best for your documents?
• The No. 1 Deep Researcher beats Claude/ChatGPT using a counterintuitive trick.

LLM INFERENCE

This inference engine runs LLMs 4x faster than llama.cpp:

We ran Qwen3.5 9B on a base M5 MacBook Pro with three local inference engines, one at a time.

• llama.cpp reached 22.0 tokens/s
• MLX reached 25.1 tokens/s
• uzu reached 92.1 tokens/s

The video shows all three runs side by side.

The first two numbers don’t mean llama.cpp or MLX are badly built. Both sit right under a hard limit of local decoding.

Every new token needs one full pass through the model, and that pass reads all of the model’s weights from memory.

A base M5 reads memory at about 153 GB/s, and a 4-bit 9B model weighs about 5.2 GB.

So at one token per pass, the ceiling is roughly 153 ÷ 5.2, or about 29 tokens/s, however fast the GPU computes.

uzu gets past that ceiling by producing several tokens per pass instead of one.

↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once.

↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right so the guesses form coherent text, and keeps a few alternative branches as a tree.

↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with.

The main model still decides every token that ends up in the output.

In our run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1.

The weights and kernels are built around this path too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU’s int8 matrix path.

uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript and Rust bindings, so the same engine runs inside a Mac or iOS app.

Find the GitHub repo here →

(don’t forget to star it 🌟)

OCR

Which OCR solution works best for your documents?

The OCR model with the best average score can still be the wrong pick for your documents.

We ran DeepSeek-OCR, PaddleOCR-VL and Datalab Surya on the same 150 pages, from handwritten notes and rotated scans to dense tables and a filled tax form.

The video shows every page, what each parser returned, and why each test passed or failed.

Most OCR comparisons end with one number per parser.

That number tells you who’s ahead on average. It doesn’t tell you where a parser breaks on documents like yours.

And nobody parses an average document. An insurance team parses scanned claim forms, a research team parses two-column papers full of equations, and a finance team parses dense tables.

So instead of one score, we checked every page with simple yes/no tests:

• Is this line present in the output?
• Is this cell under the right headings?
• Does A come before B?

Each test passes or fails, and every failure comes with a reason, like “text missing” or “marked as table, not heading”.

The scoring runs as open code. No LLM judges the output.

Here’s the overall scoreboard from our 150-page run:

→ Datalab: 88.2

→ DeepSeek-OCR: 77.5

→ PaddleOCR-VL: 73.4

If we had stopped at the overall column, the answer would look obvious.

DeepSeek-OCR was the best on tables at 87.2, ahead of Datalab at 78.9 and PaddleOCR-VL at 78.0.

So a team whose documents are mostly dense tables would pick a different parser than the average suggests.

On handwritten notes, the gap went the other way. One parser (Datalab) passed 5 of 6 tests, while the other two passed 0.

The filled 1040 tax form showed the starkest failure.

PaddleOCR-VL returned only the form’s title, so every test looking for a filled-in value failed with “text missing”.

On the same page, Datalab passed 8 of 8 tests and DeepSeek-OCR passed 6 of 8.

A parse that returns only a title is a different risk from one with a few wrong cells. A single average folds both into the same number.

No single parser won on every type of page. The right pick depends on your documents, not on who has the best average.

The tool we used is OmniParse Bench, and it’s 100% open-source.

It has 16,288 tests across 2,937 pages in 92 languages, drawn from 2,343 source documents.

It builds on what olmOCR-bench showed, that pass/fail unit tests are a simple proxy for parsing quality that anyone can check. Datalab’s view is that olmOCR-bench has saturated, and what now separates the top systems there is mostly output conventions and overfitting rather than reading.

Every test belongs to one of three families: text, tables and layout. A family’s score is the share of its tests that pass, and the headline is the mean of the three.

A page where the parser call fails counts as 0, so a parser can’t look better by quietly dropping the hard pages.

Each test is tagged by what makes its content hard, like handwriting, rotation, tiny text, degraded scans, forms, tables, math and multi-column layouts, along with its script and language.

Tags never change whether a test passes. They only select tests, so you can score a parser on handwriting alone, or on French tables alone.

Every score also comes with the number of tests and documents behind it, so a thin slice tells you it’s thin.

Each test also records how its answer was established, such as the publisher’s own markup, a form the authors filled in themselves, or agreement across a panel of independent models.

One thing to note is that Datalab built this benchmark, and Surya is Datalab’s own model.

That’s exactly why the open scorer matters. You don’t have to trust anyone’s leaderboard, since you can rerun every test yourself and read why each one passed or failed.

So you can plug in any OCR solution and see exactly where it holds up and where it breaks.

Before you commit to a parser, run it on the slices that match your documents and read the failures yourself.

GitHub repo: https://github.com/datalab-to/omni_parse_bench

(don’t forget to star 🌟)

AI AGENTS

The No. 1 deep researcher beats Claude and ChatGPT with a trick neither uses.

We studied the open-source architecture behind it.

A counterintuitive thing we found is that the orchestrator agent that runs the entire research strategy has no search access.

It can’t query the web or open URLs.

This looks wrong at first glance. Every other deep research system gives its coordinator far more capability.

For instance:

• OpenAI’s approach trains a single model for many consecutive tool calls. It searches, reads, reasons, and writes the report in a long sequential chain.

↳ The researchers behind the No. 1 system (Onyx; GitHub Repo) observed that this causes the model to spend cycles on low-value searches instead of maintaining a high-level research strategy.

• Anthropic and Google use an orchestrator-researcher pattern similar to Onyx’s system. The key difference is how aggressively Onyx constrains the orchestrator.

Most orchestrators have access to search and retrieval tools alongside dispatch capabilities. And the moment an orchestrator can search, it will.

So instead of decomposing a query into focused research threads, it starts answering the question itself.

It pulls a few results, skips proper task decomposition, and produces a surface-level report from whatever it found first.

Stripping search from the orchestrator forces it to write self-contained and coherent task briefs for each research agent.

The researchers also kept the architecture only two levels deep. When info passes through multiple agents, each one subtly distorts it through summarization/reinterpretation. Keeping it to two levels prevents this.

These two constraints sit inside a larger three-phase pipeline (the visual below maps this):

• Phase 1 decomposes the query into up to 6 research directions. No tool access prevents the model from prematurely answering.
• Phase 2 dispatches 3 isolated research agents. Each runs up to 8 sub-cycles of search, read, and think, to produce an intermediate report with citations. The agents can also search internal enterprise docs (Confluence, Slack, 100+ connectors) with document-level permissions enforced, unlike proprietary solutions.
• Phase 3 runs a deterministic step that renumbers and deduplicates to produce a report with a unified citation map.

This pattern has been ranked No. 1 on DeepResearch Bench. The whole implementation is available on GitHub, and you can try it yourself.

Here’s the Onyx Repo → (don’t forget to star ⭐ )

👉 Over to you: Which page type breaks your current OCR pipeline?

NO-FLUFF RESOURCES TO...

Succeed in AI Engineering roles

All businesses care about impact. That’s it!

• Can you reduce costs?
• Drive revenue?
• Can you scale ML models?
• Predict trends before they happen?

We have discussed several other topics (with implementations) in the past that align with such topics.

Here are some of them:

• Learn MLOps from first principles to production in this course with 18 parts →
• Learn everything about MCPs in this course with 9 parts →
• Learn how to build Agentic systems in this course with 14 parts.
• Learn how to build real-world RAG apps, evaluate, and scale them in this course.
• Learn sophisticated graph architectures and how to train them on graph data in this course.
• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
• Learn how to run large models on small devices using Quantization techniques.
• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
• Learn how to identify causal relationships and answer business questions using causal inference in this course.
• Learn how to scale and implement ML model training in this practical guide.
• Learn techniques to reliably test new models in production.
• Learn how to build privacy-first ML systems using Federated Learning.
• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.