faster llms with speculative decoding (how it works)

LLMs can be made faster with speculative decoding, even while doing more computation (wut???). It does this because the extra compute is spare GPU compute that would be idle while waiting for other parts of the system to finish its tasks.

But this only works in specific situations (usually low batch size). It's still useful loads of the time, and you can dynamically adjust based on the situation.

This Wednesday, Joe and I are giving a free talk on why this works. We’ll walk through the details and diagrams from his blog post: how tokens are drafted and checked, why memory bandwidth matters, and how this changes as batch size and other things change.

Join us from 7:30–8:30 p.m. Eastern, and bring questions:

Joe’s post explains it visually, so you can read ahead and get questions ready if you'd like:

https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/

Isaac

P.S. Joe and I also teach PDF to Production on November 14. Registration closes November 13. If you want help choosing, evaluating, and deploying an OCR system, the curriculum is here: https://isaacflath.com/pdf2production