RAG vs. Vectorless RAG, clearly explained!

In today's newsletter:

• Speculative decoding, now 2.6x faster on a MacBook:
• RAG vs. Vectorless RAG, clearly explained!
• [Hands-on] Make RAG systems 32x Memory Efficient!

LLM INFERENCE

Speculative decoding, now 2.6x faster on a MacBook:

We ran Qwen3.5 9B on an M5 MacBook Pro with the same prompt, three ways:

• No speculative decoding: 22.0 tokens/s, done in 42.2s
• Regular speculative decoding: 34.7 tokens/s, done in 30.4s
• DFlash + Weaver in Uzu: 92.1 tokens/s, done in 11.0s

That is about 4x faster than plain decoding and 2.6x faster than regular speculative decoding. The video shows all three runs side by side.

Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass.

The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4.

One early mismatch, and everything after it gets thrown away.

Weaver fixes this. It is a tiny adapter, a single Transformer layer with 56.7M parameters, that sits on top of DFlash and:

↳ picks among DFlash’s top candidates left to right, so each pick depends on the last one.

↳ grows a tree of likely continuations instead of a single chain.

↳ lets the main model verify the whole tree in one pass.

On the MacBook, every pass of the 9B model produced 7.5 tokens instead of 1.

On a B200 GPU, the Weaver paper reports a 4.08x speedup over standard decoding, measured across chat, math, and code benchmarks.

Mirai open-sourced the whole stack in Uzu, its inference engine built for Apple Silicon. Trying it takes two commands:

GitHub repo: https://github.com/trymirai/uzu

(don’t forget to star 🌟)

We also wrote a full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible code:

Read it here →

RAG

RAG vs. Vectorless RAG, clearly explained!

The chunk closest to a question is often not the chunk that answers it.

In a standard RAG setup, documents are split into chunks, embedded, and stored in a vector DB. The query gets embedded too, and the top-k closest chunks go to the LLM.

This works when the closest text also holds the answer. On long, structured documents, that assumption breaks in two ways.

Problems with similarity search

Five annual reports from one company usually follow the same template every year, and only the numbers change.

So a question about FY2022 can land just as close to an FY2025 chunk as to the FY2022 one. The embedding captures what a chunk is about, and both chunks are about the same thing.

The LLM then answers from the wrong year. Nothing in the pipeline flags it, because the retrieved chunk really is on topic.

Cross-references cause the second failure. A chunk might say “see Appendix G”, while the table in Appendix G holds the actual numbers.

That table looks nothing like the query, so it never makes it to the top-k. The pointer gets retrieved, but the answer it points to does not.

Fixes on the vector side do help. Anthropic’s contextual retrieval adds document context to each chunk before embedding, and with a reranker it cut top-20 retrieval failures from 5.7% to 1.9%.

But these fixes still rank text by how similar it looks to the query.

How vectorless RAG works

Vectorless RAG drops the embedding model and the vector DB, and replaces similarity search with LLM reasoning.

There are three parts to the process.

First, a preprocessing step parses each document’s layout into a ToC tree index. Every node holds a section title, its page range, and a short summary.

Nothing gets chunked, so a table never gets split across two chunks. And the whole tree is small enough to fit in the LLM’s context. In PageIndex, the most popular open-source implementation, the structure comes from the layout itself, and a basic model only writes the summaries.

Next, at query time, the LLM gets the query along with the tree. It reads the section titles and summaries and picks the few sections most likely to hold the answer.

For example, for a question about debt, it might pick the “Financial Summary” section and also “Appendix G”, because its summary says the debt details are there.

That is how an analyst reads a 10-K. They open the table of contents, go to the financial summary, and follow the note to the appendix.

Finally, the system pulls those full sections and checks whether the evidence is enough.

↳ If it is, the LLM answers from the full sections, with page citations.

↳ If not, it goes back to the tree and opens more nodes.

The LLM also sees the chat history. So if you ask about assets and then follow up with “what about liabilities?”, it goes back to the same report.

What it costs

The trade-off is that every query now runs LLM calls over the tree.

PageIndex tested this on 63 questions over NVIDIA’s FY2022 to FY2026 annual reports. Note that the benchmark comes from the PageIndex team itself.

• Tree retrieval answered all 63 correctly, at about $110 per 1,000 queries.
• The best vector setup reached 92.1%, and only by sending the top 100 chunks, at about $134.
• At top-10, vector search cost about $15 per 1,000 queries, but no embedding model got past 76.2%.

So vector search is cheaper per query at a normal k, but it can be less accurate on documents like these.

Latency follows the same pattern. Every hop through the tree is another LLM call, and PageIndex has not published latency numbers.

The index costs more too. One independent benchmark across about 1,000 documents skipped the tree entirely, because building one took 2 to 5 minutes of LLM calls per document.

And the tree only helps when documents have real structure. A pile of support tickets has no table of contents to reason over.

So the two suit different data:

↳ Use a tree for long, structured documents like filings, contracts, and manuals.

↳ Stick with a vector DB for lots of short, unrelated text.

Vector search asks which text looks like the question. Vectorless RAG asks where in the document the answer should be.

If you want to dive deeper, our full RAG course covers RAG from basics to beyond:

• Part 1: Build the complete RAG workflow, including chunking, embeddings, retrieval, and reranking →
• Part 2: Evaluate retrieval and generation quality with RAG-specific metrics →
• Part 3: Reduce RAG latency with faster retrieval and generation paths →
• Part 4: Extend RAG beyond text to multiple data types →
• Part 5: Understand CLIP embeddings, multimodal prompting, and tool calling →
• Part 6: Build a multimodal RAG system over real-world data →
• Part 7: Use Graph RAG for relationships that ordinary chunk retrieval misses →
• Part 8: Improve retrieval with ColBERT and ColBERTv2 →
• Part 9: Build vision-driven retrieval with ColPali →
• Part 10: Apply the first set of techniques for production RAG systems →
• Part 11: Complete the production RAG techniques and implementation patterns →
• Part 12: Reduce prefill latency by reusing KV caches without a shared prefix →
• Part 13: Preload a corpus before queries arrive and examine the production constraints →
• Part 14: Compress preloaded caches and measure where the approaches break →
• Part 15: Implement Block-Attention, Cartridges, and production preloading →

👉 Over to you: Do your documents have enough structure for a tree index, or does a vector DB still fit better?

HANDS-ON

Make RAG systems 32x Memory Efficient!

There’s a simple technique that’s commonly used in the industry that makes RAG ~32x memory efficient!

• Perplexity uses it in its search index
• Azure uses it in its search pipeline
• HubSpot uses it in its AI assistant

To learn this, we’ll build a RAG system that queries 36M+ vectors in <30ms.

And the technique that will power it is called Binary Quantization.

Tech stack:

• Llama Index for orchestration (open-source).
• Milvus as the vector DB (open-source).
• Kimi-K2 as the LLM hosted on Groq (hosted).

Here's the workflow:

• Ingest documents and generate binary embeddings.
• Create a binary vector index and store embeddings in the vector DB.
• Retrieve top-k similar documents to the user's query.
• LLM generates a response based on additional context.

Let's build it!

Setup Groq

Before we begin, store your Groq API key in a .env file and load it into your environment to leverage the world's fastest AI inference.

Load data

We ingest our documents using LlamaIndex's directory reader tool.

It can read various data formats including Markdown, PDFs, Word documents, PowerPoint decks, images, audio, and video.

Generate Binary Embeddings

Next, we generate text embeddings (in float32) and convert them to binary vectors, resulting in a 32x reduction in memory and storage.

This is called binary quantization.

Vector indexing

After our binary quantization is done, we store and index the vectors in a Milvus vector database for efficient retrieval.

Indexes are specialized data structures that help optimize the performance of data retrieval operations.

Retrieval

In the retrieval stage, we:

• Embed the user query and apply binary quantization to it.
• Use Hamming distance as the search metric to compare binary vectors.
• Retrieve the top 5 most similar chunks.
• Add the retrieved chunks to the context.

Generation

Finally, we build a generation pipeline using the Kimi-K2 instruct model, served on the fastest AI inference by Groq.

We specify both the query and the retrieved context in a prompt template and pass it to the LLM.

Finally, we wrap up the app in a streamlit app and run it:

Moving on, to truly assess the scale and inference speed, we test the deployed setup over the PubMed dataset (36M+ vectors).

Our app:

• queried 36M+ vectors in <30ms.
• generated a response in <1s.

Done!

We just built the fastest RAG stack leveraging BQ for efficient retrieval and using ultra-fast serverless deployment of our AI workflow.

You can find the code for this demo here →

NO-FLUFF RESOURCES TO...

Succeed in AI Engineering roles

All businesses care about impact. That’s it!

• Can you reduce costs?
• Drive revenue?
• Can you scale ML models?
• Predict trends before they happen?

We have discussed several other topics (with implementations) in the past that align with such topics.

Here are some of them:

• Learn MLOps from first principles to production in this course with 18 parts →
• Learn everything about MCPs in this course with 9 parts →
• Learn how to build Agentic systems in this course with 14 parts.
• Learn how to build real-world RAG apps, evaluate, and scale them in this course.
• Learn sophisticated graph architectures and how to train them on graph data in this course.
• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
• Learn how to run large models on small devices using Quantization techniques.
• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
• Learn how to identify causal relationships and answer business questions using causal inference in this course.
• Learn how to scale and implement ML model training in this practical guide.
• Learn techniques to reliably test new models in production.
• Learn how to build privacy-first ML systems using Federated Learning.
• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

PARTNER WITH US

ADVERTISE TO 950k+ AI PROFESSIONALS

Our newsletter puts your products and services directly in front of an audience that matters, including thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.

Get in touch today by replying to this email.