Agents 2.0 is here!

In today's newsletter:

• This inference engine runs LLMs 4x faster than llama.cpp.
• Agents 2.0 is here!
• Continuous batching in LLMs, explained visually.

OPEN-SOURCE

This inference engine runs LLMs 4x faster than llama.cpp:

We ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.

• llama.cpp reached 22.0 tokens/s
• MLX reached 25.1 tokens/s
• Uzu reached 92.1 tokens/s

The video shows all three runs side by side.

The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.

Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.

A base M5 provides about 153 GB/s of memory bandwidth.

At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.

Uzu gets past that by optimizing speculative decoding for Apple Silicon.

Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.

Uzu pushes this further in two ways.

→ It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.

→ It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.

The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.

In our run, every pass of the 9B model produced 7.5 output tokens on average.

Ordinary decoding produces exactly 1.

That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.

The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.

It is an open-source inference engine with bindings for Rust, Swift, Python, and TypeScript.

The repository is here: https://github.com/trymirai/uzu

(don't forget to star 🌟)

TOGETHER WITH SWITCH

Agents 2.0 is here!

Anthropic. OpenAI. xAI.

Are all racing to build an AI coworker.

These are agents that can work alongside humans, talk to other agents, and coordinate tasks in a shared workspace.

But the catch is that each product ties the model, runtime, memory, and orchestration to one company’s stack.

Most teams won’t use one provider for every job. They may use Claude Code, Codex, OpenCode running locally, and custom agents for internal systems. Making them collaborate with each other and your team usually means building the glue yourself.

That is vendor lock-in.

In the video above, we have shown how to bring humans and agents from different providers into one shared workspace using an open-source coordination layer you can host yourself.

Humans can talk directly to agents. Agents can talk to each other. And people can join the conversation, add context, or give further instructions as the work unfolds.

The project is called Switch, and it brings this collaboration into Slack, Teams, Discord, Telegram, or Mattermost.

You give each agent a role and can work through a Manager. The Manager creates rooms, brings in the relevant agents, carries context between them, and returns the result.

In the video below, we gave the Manager two incidents.

→ The inference API started returning 500s after a morning deploy

→ A recommendation-model training job had been frozen for six hours

From one message, it created two linked rooms, brought in a Software Engineer, Architect, and QA Lead, and ran both investigations in parallel.

Then the engineer investigating the training failure checked the linked inference room and spotted the same config push behind both failures.

The Manager returned one consolidated RCA to the original channel.

Throughout the run, we could follow the conversations, give additional instructions, and inspect the underlying agent sessions in Switch Console.

That is what we mean by Agents 2.0. Humans and agents working together, sharing context, and contributing as the work unfolds.

We build everything locally with Docker and Mattermost and follow the run through to the final handoff.

Models will keep changing, so the coordination layer should be able to run separately from any one provider.

GitHub: http://github.com/sandbox-quantum/switch

(don’t forget to star 🌟)

LLMOps

Continuous batching in LLMs, explained visually:

In traditional ML inference, a batch is a matrix.

Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.

LLM decoding does not work that way.

One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.

Under static batching, membership is fixed when the batch starts.

A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400.

The GPU keeps paying the full weight read for a batch that is mostly empty.

Loading model weights out of HBM costs the same whether four slots are producing tokens or one.

Continuous batching moves the decision boundary.

Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again.

A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done.

Anyscale benchmarked both OPT-13B on a single A100.

With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving.

Variance drives the entire gap.

Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst.

None of this alters the model.

vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA implements the same mechanism under the name in-flight batching.

The animation below depicts the continuous batching process:

To dive deeper into continuous batching specifically, we wrote a full breakdown of the scheduler underneath it.

It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation.

Read it here →

NO-FLUFF RESOURCES TO...

Succeed in AI Engineering roles

All businesses care about impact. That’s it!

• Can you reduce costs?
• Drive revenue?
• Can you scale ML models?
• Predict trends before they happen?

We have discussed several other topics (with implementations) in the past that align with such topics.

Here are some of them:

• Learn MLOps from first principles to production in this course with 18 parts →
• Learn everything about MCPs in this course with 9 parts →
• Learn how to build Agentic systems in this course with 14 parts.
• Learn how to build real-world RAG apps, evaluate, and scale them in this course.
• Learn sophisticated graph architectures and how to train them on graph data in this course.
• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
• Learn how to run large models on small devices using Quantization techniques.
• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
• Learn how to identify causal relationships and answer business questions using causal inference in this course.
• Learn how to scale and implement ML model training in this practical guide.
• Learn techniques to reliably test new models in production.
• Learn how to build privacy-first ML systems using Federated Learning.
• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Partner with US

ADVERTISE TO 950k+ AI Professionals

Our newsletter puts your products and services directly in front of an audience that matters, including thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.

Get in touch today by replying to this email.

Today’s email was brought to you by Avi Chawla and Akshay Pachaar.

Looking for more? Unlock our premium DS/ML resources.

© 2026 Daily Dose of Data Science