Inference Engineering: Part 5/8: When Speculative Decoding Backfires

Hey everyone, welcome to the fifty-eighth issue of The Main Thread. This is part 5 of our 8-part series on inference
engineering.

In the previous parts of this series, we saw our laptop shopping assistant generate its response one token at a time.
It uses an autoregressive language model. This kind of model writes one token, adds it to the text written so far, and then uses that updated text to choose the next token.

Suppose the model has processed the customer's question and product details. It gives every token it knows a score based
on how likely that token is to come next. The decoding rule uses those scores to choose one token. Greedy decoding picks
the token with the highest score. Sampling can choose among several likely tokens using settings such as temperature
and top-p.

Assume the decoding rule chooses the word “For”. The server adds “For” to the response. The model must now use “For“ as part of its input before it can score the next token. It processes “For” with the cached results for the prompt, and the decoding rule might choose “college”. The server adds “college” and the same process begins again.

The GPU can perform many numerical operations inside one model pass in parallel. It can also process tokens belonging
to different requests together. It cannot calculate the correct second output token for this response before the first one has been selected, because the first token changes the input to the second calculation. The third calculation then depends on both selected tokens. Generating 100 output tokens therefore creates a chain of 100 dependent token-selection steps. This serial dependency is the problem we want to reduce.

Suppose a cheaper model guesses several possible next tokens first. The main model can then check those proposed positions together in one pass. If several guesses are accepted, the assistant commits several output tokens while
avoiding the same number of successive passes through the larger model. This draft-and-check method is called speculative decoding. After the shopping assistant enables it, the main model needs fewer passes to write some answers. Yet a few of those answers take longer because creating and checking the draft costs more time than the avoided passes would have taken.

In part 4, we defined time to first token (TTFT), inter-token latency (ITL), time per output token (TPOT), and goodput. We will use those measurements here. First, we need to derive what speculative decoding changes inside one generation cycle.

Ordinary Generation is Sequential

For our use case, suppose the shopping assistant has already written:

For college, prioritize

The main model produces the final answer. Research papers often call it the target model. We will use main model because that name makes its role clearer. Starting from the committed text above, one pass scores the possible tokens
that could come after prioritize. Assume the decoding rule chooses battery. The committed sequence is now:

For college, prioritize battery

Only now can another pass score the possible tokens after “battery“. If the decoding rule chooses, the main model ”life” needs a third pass before it can score the possible tokens after “life”.

This dependency makes ordinary autoregressive decoding sequential:

main model pass 1 -> battery
main model pass 2 -> life
main model pass 3 -> and

In a single model pass, the GPU performs many calculations at once. The server can also use that pass for several customer requests. This parallel work cannot generate many future tokens for one answer. The main model must choose “battery” before it can choose “life”, and it must choose “life” before it can choose “and. Generating these three tokens therefore requires three passes through the main model, one after another. The original speculative decoding paper starts from this same constraint: generating multiple tokens ordinarily requires multiple sequential runs of the main model.

Draft Possible Tokens, Then Verify Them Together

Speculative decoding first asks a smaller language model to guess the next few tokens. We call it the draft model. It needs fewer calculations than the main model so that it can produce each guess faster.

The draft model starts with the text that the main model has already accepted. It guesses the first token and temporarily adds that guess to its own copy of the text. It then uses this longer text to guess the second token. The second guess must be ready before it can guess the third. It repeats this process until it has proposed four tokens (four is just an arbitrary number here).

Producing four draft tokens therefore takes four steps, one after another. The draft model is smaller, so each step can
be faster than a pass through the main model. Those four steps still take time, and their times add together.

At this point, all four guesses already exist. This changes the work the main model has to do. During ordinary generation, the main model had to wait for one token before it could work on the next one. During verification, it already has the four guesses made by the draft model. The main model checks all four guesses in one pass. For each guess, it uses the accepted text and any earlier guesses. It never uses a later guess. The input needed for all four checks is ready before the pass begins so that the GPU can calculate the four positions together.

After the pass, the server reads the results from left to right. It keeps each accepted guess until it reaches the first rejection. It discards every guess after that rejection because those guesses were based on the rejected token. The main model supplies a replacement for the rejected token. The next draft begins after the accepted guesses and replacement.

If all draft tokens are accepted, the main model can produce one additional token from the distribution calculated after
the complete draft. This is often called the bonus token. With a draft length of D, one verification cycle can therefore commit between 1 and D + 1 output tokens.

The main model still does more arithmetic in a verification pass than in an ordinary one-token pass. It is evaluating
several positions. The opportunity comes from doing that work together while reading the main model's weights and
existing KV state for one pass instead of repeating the complete sequence of ordinary passes. NVIDIA's analysis of draft and verification costs shows why both the draft length and the number of tokens eventually accepted affect the result.

One Verification Pass Can Commit Several Tokens

Let's go back to the assistant's committed text:

For college, prioritize

The small draft model proposes 4 tokens:

battery | life | over | gaming

For this greedy example, the main model agrees with the first 2 positions. At position 3, its own selected token is “and”, so it rejects “over”. The token “gaming” was proposed after “over”; that prefix will no longer exist, so the server discards “gaming” without accepting or rejecting it as an independent continuation.

The cycle commits 3 output tokens:

For college, prioritize battery life and

The next drafting cycle begins after “and”.

The vertical order shows the dependency, not elapsed time. In particular, the four steps taken by the draft model are
sequential in this example even though the main model verifies the positions together.

Exactness Means Different Things For Greedy And Sampled Output

Greedy decoding chooses the token with the highest score. During verification, the main model checks whether the draft
model guessed that same token. Matching guesses are accepted. At the first mismatch, the server rejects the draft guess and uses the main model's choice. The final tokens are then exactly the ones the main model would have produced without the draft model. This requires the same main model version, input text, tokeniser, and decoding rule.

Sampling makes a random choice using the probabilities from the model. Suppose only two tokens can come next: “battery”
and “life”. The main model gives “battery” a 40% chance and “life” a 60% chance. Imagine sending the same prompt 100 times. The main model would choose about 40 “battery” tokens and 60 “life” tokens. The exact counts vary, but they should stay close to that 40/60 split over many runs.

Now suppose the draft model uses different probabilities. It gives “battery” an 80% chance and “life” a 20% chance. It
would propose about 80 “battery” tokens and 20 “life” tokens across the same 100 runs. Accepting every draft proposal would give us the draft model's 80/20 split. We need to keep the main model's 40/60 split.

The server corrects the draft proposals this way:

1. The draft model proposes “battery” about 80 times. We need only 40, so the server accepts about half and rejects the other 40.

2. The draft model proposes “life” about 20 times. We need 60, so the server accepts all 20. We are still missing 40 “life” tokens.

3. The 40 rejected “battery” proposals leave 40 empty positions. The server fills all 40 with “life”.

The final result is about 40 “battery” tokens and 60 “life” tokens. This matches the main model's 40/60 split.

The formula describes step 1 for any token. p(x) is the probability from the main model. q(x) is the probability
from the draft model. For “battery”, the server accepts p(x) / q(x) = 40 / 80 = 0.5, or half of the proposals. If the draft model proposes a token less often than the main model wants it, the server accepts every proposal. The min
chooses the smaller of the two answers: the calculated fraction or 1.

min(1, p(x) / q(x))

The second formula describes steps 2 and 3. It asks how much probability is still missing for each token. For “battery”,
40 - 80 is below zero, so nothing is missing. The server changes that negative result to zero. For “life”,
60 - 20 = 40, so all the missing probability belongs to “life”. The server turns the remaining amounts into percentages that add up to 100%. In this example, every replacement is “life”.

normalize(max(0, p(x) - q(x)))

Together, the acceptance rule and replacement rule keep the main model's 40/60 split. The speculative sampling algorithm and correctness proof derive this result formally. Both paths must use the same main model, earlier text, tokeniser, and sampling settings. An approximate rule can change how often different tokens appear, so its effect on answer quality must be measured.

When Does Speculative Decoding Save Time?

Speculative decoding creates guesses, checks them, and decides which tokens to keep. It is faster only when all this work takes less time than generating the same tokens normally. If both paths take the same time, they break even.

Let:

• D be the number of proposed draft tokens.

• A is how many output tokens one cycle adds. If a proposal is rejected, count the accepted tokens and the
  replacement. If all proposals are accepted, count them and the bonus token.

• T_base be the time for one ordinary pass through the main model.

• T_draft(D) be the time needed to create D proposals.

• T_verify(D) be the time for the main model to verify those positions.

• T_overhead be the time for acceptance checks, KV-state updates, rollback, and other state management.

Without drafting, ordinary decoding uses A successive passes through the main model to produce A output tokens. Drafting wins for the cycle only when:

T_draft(D) + T_verify(D) + T_overhead < A * T_base

The left side is the full time spent to commit the cycle's output. The right side is the baseline time for the same number of output tokens. Fewer passes through the main model help only when the left side is smaller. Here is a simple example. Suppose one ordinary pass takes 10ms. Adding three output tokens normally takes three passes, or 30ms.

Now suppose a speculative cycle also adds three output tokens. Its costs are:

The speculative cycle uses only 1 verification pass, but its full cost is 36ms. Ordinary decoding takes 30ms for the
same three output tokens. Speculative decoding is therefore 6ms slower. For this cycle, D = 4 and A = 3. The speedup condition fails because 36 ms < 30 ms is false.

The verification pass takes longer than one ordinary pass because it checks 4 token positions instead of producing 1
token. It can still take less time than 4 ordinary passes because the main model can share some work across those positions.

These numbers are invented to make the comparison easy to see. Real times change with the batch size, sequence length,
hardware, model setup, and other requests using the GPU. The comparison must therefore be measured under real traffic.

Acceptance changes across the workload

The value of A (output tokens in one cycle) is not fixed for a chosen draft length. It depends on how closely the draft model follows the target for the current context.

Some text is easy for the draft model to guess. For example, a repeated product name, a standard disclaimer, or a fixed response format. In these cases, several draft tokens may match what the main model wants.

Other text is harder to guess. For example, the models may disagree sooner when comparing laptops with very similar
specifications. Changing the sampling settings can also change how often their choices agree. The match rate can change again if the main model is updated or users start asking different kinds of questions.

The average acceptance rate does not tell us where the first rejection happens. That is important because every draft token after the first rejection must also be discarded. Suppose the draft model creates 8 tokens, but the main model rejects the second one. The server can keep only the first draft token. It must throw away tokens 2 through 8.

Why did the main model check all 8? It checks every position together in one pass. Only afterwards does the server compare the results in order and find the rejection at token 2. By then, the calculations for tokens 3 through 8 are already complete, even though their results must be discarded.

A longer draft gives the server a chance to accept more tokens. But when a rejection happens near the beginning, most
of that extra work is wasted.

Track 2 numbers:

• Accepted draft tokens: Draft guesses the main model accepts.

• Output tokens added: All tokens the cycle adds to the response.

They can differ. In our example, the main model keeps two guesses and replaces the rejected third guess. The cycle adds
three tokens: two accepted guesses and one replacement. If all four guesses were accepted, the main model could add one extra token. The cycle would add 5 tokens: four accepted guesses and one extra token. The first count measures draft accuracy. The second measures the cycle's progress.

The extra state matters as well. A separate draft model brings its weights and KV cache. Verification needs temporary state for proposed positions. The engine must preserve committed KV entries, hold speculative entries until acceptance is known, then commit or discard them. Implementations differ, but none of these actions has zero cost.

When the server is quiet, the extra memory may not matter. When it is busy, that memory can force the GPU to handle fewer requests at once. More requests wait in the queue, so TTFT increases. Verification can keep the GPU busy longer.
Other streaming requests wait longer between tokens. One request may become faster while the service completes fewer
requests within its latency target. Its goodput goes down.

Measuring the Whole Service

To compare both methods fairly, we can run them on the same real requests while keeping the main model, tokeniser,
hardware, sampling settings, output length, and quality checks unchanged. Any difference then comes from speculative
decoding. Answer quality remains part of the comparison. Exact sampling should keep token probabilities unchanged, but quality checks can catch bugs. With approximate acceptance, the changed answers still need to remain good.

For each verification cycle, useful engine measurements are:

1. Draft length and the position of the first rejection.

2. Accepted draft tokens.

3. Committed output tokens per verification pass.

4. Time spent drafting, verifying, checking acceptance, and updating or rolling back state.

5. Memory used by draft weights, draft KV state, speculative target KV state, and temporary verification work.

These explain why a setting wins or loses, but not what the customer experienced. Timing needs two measurement points:

• Browser: TTFT (time to first token), total response time, and when each streamed update arrives.

• Server: Queue time, the wait before the model runs, and when the output is ready.

Comparing them shows whether a delay came from text generation, server waits, or delivery. Speculative decoding may send several tokens in one update. This changes how latency metrics behave. ITL measures the time between updates. When several tokens come together, ITL sees only one update. TPOT works differently. It divides the time after the first output across the remaining output tokens:

TPOT = (end_to_end_latency - TTFT) / (output_tokens - 1)

The vLLM benchmark definitions use the same distinction. Our report records how tokens were grouped into updates and which formulas we used.

The comparison needs several arrival rates, concurrency levels, question types, and answer lengths. That mix
includes short product questions, long conversations, structured answers, repeated catalog text, and difficult comparisons. Both versions start with the same cache state. Cold and typical warm-cache cases remain separate because cached prefixes change the work surrounding decode.

For each load, our report includes:

• TTFT, client-observed ITL, TPOT, and end-to-end latency distributions.

• Request and output-token throughput.

• The goodput definition from part 4: complete answers per second that meet the shopping assistant's stated TTFT and streaming-pause targets.

• Failures, rejections, timeouts, and requests left unfinished.

• Queue length over time and the maximum sustainable arrival rate.

• Answer-quality results under the same checks.

• Peak memory and the number of requests that can stay active together.

Queue growth and active concurrency matter at every load. A faster isolated request is still a loss if speculative state
leaves more requests waiting or longer verification passes delay other streams. The setting earns its place only when
the same workload and quality checks show better latency and goodput.

Conclusion

Speculative decoding solves a simple problem. Ordinary generation needs one main-model pass for each token. A draft
model guesses several tokens, and the main model checks them together. This helps only when enough guesses are accepted. Wrong guesses, expensive checks, extra memory, and a busy server can remove the gain. Fewer main-model passes do not always mean faster answers.

The final decision comes from real traffic. We need to compare latency, streaming pauses, goodput, memory use, and
answer quality. We keep speculative decoding only when users receive good answers faster and the service handles more
work.

Follow the rest of the series

Subscribe to The Main Thread to receive the remaining 3 issues in the inference engineering series. In the next issue, we will examine when separating prefill and decode across different GPUs helps, and what the KV transfer adds to every request.

Namaste!

© 2026 The Main Thread

Gulmohar Street
Bengaluru, Karnataka 560037, India