Inference Engineering: Part 4/8: Measuring LLM Serving Under Load: Throughput and Latency

Hey everyone, welcome to the fifty-seventh issue of The Main Thread. This is part 4 of our 8-part series on inference
engineering.

In part 1, we followed a question through the server running our laptop shopping assistant. In part 2, we saw why an answer can pause halfway through. In part 3, we saw why a new question can be kept waiting.

Now we need a way to tell whether the service is getting better for its customers.

One number often used to judge an LLM service is output tokens per second. It counts how much answer text the server
produces for everyone together. But it does not tell us how long a customer waited for an answer to start or whether the
answer paused halfway through.

We will measure when each answer starts and finishes, and how long any pauses last. Then we will count how many answers
meet our time limits.

The requests the server receives

To measure the shopping assistant, we send it questions and track its answers. A short laptop question and a long
conversation with product manuals each count as one request, but the long request takes more work before its answer can
start. If we just count the number of requests, this distinction is not clear.

For our shopping assistant, we track:

1. When questions arrive. Ten customers can ask questions over a minute or all at once. If they all ask at once, the server has ten requests to handle with one GPU.

2. How much text comes in. The input can include the customer's question, instructions, product details, and chat history. We should count how many inputs are short, medium, or long. The GPU must process any uncached input before an answer can start; this is the prefill step.

3. How long the answers are. Longer answers need more decode steps and keep the request's KV cache in GPU memory for longer. We should count the tokens the server actually writes, because an answer can stop before it reaches the maximum number of tokens allowed.

4. How many requests are unfinished. Some requests are waiting for an answer to start; other requests are already receiving their answers. Both still need work from the GPU. If answers slow down, more requests remain unfinished at the same time.

5. How much input repeats. If requests begin with the same instructions and those tokens are still cached, the server can reuse the earlier work. Here, we should count how often this happens and how many input tokens it saves.

6. What else uses the GPU? A background job that writes catalogue summaries can delay customer answers because it uses the same GPU. We must include that work if the real service runs it. Our comparison later uses customer questions only.

In part 2, a long new input paused answers already being written. A test containing only short questions would miss that delay. Sarathi-Serve's analysis examines the same interaction between new inputs and ongoing answers.

Time One Request

To measure a customer’s wait, we start a timer when their browser sends the question. We note when each part of the answer reaches the browser.

For the diagram below, assume the answer arrives one output token at a time and ends with the last token. Headers and
empty updates contain no answer text, so they do not mark the start of the answer.

Let's assume the browser sends the question at 0 ms. The model server separately measures how long the question waits in its queue and how long it takes to process the input.

The first token reaches the browser 500 ms after it sends the question. Token 2 arrives at 550 ms and token 3 at 850 ms, leaving 300 ms with no new text. The model server was doing other GPU work during that gap. The customer still waited the full 300 ms.

The gateway sits between the browser and the model server. It can time a request from when it receives the question to when it forwards each part of the answer. That misses the time spent between the browser and gateway. A timer at the model server also misses the gateway's work. These timings cover different parts of the request, so a report must say where they were measured. We use the browser's timings throughout this article.

Let’s assume:

• t_send is when the browser sends the question.

• t_i is when the token i reaches the browser, so t_1 is when the first token arrives.

• N is the number of output tokens.

• t_done is when the browser receives the complete answer.

Time to first token (TTFT)

This measures the wait before an answer starts:

TTFT = t_1 - t_send = 500 - 0 = 500 ms

TTFT tells us the browser waited 500 ms for the answer to start. It does not show where that time went. The question
may have waited in the server's queue or taken longer for the GPU to process. We need server timings to see how long
each step took. TTFT also misses the 300 ms pause later in the answer.

Inter-token latency (ITL)

This measures each subsequent gap:

token_times_ms = [500, 550, 850, 900]
gaps_ms = []
for i in range(1, len(token_times_ms)):
    gaps_ms.append(token_times_ms[i] - token_times_ms[i - 1])
# gaps_ms is [50, 300, 50]

The 300 ms gap shows the pause in the answer. ITL excludes the initial wait because no earlier output token exists
for token 1.

End-to-end latency

This measures the entire request:

e2e_latency = t_done - t_send = 900 - 0 = 900 ms

This tells us when the customer has the complete answer. It combines the initial wait with generation time, so longer answers can have higher end-to-end latency even when their streams move steadily.

Time per output token (TPOT)

This averages the time after the first token across the remaining output tokens:

TPOT = (e2e_latency - TTFT) / (N - 1)
     = (900 - 500) / (4 - 1)
     = 133.3 ms per output token

The 300 ms stall has disappeared into that average. A TPOT limit would constrain the average; it would not guarantee a limit on every gap. For one-token outputs, this formula is undefined. We mark TPOT as unavailable and exclude those requests from TPOT statistics, while retaining their completion status and TTFT. To get one number across many answers, we can average their TPOT values: each answer contributes one number. Or we can average all the gaps between tokens: a long answer contributes more gaps than a short one. The results can differ, so a report must say which method it used.

Our diagram shows one token arriving at a time. In a real reply, several tokens may arrive in one update. The browser
can time each update, but it cannot measure a separate gap for tokens that arrived together. The vLLM benchmark definitions call the time between updates ITL. They combine those gaps from all successful answers and calculate TPOT separately for each answer.

The browser may receive a separate signal that the answer is complete after the last text arrives. Measure when the
last text arrives and when that signal arrives. If the signal comes later, the TPOT formula above includes that extra wait, even though no new text appeared. In our diagram, the last token and completion happen together.

Counting Output From All Customers

Output tokens per second counts the answer tokens that reach customers during a set period. In our 5-second comparison, we count those tokens and divide by 5. We do not count the tokens in the customers' questions.

Output tokens per second = answer tokens received by all customers / seconds measured

If one customer is still waiting for an answer, the text sent to other customers still adds to this number. Remember, the number doesn’t tell us how long anyone waited.

We compare two runs with the same questions and the same GPU. In run A, the server does not wait to collect questions
before starting work on a new one. In run B, the server may hold a new question for up to 250 milliseconds, giving
other questions time to arrive so it can process them together. That wait is the only server setting we change.

Processing several questions together can increase output because one GPU pass uses the model's weights for several
questions. Sarathi-Serve's analysis explains this batching benefit. But the time B spends waiting for more questions can delay the start of an answer. The following example shows how these two results can occur together. They do not predict what a 250-ms wait would do on a real server.

How Many Answers Meet Our Time Limit

For this comparison, request goodput means the number of answers per second that finish and meet our time limits.
An answer counts only if:

1. The browser receives the complete answer before the 5-second comparison ends.

2. The first text reaches the browser within 1 second of the question being sent.

3. No pause between text updates exceeds 100 ms.

Hence,

Request goodput = answers that pass all three checks / seconds measured

Unfinished, failed, rejected, and empty answers don’t count; we report them separately. An answer sent in one text update has no pause to check, but it still must start within 1 second.

These time limits are choices for our shopping assistant. DistServe uses a different goodput measure: the highest request rate one GPU can handle while at least a chosen percentage of requests meet TTFT and TPOT limits. We check each answer's longest pause because an average TPOT can hide a single long pause.

More Tokens Per Second Can Come With Slower Starts

We want to know which setting gives more customers a timely answer. For this example, an answer is timely if it
finishes during the 5 seconds, starts within 1 second of the customer sending the question, and never pauses for
more than 100 ms.

Both runs use the same model and GPU. They receive the same mix of short and long questions at the same times, and both start with an empty prefix cache. Each completed answer has 100 tokens and passes the same quality check. The first row shows the only change: Run A adds no wait to group new questions; Run B may wait up to 250 ms.

Run B finishes all 20 answers in 5 seconds. Run A finishes 18 and leaves 2 questions waiting. That gives Run B the
higher tokens-per-second score.

But answers take longer to start in Run B. The p50, p95, and p99 first-token waits are all higher in Run B. Ten completed answers in Run B start after 1 second. In Run A, two completed answers start after 1 second. Pauses after an answer starts stay below 100 ms in both runs.

Run A gives 16 customers an answer that meets both time limits. Run B gives 10. We still need to track the two customers waiting in Run A and test longer to see whether either setting can keep up with new questions.

These figures include only completed answers. The first-token figures use one wait per answer; the pause figures use
each answer's longest pause. Combining every pause instead would give long answers more weight because they have more
pauses. Failed and unfinished questions are absent from these figures, so a report must count them separately.

Find the Arrival Rate Service Can Sustain

If we test one question at a time, we miss the delays caused by other customers' questions. If we test only short questions, we miss the time the server spends reading long ones. We should be able to find out how long customers need to wait when a variety of questions come in together.

If many questions have the same prefix, the server can reuse work it did when it first read those words. The server then has less to process for later questions, which can raise the tokens-per-second score. vLLM warns that sending the same questions again can raise the score because the first run may have left that work in the cache. We should say what was already in the cache when each run began and make questions share the same opening as often as customer questions do.

The table above covers 20 questions over 5 seconds. To find how many questions per second Run A and Run B can handle, we need to keep sending questions and see whether the number waiting grows. Compare the settings this way:

1. Keep the setup and answer quality the same

Keep the model, GPU, software, and answer settings the same in both runs. Write down the versions and settings so someone else can repeat the comparison. Check that both runs recommend products that exist and fit each customer's question.

2. Send the same questions

Send each question at the same time in both runs. Include bursts, short and long questions, questions that lead to short and long answers, shared openings, and any background work. Track each question and the length of its answer. If one run writes less text, that alone may explain a better speed score.

3. Start with the same cache state

Load the model and finish startup work before measuring. Then either clear the prefix cache before both runs or fill it with the same earlier questions. Report how many questions used the cache and how many input tokens were reused. Reusing a few words saves less work than reusing a long opening.

4. Send more questions each second

Increase how often questions arrive, but keep the bursts. Check that each question was sent when planned. If the sender waits for an answer before sending another question, it sends fewer questions as the server slows. That can hide the point where the server stops keeping up.

5. Measure waits, failures, and cost

At each rate, measure how long answers take to start, the pauses while text appears, and the time to finish. Keep separate results for short and long questions. Count timely answers, failures, rejections, and questions still waiting. A growing waiting line means the server cannot keep up; check rejections too, since they can stop the line from growing. Divide the full server cost, including work on failed questions, by the number of timely answers. If no answers are timely, say so.

6. Run long enough to see slow answers

Measure each setting for the same amount of time and repeat the runs. Count answers that finish during that time, even if their questions came earlier. Track questions still waiting when measurement starts or ends, and find out what eventually happens to them. Report how many answers were used
to calculate p95 and p99. A p99 based on 20 answers tells us little.

For a longer run of the shopping assistant, we could decide in advance that a server setting passes only when:

• At least 95% of questions get a complete answer that starts within 1 second and has no pause longer than 100 ms.

• At most 1% of questions fail or are rejected.

• The number of questions waiting does not keep growing.

When checking the 95% and 1% rules, count every question sent during the measured period. Keep following questions
that are still unfinished when the period ends. A rejected question counts among rejections and cannot count as a
timely answer. Separately, count timely answers that finish during the period to calculate goodput. A question sent
near the end may finish afterwards, so these counts can differ.

Send more questions each second in each new run. Check the three rules, then repeat the run to see whether the result
holds. Do this for Run A and Run B. The highest number of questions per second that passes the rules is what each
setting has shown it can handle.

Conclusion

Output tokens per second counts how much answer text the server writes for all customers together. It does not tell us
how long an answer takes to start, whether it pauses, or whether it finishes. In our example, the setting with the
higher score gave fewer customers timely answers.

We measured those waits from the customer's side and checked each complete answer against the 1-second start and 100 ms pause limits. The number of answers that meet both limits per second is goodput for the shopping assistant. We also counted unfinished questions because those customers were still waiting.

The numbers depend on which questions arrive and when. A burst makes questions compete for GPU time. A long question
takes more work to read before its answer starts. A repeated opening can let the server reuse cached work. Report the question mix and cache state alongside the score.

Before changing a server setting, check whether it keeps delivering complete, accurate answers within the time
limits as more questions arrive. Count failures and questions still waiting, and calculate what each timely answer costs.

Subscribe to The Main Thread to receive the remaining 4 issues in the inference engineering series. In the next issue, we will examine when speculative decoding speeds up sequential generation and when the extra work makes it slower.

Namaste!

© 2026 The Main Thread

Gulmohar Street
Bengaluru, Karnataka 560037, India