[Live Session] Green dashboard. Half an answer.

Hi ,

One more before October 20.

Last time I wrote about tests that measure the wrong thing. Here’s a quieter problem: tests that check the wrong thing.

A user pastes a long, messy support thread into your AI assistant. The model hits its max-token ceiling and stops mid-sentence. The API still returns HTTP 200. Your downstream JSON parser fails, or the user gets half a summary.

Your load test logs a success, because it only checks status codes. Two more ways LLM load tests pass when they shouldn’t:

→ A closed workload model. A fixed pool of looping virtual users waits on slow responses, so concurrency caps itself and the queue never builds. Real users keep arriving. Only an open model (users arriving at a rate) shows the pileup.
→ Testing against a mock. Providers rate-limit in tokens per minute, not requests. A handful of long-output requests can trigger 429s while a flood of short ones gets through. A mock never shows you this.

The goal isn’t zero failures. It’s graceful degradation: at capacity, the system sheds load with a fast 503 instead of melting into a latency cascade.

On October 20, Heather Thacker (Gatling) and Stefano Cereda (Akamas) show how to build tests that catch this, then tune the stack:

→ Load tests built for streaming endpoints: TTFT, inter-token latency, real prompt mixes
→ Auto-tuning your serving stack for the lowest cost per million tokens, without breaking your latency SLO

📅 October 20, 2026
🕝 2:30 PM CEST / 8:30 AM ET
⏱ 45 minutes, live + recording

P.S. Open your last LLM load test. Does it assert on anything besides status codes and latency? If not, it can’t see truncated responses or what each request costs you.