Chaos Engineering Clearly Explained
Chaos Engineering Clearly Explained
What if the best way to prevent an outage is to deliberately cause a failure?
Build Skills Where Enterprise Software Is Opening Up
Presented by Workday
For most of Workday’s history, building on its platform was limited to customers and partners. Now it’s opening access to independent developers, creating a new path to build with Workday applications, data, and AI while developing skills around enterprise HR and finance systems.
Chaos Engineering Clearly Explained
Why would a reliability team deliberately break a system that is already working?
At first, the idea sounds backwards.
If the goal is reliability, shouldn’t we avoid failures rather than create them?
The problem is that modern systems rarely fail in perfectly predictable ways.
A server disappears. A dependency slows down. Network latency spikes. Traffic suddenly increases. One service starts timing out and creates problems somewhere completely different.
You can design for these scenarios.
But until they actually happen, you don’t always know whether the system will behave the way you expect.
Chaos engineering tries to answer that question before a real incident does.
Instead of waiting for failures to appear unexpectedly, teams deliberately introduce controlled disruptions and observe what happens.
The goal isn’t chaos for its own sake.
It’s to discover weaknesses while the experiment is still under your control.
Why chaos engineering exists
Downtime is expensive.
At minimum, it frustrates users and damages trust. In more critical systems, such as healthcare, the consequences can be far more serious.
But preventing every possible failure becomes difficult as systems grow.
A simple application might depend on a database and a few services. A distributed system can involve dozens or hundreds of services, replicas, queues, caches, APIs, networks, and third-party dependencies.
That creates an uncomfortable reality: There will be failures.
The important question becomes whether the system can handle them. Traditional testing helps verify that software behaves correctly under known conditions.
But many production incidents happen when those conditions change.
What happens if a database suddenly becomes unavailable?
What if traffic suddenly doubles?
This is where chaos engineering changes the approach.
Instead of asking: Does the system work when everything behaves normally?
It asks: What happens when something stops behaving normally?
How chaos engineering works
Chaos engineering is often described as deliberately injecting faults into a system.
That’s true, but simply breaking things isn’t enough.
A useful chaos experiment follows a much more deliberate process: Understand normal behavior → Form a hypothesis → Introduce a disruption → Observe the system → Learn from the result
Each step matters.
1. Define the steady state
Before introducing failure, you need to know what “healthy” looks like.
That could mean:
Successful request rate
Response latency
Error rate
Throughput
Queue depth
Availability
These metrics establish the system’s steady state.
Without that baseline, it’s difficult to tell whether the experiment actually affected the system.
2. Form a hypothesis
Next, define what you expect to happen.
For example: If one application instance fails, traffic should automatically move to the remaining healthy instances without significantly affecting users.
Now the experiment has something concrete to test.
Chaos engineering is therefore less about random destruction and more about testing assumptions about resilience.
3. Introduce a realistic failure
Now introduce a disruption that resembles something the system could genuinely experience.
For example:
Terminate a server
Add network latency
Drop network connections
Overload a service
Make a dependency unavailable
Increase traffic
The important part is realism.
Random failures may produce interesting results, but realistic failures tell you much more about how the system might behave during an actual incident.
4. Observe what happens
This is where monitoring and observability become essential.
Did traffic fail over correctly?
Did latency suddenly increase?
Did retries amplify the problem?
Did another service fail unexpectedly?
Sometimes the original hypothesis holds.
Sometimes the experiment reveals a dependency or failure mode nobody had considered.
And that is where much of the value comes from.
5. Learn and improve
The experiment isn’t finished when the fault is removed.
Teams should review what happened, identify weaknesses, and decide what needs to change.
That could mean improving:
Failover mechanisms
Timeouts
Retry policies
Capacity
Monitoring
Incident procedures
System architecture
Then the experiment can be repeated.
The goal is to continuously improve the system’s ability to handle failure.
The hard part: testing somewhere realistic
There’s an important tradeoff in chaos engineering.
The more realistic the environment, the more useful the experiment. But the more realistic the environment, the greater the potential risk.
A staging environment is safer.
However, staging may not have the same traffic patterns, data volume, dependencies, network conditions, or infrastructure behavior as production.
Production gives you the most realistic conditions.
But deliberately introducing failure into production is obviously more dangerous.
That creates one of the central tensions in chaos engineering: How do you make experiments realistic without creating the outage you’re trying to prevent?
The answer is to control the blast radius.
Experiments should start small: Instead of disabling an entire service, test one instance. Instead of affecting every user, expose only a small portion of traffic. Instead of immediately running experiments in production, build confidence in staging first.
As confidence grows, experiments can gradually become more realistic.
Common chaos engineering practices
Fault injection is the practice most closely associated with chaos engineering, but resilience can be tested from several different angles.
These techniques test different parts of resilience.
Some test the software. Others test the infrastructure. And some test the people and processes responsible for responding when things go wrong.
That’s important because resilience isn’t only an architectural property.
A technically recoverable failure can still become a major incident if nobody detects it quickly or knows how to respond.
Implementing chaos engineering
The easiest mistake is starting with the biggest possible experiment.
A better approach is to build confidence gradually.
Start with critical failure scenarios
Begin by identifying parts of the system where failure would matter most.
Ask questions such as:
Which services are critical to the user journey?
Which dependencies could take down multiple services?
Where do we assume failover will work?
Which components have never experienced a real failure?
Which incidents would be most damaging?
These questions help identify useful experiments.
Test in a controlled environment
Start in staging or another controlled environment.
Introduce a small failure and verify that your observability, rollback procedures, and experiment design work as expected.
This gives the team experience with the process before increasing the stakes.
Increase realism gradually
Once the team understands how the system responds, experiments can move closer to real production conditions.
But each increase in realism should come with safeguards.
That might include:
Limiting the number of affected instances
Restricting experiments to a subset of traffic
Using feature flags
Preparing rollback procedures
Setting automatic abort conditions
Closely monitoring system health
The goal is to learn as much as possible while exposing as little of the system as necessary.
Make learning part of the process
The technical experiment is only half of chaos engineering.
The other half is what happens afterward.
Teams need to discuss:
What did we expect?
What actually happened?
What surprised us?
What failed?
What protected us?
What should change before we run the experiment again?
That turns each failure into information.
Over time, the organization becomes better at finding weaknesses before those weaknesses become incidents.
Does chaos engineering make sense for smaller companies?
Chaos engineering is often associated with large distributed systems.
That makes sense.
The more infrastructure, services, and dependencies a company operates, the more possible failure combinations it has to think about.
But the underlying principle applies much more broadly.
A startup doesn’t need a sophisticated chaos engineering platform or hundreds of automated experiments. It might begin with a much simpler question: What happens if our database becomes unavailable? Or: What happens if our most important third-party API stops responding?
Run the scenario in staging.
Observe the behavior.
Fix what breaks.
Repeat.
As the system grows, the experiments can grow with it.
So the difference between a startup and a large technology company isn’t whether chaos engineering is useful.
It’s how much chaos engineering makes sense for the system’s current complexity and risk.
Wrapping up
The hard thing about building reliable systems is that avoiding failure isn’t possible.
But you need to understand what happens when failure inevitably arrives.
That is the idea behind chaos engineering.
Instead of assuming failover works, test it.
Instead of assuming retries behave correctly, stress them.
Instead of discovering an unknown dependency during an outage, try to expose it beforehand.
Don’t create failure just to see what breaks. Create controlled failure to discover what needs to become stronger.
👋 If you found this useful → Like + Restack to help others learn system design.
Subscribe to get high-signal and clear system design deep dives, case studies, and breakdowns straight to your inbox:
Subscribed© 2026 Level Up Coding
548 Market Street PMB 72296, San Francisco, CA 94104