mailwal

The 21 DevOps Interview Questions: Production Scenarios Every Top IT Company Expects You to Understand

Arvind Verma

The 21 DevOps Interview Questions: Production Scenarios Every Top IT Company Expects You to Understand

Linux | Docker | Kubernetes | AWS | Terraform | CI/CD | SRE | DevSecOps

DevOps interviews are no longer just about remembering commands or explaining textbook definitions. Interviewers want to know how you respond when a production server runs out of disk space, Kubernetes returns HTTP 503 errors, Terraform fails halfway through deployment, or an application suddenly becomes slow.

This guide covers 21 production-oriented practice questions across six essential areas. Every question includes eight learning attributes to help you understand the concept, investigate failures, and explain your approach confidently.

Note: These are general DevOps interview preparation scenarios, not verified questions from any specific company’s official interview.

Section 1: Linux and Networking

Q1. A Linux server is running out of disk space. How would you troubleshoot it?

Simple Concept

Disk troubleshooting means identifying which filesystem or files are consuming storage and safely recovering space.

Why Do We Need It?

A full disk can prevent applications from writing logs, databases from operating correctly, and services from starting.

How Does It Work?

Check filesystem usage, identify large directories, inspect logs and temporary files, and determine whether deleted files are still held open by running processes.

Production Example

An application generates excessive logs until /var reaches 100% usage. The application then fails to write new logs and starts returning errors.

Command / YAML

df -h
df -i
du -xhd1 /var 2>/dev/null | sort -h
sudo lsof +L1

Common Interview Trap

Deleting files blindly. Confirm what the files are, whether they are required, and whether log rotation or service configuration needs correction.

Candidate-Style Answer

“I would identify the affected filesystem using df -h, inspect directory usage with du, and check inode exhaustion with df -i. I would locate the largest files, check logs and deleted-but-open files, recover space safely, and implement appropriate log rotation or monitoring.”

Memory Trick

Disk → Directory → File → Cause → Cleanup → Prevention.

Q2. Explain the difference between a load balancer, reverse proxy, and API gateway.

Simple Concept

A load balancer distributes traffic, a reverse proxy receives requests on behalf of backend servers, and an API gateway adds API-focused routing and management capabilities.

Why Do We Need It?

These components help manage traffic, protect backend services, improve availability, and control access to applications.

How Does It Work?

A load balancer selects eligible targets. A reverse proxy forwards requests to backend services. An API gateway may also enforce authentication, rate limits, request transformations, and API policies.

Production Example

An application uses an AWS Application Load Balancer to distribute requests across targets, while an API gateway handles authentication and throttling for public APIs.

Command / YAML

curl -I https://example.com
curl -v https://example.com/api/health

Inspect the actual proxy, listener, routing, and API policy configuration for the environment.

Common Interview Trap

Claiming that these components are mutually exclusive. Their functions overlap, and a single product may perform multiple roles.

Candidate-Style Answer

“A load balancer distributes traffic across targets, a reverse proxy forwards client requests to backend services, and an API gateway provides API-specific controls such as authentication and throttling. I select the component based on routing, security, availability, and operational requirements.”

Memory Trick

Load Balancer = Distribute. Reverse Proxy = Forward. API Gateway = Manage APIs.

Q3. A server is running, but the application is unreachable. How would you investigate?

Simple Concept

A running server does not guarantee that the application, network, or service endpoint is healthy.

Why Do We Need It?

Connectivity problems can originate from DNS, firewall rules, listeners, application processes, proxies, or dependencies.

How Does It Work?

Test the request path from name resolution to network connectivity, listening ports, application health, and downstream dependencies.

Production Example

The virtual machine is healthy, but the application is listening on 127.0.0.1 instead of the expected network interface.

Command / YAML

getent hosts example.com
ip route
ss -lntp
curl -v http://localhost:8080/health
sudo journalctl -u myapp --since “15 minutes ago”

Common Interview Trap

Restarting the server before identifying the failing layer.

Candidate-Style Answer

“I would reproduce the issue and check DNS resolution, routes, firewall rules, listening ports, application logs, and local health checks. I would then test connectivity from the relevant network path and isolate whether the issue is in the application, server, or network.”

Memory Trick

DNS → Route → Port → Process → Application → Dependency.

Q4. A Linux server suddenly experiences high CPU and memory usage. What would you do?

Simple Concept

Resource troubleshooting identifies processes or system conditions consuming excessive CPU or memory.

Why Do We Need It?

Resource exhaustion can increase latency, trigger out-of-memory events, and cause application failures.

How Does It Work?

Identify resource-intensive processes, inspect system metrics, check logs and recent changes, and distinguish CPU saturation from memory pressure.

Production Example

A newly deployed application creates excessive worker processes, driving CPU usage close to 100% and increasing request latency.

Command / YAML

top
free -h
ps aux --sort=-%cpu | head
ps aux --sort=-%mem | head
vmstat 1 5

Common Interview Trap

Killing the largest process immediately without checking whether it is a critical database, system service, or production workload.

Candidate-Style Answer

“I would identify the resource bottleneck using top, free, and process-level metrics. Then I would inspect logs, recent deployments, process behavior, and system pressure. I would apply a controlled fix and monitor resource usage and application latency afterward.”

Memory Trick

Measure → Identify → Investigate → Recover → Monitor.

Section 2: Docker and Kubernetes

Q5. A Kubernetes pod is stuck in CrashLoopBackOff. How would you troubleshoot it?

Simple Concept

CrashLoopBackOff means Kubernetes is repeatedly restarting a container that exits, with increasing delays between restart attempts.

Why Do We Need It?

The cause may be a configuration error, application crash, missing dependency, invalid command, or resource limit.

How Does It Work?

Inspect pod events, current and previous container logs, termination reasons, environment configuration, probes, and resource usage.

Production Example

An application crashes because its database connection string is missing from the deployment configuration.

Command / YAML

kubectl get pods
kubectl describe pod <pod-name>
kubectl logs <pod-name> --previous
kubectl get events --sort-by=.lastTimestamp

Common Interview Trap

Assuming every restart is caused by insufficient memory. Check the actual termination reason before changing resource limits.

Candidate-Style Answer

“I would inspect pod events and previous container logs, then review exit codes, environment variables, mounted secrets, probes, and resource limits. After identifying the cause, I would correct the configuration or application defect and verify that the pod remains healthy.”

Memory Trick

Events → Previous Logs → Exit Code → Configuration → Fix.

Q6. A Kubernetes Service returns HTTP 503 errors. How would you resolve it?

Simple Concept

HTTP 503 indicates that a service cannot currently handle a request. The response may originate from an Ingress controller, gateway, load balancer, or application.

Why Do We Need It?

A successful deployment does not guarantee that traffic can reach healthy application endpoints.

How Does It Work?

Trace traffic through the load balancer, Ingress, Service, pod endpoints, and application dependencies.

Production Example

All pods exist, but failed readiness probes leave the Service without ready endpoints. The Ingress controller returns 503.

Command / YAML

kubectl get pods -o wide
kubectl get svc
kubectl get endpointslices
kubectl describe ingress <ingress-name>
kubectl logs deployment/<controller-name> --since=15m

Common Interview Trap

Restarting every pod without checking which component generates the 503.

Candidate-Style Answer

“I would identify the component returning 503, then check pod readiness, Service selectors, EndpointSlices, Ingress routing, and load balancer target health. I would fix the confirmed cause, verify successful requests, and monitor error rates.”

Memory Trick

Identify Responder → Trace Traffic → Check Endpoints → Verify.

Q7. How would you perform a zero-downtime Kubernetes deployment?

Simple Concept

A zero-downtime deployment aims to release a new application version without interrupting successful user requests.

Why Do We Need It?

Unplanned downtime affects customer experience, revenue, and service reliability.

How Does It Work?

Use multiple replicas, readiness probes, rolling updates, sufficient capacity, compatible application versions, and controlled traffic routing.

Production Example

A Deployment updates application replicas gradually. New pods become ready before they receive traffic, while existing replicas continue serving requests.

Command / YAML

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

This is a partial Deployment configuration. The application must also have suitable readiness probes, sufficient cluster capacity, and compatible shutdown behavior.

Common Interview Trap

Assuming a rolling update alone guarantees zero downtime. Database migrations, broken readiness checks, and insufficient capacity can still interrupt service.

Candidate-Style Answer

“I would use multiple replicas and a rolling update strategy with appropriate readiness probes and capacity. I would verify backward compatibility, monitor health and latency during rollout, and maintain a tested rollback procedure.”

Memory Trick

Replicas → Readiness → Gradual Rollout → Monitor → Rollback if Needed.

Q8. What is the difference between Deployment, StatefulSet, and DaemonSet?

Simple Concept

A Deployment manages replicated application workloads, a StatefulSet manages workloads requiring stable identities or persistent storage associations, and a DaemonSet runs a pod on each eligible node.

Why Do We Need It?

Different workloads have different requirements for identity, storage, scheduling, and lifecycle management.

How Does It Work?

Deployments manage ReplicaSets, StatefulSets maintain stable ordinal identities, and DaemonSets ensure eligible nodes run the required pod.

Production Example

Use a Deployment for a stateless web API, a StatefulSet for a database that requires stable identities, and a DaemonSet for node-level logging agents.

Command / YAML

kubectl get deployments
kubectl get statefulsets
kubectl get daemonsets
kubectl describe statefulset <name>

Common Interview Trap

Claiming that StatefulSets automatically provide database backups or high availability. Those capabilities require additional design and configuration.

Candidate-Style Answer

“I use Deployments for stateless services, StatefulSets when stable identity or persistent storage associations are needed, and DaemonSets for node-level agents. I choose based on workload requirements rather than treating every workload the same way.”

Memory Trick

Deployment = Stateless. StatefulSet = Stable Identity. DaemonSet = Per Eligible Node.

Section 3: AWS and Terraform

Q9. Terraform apply fails halfway through creating infrastructure. What is your recovery plan?

Simple Concept

A failed Terraform apply may leave some resources successfully created or modified.

Why Do We Need It?

Blind retries or manual deletion can cause state inconsistencies and unexpected infrastructure changes.

How Does It Work?

Inspect the error, determine completed changes, compare actual infrastructure with Terraform state, correct the cause, and review a fresh plan.

Production Example

Terraform creates a VPC and subnets but fails while creating an IAM role due to insufficient permissions.

Command / YAML

terraform state list
terraform show
terraform plan

Common Interview Trap

Assuming that a failed apply automatically rolls back every change.

Candidate-Style Answer

“I would inspect the error and determine which resources were successfully changed. I would compare state with actual infrastructure, fix the underlying issue, and review a new plan before applying again.”

Memory Trick

Inspect → Reconcile → Fix → Plan → Review.

Q10. How would you structure Terraform for development, staging, and production?

Simple Concept

Multi-environment infrastructure management keeps environments organized while enabling consistent, reusable provisioning.

Why Do We Need It?

It reduces configuration duplication and helps prevent accidental changes to production.

How Does It Work?

Use reusable modules, environment-specific configuration, separate state boundaries where appropriate, and controlled permissions for each environment.

Production Example

A team uses shared VPC and EKS modules but applies different sizing, scaling, and access policies in development and production.

Command / YAML

terraform/
├── modules/
│   ├── vpc/
│   └── eks/
└── environments/
    ├── dev/
    ├── staging/
    └── prod/

Common Interview Trap

Assuming that separate directories alone guarantee isolation. State storage, credentials, permissions, and deployment controls also matter.

Candidate-Style Answer

“I would create reusable modules and separate environment-specific configuration and state. I would apply least-privilege access, protect production changes through reviews, and use plans to validate differences before deployment.”

Memory Trick

Reusable Modules + Separate Configuration + Isolated State + Controlled Access.

Q11. Why is Terraform state locking important?

Simple Concept

State locking prevents supported Terraform operations from simultaneously modifying the same state.

Why Do We Need It?

Concurrent changes can conflict and leave infrastructure management inconsistent.

How Does It Work?

A supported backend acquires a lock before an operation that requires exclusive state access and releases it when the operation finishes.

Production Example

Two engineers attempt to apply changes to the same production state. Locking prevents conflicting operations from proceeding simultaneously.

Command / YAML

terraform plan
terraform apply
terraform force-unlock <LOCK_ID>

Use force-unlock only after verifying that the lock is genuinely stale and no operation is still running.

Common Interview Trap

Manually deleting lock records or force-unlocking an active operation without investigation.

Candidate-Style Answer

“State locking protects shared Terraform state from concurrent modifications. I use a backend that supports locking and investigate failed operations carefully. I only force-unlock after confirming that no active operation owns the lock.”

Memory Trick

One State → One Active Writer.

Q12. How would you import existing infrastructure into Terraform?

Simple Concept

Terraform import associates an existing resource with a Terraform resource address so it can be managed through infrastructure as code.

Why Do We Need It?

Organizations often have manually created resources that need consistent, version-controlled management.

How Does It Work?

Define the resource configuration, import the existing resource, inspect the state, and adjust configuration until the plan reflects the intended infrastructure.

Production Example

An S3 bucket was created manually. The team imports it into Terraform to manage its configuration and future changes consistently.

Command / YAML

resource “aws_s3_bucket” “existing” {
  bucket = “example-existing-bucket”
}
terraform import aws_s3_bucket.existing example-existing-bucket
terraform plan

Common Interview Trap

Assuming import automatically generates a complete, correct configuration. Imported state does not replace the need to define and validate configuration.

Candidate-Style Answer

“I would define the matching Terraform resource, import the existing resource into the correct address, and review the state. Then I would refine the configuration and run a plan to ensure Terraform does not propose unintended replacements or deletions.”

Memory Trick

Define → Import → Inspect → Configure → Plan.

Section 4: CI/CD and Automation

Q13. A CI/CD deployment pipeline fails. How would you investigate?

Simple Concept

Pipeline troubleshooting identifies the stage and underlying cause of a failed build, test, security scan, artifact operation, or deployment.

Why Do We Need It?

A pipeline failure can delay releases or introduce risk if teams bypass controls to get a deployment through.

How Does It Work?

Identify the first failing stage, inspect logs and exit codes, reproduce the issue, and check dependencies, credentials, artifacts, and environment configuration.

Production Example

A pipeline fails during deployment because the CI identity no longer has permission to update the target environment.

Command / YAML

git status
docker build -t my-app:test .
kubectl rollout status deployment/my-app

Use commands appropriate to the failed stage; inspect CI logs to identify the first meaningful error.

Common Interview Trap

Rerunning the entire pipeline repeatedly without understanding why it failed.

Candidate-Style Answer

“I would locate the first failing stage and inspect its logs, exit code, recent changes, credentials, and dependencies. I would reproduce the issue when possible, correct the cause, rerun the relevant checks, and confirm the deployment succeeded.”

Memory Trick

Stage → Logs → Cause → Fix → Validate.

Q14. How would you implement automatic rollback after a failed deployment?

Simple Concept

Automatic rollback returns an application to a known-good version when defined deployment health criteria fail.

Why Do We Need It?

It can reduce the time a faulty release affects users.

How Does It Work?

Deploy a new version, evaluate health checks and service metrics, stop or reverse the rollout when conditions fail, and verify recovery.

Production Example

A Kubernetes deployment introduces a broken application version. The rollout is halted and the previous Deployment revision is restored.

Command / YAML

kubectl rollout status deployment/my-app
kubectl rollout history deployment/my-app
kubectl rollout undo deployment/my-app

Common Interview Trap

Assuming application rollback also reverses database schema changes or external side effects.

Candidate-Style Answer

“I would define measurable health criteria, monitor the rollout, and trigger rollback when the release fails those criteria. I would verify the previous version is healthy and ensure database changes remain backward compatible.”

Memory Trick

Deploy → Measure → Detect Failure → Roll Back → Verify.

Q15. Your CI/CD pipeline takes too long. How would you optimize it?

Simple Concept

Pipeline optimization reduces execution time while preserving quality and security.

Why Do We Need It?

Slow pipelines delay feedback and increase delivery lead time.

How Does It Work?

Measure stage durations, parallelize independent tasks, cache dependencies, optimize Docker layers, and eliminate redundant work.

Production Example

Unit tests and lint checks run sequentially despite being independent. Running them in parallel reduces total pipeline duration.

Command / YAML

pipeline {
    agent any
    stages {
        stage(’Checks’) {
            parallel {
                stage(’Tests’) {
                    steps {
                        sh ‘./run-tests.sh’
                    }
                }
                stage(’Lint’) {
                    steps {
                        sh ‘./run-lint.sh’
                    }
                }
            }
        }
    }
}

Common Interview Trap

Removing important tests or security scans just to make the pipeline faster.

Candidate-Style Answer

“I would measure stage duration, identify the bottlenecks, and optimize independent stages through parallelism and caching. I would compare before-and-after execution times while preserving test coverage and required security controls.”

Memory Trick

Measure → Optimize → Benchmark → Validate.

Q16. How would you manage secrets securely in CI/CD pipelines?

Simple Concept

Secrets management protects passwords, API keys, tokens, and certificates from unauthorized access.

Why Do We Need It?

Leaked credentials can expose cloud infrastructure, source code, databases, and production systems.

How Does It Work?

Use an approved secret manager, short-lived credentials where possible, least-privilege access, masked logs, and controlled secret rotation.

Production Example

A pipeline uses an AWS role to access ECR rather than storing long-lived AWS access keys directly in source code.

Command / YAML

aws sts get-caller-identity
aws secretsmanager describe-secret \
  --secret-id production/database

Common Interview Trap

Believing that masked CI variables alone eliminate secret exposure risks.

Candidate-Style Answer

“I centralize secrets in an approved manager, restrict access, and use temporary credentials where possible. I prevent secrets from entering source code, images, and logs, and I implement rotation and auditing.”

Memory Trick

Store Securely → Restrict → Deliver Safely → Rotate → Audit.

Section 5: Production and Incident Management

Q17. An application has high response time. How would you find the root cause?

Simple Concept

Latency troubleshooting identifies the component delaying requests.

Why Do We Need It?

Slow applications can affect customers even when servers and pods appear healthy.

How Does It Work?

Examine latency percentiles, error rates, traces, application logs, resource utilization, and downstream service performance.

Production Example

The API becomes slow because database queries take longer after a change in query execution plans.

Command / YAML

kubectl top pods -A
kubectl top nodes
kubectl logs deployment/my-app --since=15m
kubectl get events -A --sort-by=.lastTimestamp

Common Interview Trap

Scaling application replicas without checking whether the database or another dependency is the actual bottleneck.

Candidate-Style Answer

“I would establish the scope of the latency and correlate metrics, logs, and traces. I would investigate application resources, database queries, network behavior, and recent deployments, then fix the confirmed bottleneck and validate response-time improvements.”

Memory Trick

Measure → Trace → Isolate → Fix → Verify.

Q18. Your production application is completely down. How do you handle the incident end to end?

Simple Concept

Incident management restores service safely, coordinates responders, and reduces the impact on users.

Why Do We Need It?

A structured response reduces confusion, limits downtime, and preserves evidence for investigation.

How Does It Work?

Assess impact, declare and communicate the incident, collect evidence, restore service, verify recovery, and conduct a blameless post-incident review.

Production Example

A deployment causes widespread failures. The team stops further releases, investigates the issue, rolls back to a healthy version, and verifies that customer traffic recovers.

Command / YAML

kubectl get pods -A
kubectl get events -A --sort-by=.lastTimestamp
kubectl rollout history deployment/my-app
kubectl rollout undo deployment/my-app

Common Interview Trap

Focusing on finding someone to blame instead of restoring service and understanding the system failure.

Candidate-Style Answer

“I would assess the impact, notify the relevant stakeholders, and establish a clear incident timeline. I would use metrics and logs to isolate the cause, restore service through the safest recovery option, verify recovery, and document preventive actions.”

Memory Trick

Impact → Communicate → Investigate → Recover → Verify → Learn.

Q19. Your monitoring system generates too many alerts. How would you reduce alert fatigue?

Simple Concept

Alert fatigue occurs when excessive or low-value alerts make it difficult to recognize important incidents.

Why Do We Need It?

Engineers may miss critical alerts when noisy notifications become routine.

How Does It Work?

Review alert history, remove duplicates, tune thresholds, group related alerts, and alert on meaningful symptoms tied to user impact or service-level objectives.

Production Example

CPU alerts trigger repeatedly during normal traffic spikes. The team evaluates latency and saturation together and adjusts alerts to distinguish expected load from genuine service risk.

Command / YAML

Example Prometheus alert rule:

groups:
  - name: application
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~”5..”}[5m]) > 1
        for: 10m
        labels:
          severity: warning

This example is illustrative. The threshold must match the application’s traffic, metric labels, and service objectives.

Common Interview Trap

Disabling alerts without understanding which operational risks they protect against.

Candidate-Style Answer

“I would analyze alert frequency, actionability, and false positives. I would remove duplicates, group related notifications, tune thresholds using service objectives, and ensure critical alerts identify a clear response action.”

Memory Trick

Review → Tune → Group → Prioritize → Measure.

Q20. How would you perform Root Cause Analysis after a production incident?

Simple Concept

Root Cause Analysis identifies the technical and process conditions that contributed to an incident.

Why Do We Need It?

Without a proper review, teams may fix symptoms while leaving the conditions that caused the incident unchanged.

How Does It Work?

Build a timeline, collect evidence, identify contributing factors, distinguish root causes from symptoms, and assign measurable corrective actions.

Production Example

A memory-related outage is traced to a new feature that retains unnecessary data. The team fixes the implementation and adds realistic load testing and memory monitoring.

Command / YAML

kubectl logs deployment/my-app --since=1h
kubectl get events -A --sort-by=.lastTimestamp
git log --oneline -10

Common Interview Trap

Writing an RCA that blames an individual or lists only the immediate trigger without addressing systemic weaknesses.

Candidate-Style Answer

“I would establish a timeline using logs, metrics, and deployment history. I would identify the direct cause and contributing factors, then assign corrective actions with owners and deadlines. The review would focus on improving the system rather than blaming individuals.”

Memory Trick

Timeline → Evidence → Causes → Actions → Follow-up.

Section 6: DevOps Best Practices

Q21. What DevOps best practices do you follow to build reliable, scalable, and secure systems?

Simple Concept

DevOps best practices combine automation, collaboration, security, observability, and controlled delivery to improve software reliability.

Why Do We Need It?

Consistent engineering practices reduce manual errors, improve recovery, and make systems easier to operate at scale.

How Does It Work?

Manage infrastructure as code, automate testing and deployment, enforce least privilege, monitor service health, maintain backups, and test recovery procedures.

Production Example

A team uses Terraform for infrastructure, CI/CD for validated releases, Kubernetes for orchestration, Prometheus and Grafana for observability, and controlled rollback procedures for recovery.

Command / YAML

terraform plan
kubectl rollout status deployment/my-app
kubectl top pods -A

Common Interview Trap

Listing tools without explaining the operational problem each practice solves.

Candidate-Style Answer

“I focus on Infrastructure as Code, automated CI/CD, least-privilege security, monitoring, backup and disaster recovery, and tested rollback strategies. I also document operational procedures and review incidents to improve reliability continuously.”

Memory Trick

Automate → Secure → Observe → Recover → Improve.

The Production Interview Formula

For almost any scenario-based DevOps question, follow this sequence:

Understand the Problem → Investigate → Identify the Root Cause → Resolve → Validate → Prevent Recurrence

When answering, explain what you would check first, why that evidence matters, which commands you would use, and how you would verify the result.

Don’t rush to restart services, delete resources, or change permissions before understanding the failure.

Final Thoughts

Senior-level DevOps interviews test practical engineering judgment as much as technical knowledge.

The strongest candidates know how to investigate uncertainty, use evidence, recover safely, and improve systems after failures.

Don’t just memorize commands. Understand what they reveal, what decisions they support, and what risks they introduce.

Think like a production engineer. Troubleshoot systematically. Resolve safely. Prevent recurrence.

About the Author

Arvind Verma

Explore practical DevOps learning resources:

https://topmate.io/arvindverma021/

Follow on LinkedIn:

https://www.linkedin.com/in/arvindverma021/

Read more on Medium:

https://medium.com/@arvindverma021

Subscribe on Substack:

Hashtags

#DevOps #DevOpsInterview #AWS #Kubernetes #Docker #Terraform #Jenkins #CICD #Linux #Networking #DevSecOps #SRE #CloudEngineering #PlatformEngineering #ProductionTroubleshooting

Arvind Verma is free today. But if you enjoyed this post, you can tell Arvind Verma that their writing is valuable by pledging a future subscription. You won't be charged unless they enable payments.

Pledge your support

© 2026 Arvind Verma
548 Market Street PMB 72296, San Francisco, CA 94104