mailwal

🚨 9 Things I Wish Someone Had Told Me Before Becoming a DevOps Engineer

Arvind Verma

🚨 9 Things I Wish Someone Had Told Me Before Becoming a DevOps Engineer

Learn the tools. Understand the systems. Master the art of troubleshooting.

When you’re starting your DevOps journey, everything looks exciting.

You learn Linux, Docker, Kubernetes, Jenkins, Terraform, AWS, Ansible, Prometheus, Grafana, and dozens of other tools.

You build CI/CD pipelines, deploy applications, write YAML files, and automate infrastructure.

You start feeling confident because you know how to use the tools.

Then production breaks.

An application becomes unavailable. Kubernetes reports healthy Pods, but users cannot access the application. The deployment succeeds, but checkout stops working. CPU utilization looks normal, yet response times suddenly increase.

That’s when you realize something important:

Knowing DevOps tools and understanding production systems are two completely different skills.

Here are nine lessons I wish someone had explained to me before I started my DevOps journey.

1. Linux Matters More Than Your 14th DevOps Tool

You can learn another CI/CD tool, another Kubernetes add-on, or another automation framework.

But if you don’t understand Linux, troubleshooting will eventually become difficult.

Linux is everywhere in DevOps:

  • Application servers

  • Docker containers

  • Kubernetes nodes

  • Cloud virtual machines

  • CI/CD runners

  • Monitoring systems

  • Production infrastructure

Imagine a production application suddenly stops responding.

Do you know how to investigate running processes, memory usage, disk utilization, permissions, services, and system logs?

These commands should become second nature:

top
ps aux
free -m
df -h
du -sh *
systemctl status nginx
journalctl -xe
ss -tulnp

The real lesson: Don’t just memorize Linux commands. Understand what the operating system is doing when something goes wrong.

2. Networking Will Humble You Eventually

You can deploy an application successfully and still have users unable to access it.

Why?

Because deployment and connectivity are different problems.

A DevOps engineer must understand:

  • DNS resolution

  • TCP/IP

  • HTTP and HTTPS

  • Ports and protocols

  • Routing and subnets

  • Security groups and firewalls

  • Load balancers

  • Reverse proxies

  • Network address translation

Production example:

Your application is running inside Kubernetes, and the Pods are healthy. However, users receive HTTP 502 errors.

Where do you investigate?

Start with the request path:

User → DNS → Load Balancer → Ingress → Kubernetes Service → Pod → Application

Check DNS resolution, backend health, service endpoints, network connectivity, and application logs.

Useful commands:

nslookup example.com
curl -I https://example.com
ss -tulnp
traceroute example.com

The real lesson: A healthy application does not automatically mean a reachable application. Understand the complete network path.

3. Kubernetes Doesn’t Eliminate Distributed-Systems Problems

Kubernetes makes container orchestration easier.

It does not magically solve application architecture, network failures, dependency problems, or distributed-systems complexity.

You can have:

  • Healthy Pods but failing requests.

  • Running containers but unavailable dependencies.

  • Successful deployments but broken service communication.

  • Multiple replicas but a shared database bottleneck.

  • Autoscaling enabled but insufficient downstream capacity.

Production example:

Your frontend application has five healthy replicas. However, every request depends on a database that is running out of available connections.

Adding more frontend Pods may increase database pressure instead of fixing the issue.

You need to investigate the entire application dependency chain.

Start with:

kubectl get pods -A
kubectl get services -A
kubectl get endpoints -A
kubectl describe pod <pod-name>
kubectl logs <pod-name>
kubectl get events -A

The real lesson: Kubernetes manages workloads. You still need to understand application dependencies, networking, resilience, and failure propagation.

4. Monitoring Isn’t Observability

Monitoring tells you whether predefined conditions are healthy or unhealthy.

Observability helps you investigate why a system behaves the way it does.

They are related, but they are not interchangeable.

A production observability strategy commonly combines:

Metrics: What is happening?

Logs: What happened?

Traces: Where did a request spend its time?

For example, Grafana may show that application latency increased from 200 milliseconds to 3 seconds.

That’s useful, but it doesn’t explain the root cause.

You may need:

  • Prometheus metrics to identify resource or performance patterns.

  • Application logs to investigate errors.

  • Distributed traces to locate slow dependencies.

  • Dashboards to correlate the evidence.

The real lesson: Don’t just create dashboards. Build the ability to investigate system behavior using meaningful telemetry.

5. Automation Can Automate Disasters Too

Automation is one of the biggest advantages of DevOps.

But automation without validation can make mistakes happen faster and across more systems.

Imagine a Terraform pipeline that accidentally removes an important production resource.

Or a cleanup script that deletes files from the wrong directory.

Or a deployment pipeline that pushes an untested image to every environment.

The problem isn’t automation itself. The problem is uncontrolled automation.

Build safeguards into your workflows:

  • Input validation

  • Code reviews

  • Automated testing

  • Terraform plan reviews

  • Approval gates

  • Least-privilege permissions

  • Deployment health checks

  • Rollback procedures

For Terraform:

terraform fmt -check
terraform validate
terraform plan -out=tfplan
terraform apply tfplan

Review the plan before applying changes, especially in production.

The real lesson: Good automation reduces human error. Good engineering ensures automation doesn’t multiply the impact of an error.

6. The Database Is Part of Your Infrastructure Problem

DevOps engineers sometimes focus heavily on compute, containers, networking, and deployment pipelines.

But the database is often one of the most critical dependencies in a production system.

Your application may be healthy while the database is struggling.

Common problems include:

  • Connection pool exhaustion

  • Slow queries

  • Lock contention

  • High database CPU

  • Storage exhaustion

  • Replication lag

  • Backup failures

  • Connection timeouts

Production example:

An application suddenly becomes slow after a new release.

Kubernetes Pods are healthy. CPU and memory utilization look normal.

After investigating, you discover that a new application feature introduced inefficient database queries.

Restarting Pods won’t solve the underlying problem.

You need to investigate database metrics, query performance, connection pools, and application behavior.

The real lesson: Understand the dependencies behind your applications. Infrastructure reliability includes databases, storage, queues, and external services.

7. A Rollback Strategy Matters More Than a Beautiful Deployment Pipeline

A successful deployment pipeline is important.

But what happens when a deployment succeeds technically and breaks the application?

Can you recover quickly?

Every production deployment should consider:

  • Previous stable application versions

  • Image tagging and artifact retention

  • Database migration compatibility

  • Configuration rollback

  • Deployment health checks

  • Traffic management

  • Recovery procedures

Production example:

A new application release introduces a serious performance regression.

Your deployment strategy should allow you to return traffic to a previously validated version.

For Kubernetes, a basic Deployment rollback can be performed using:

kubectl rollout history deployment/my-app

kubectl rollout undo deployment/my-app

kubectl rollout status deployment/my-app

However, rolling back an application does not automatically reverse database schema changes or external infrastructure modifications.

Those require separate recovery planning.

The real lesson: A deployment isn’t truly production-ready until you understand how to recover when it goes wrong.

8. The Best Incident Responders Aren’t Command Encyclopedias

During production incidents, knowing hundreds of commands is useful.

But knowing which command to run next is even more important.

Strong incident responders reduce the search space faster.

They don’t randomly restart services, delete Pods, or change configurations without understanding the evidence.

They follow a structured process:

Understand Impact → Collect Evidence → Form a Hypothesis → Isolate Root Cause → Recover → Verify → Prevent Recurrence

Production example:

Your application starts returning HTTP 503 errors.

Instead of immediately restarting Kubernetes Pods:

  1. Identify the affected application and users.

  2. Check recent deployments and infrastructure changes.

  3. Inspect Pod health and application logs.

  4. Verify Service endpoints and Ingress configuration.

  5. Check load balancer health.

  6. Investigate downstream dependencies.

  7. Apply the safest recovery action.

  8. Confirm that service availability has returned.

The real lesson: Troubleshooting is not about executing more commands. It is about asking better questions and eliminating incorrect assumptions.

9. Nobody Cares How Beautiful Your YAML Is When Checkout Is Down

You can write perfectly structured Kubernetes manifests.

You can create reusable Helm charts.

You can build impressive Jenkins pipelines.

You can automate almost everything.

But if customers cannot complete their purchases, your engineering work has not achieved its most important objective.

Production engineering ultimately comes down to delivering reliable services.

Think about what matters to the business:

  • Application availability

  • Successful transactions

  • Response times

  • Data integrity

  • Security

  • Recovery time

  • Customer experience

Production example:

Your deployment pipeline reports SUCCESS.

Every Kubernetes Pod is Running.

Grafana dashboards show normal CPU and memory.

But customers cannot complete checkout because a payment service integration is failing.

Technically, the deployment succeeded.

Operationally, the business is experiencing an incident.

This is why DevOps engineers must understand application behavior and business impact, not just infrastructure status.

The real lesson: Infrastructure exists to support applications, and applications exist to deliver value. Never confuse a successful deployment with a successful outcome.

What Should You Focus on Instead?

If you’re building a DevOps career, don’t spend all your time collecting tools.

Develop these engineering capabilities:

Master the fundamentals.

Linux, networking, Git, operating systems, scripting, and databases.

Understand the complete architecture.

Learn how applications, infrastructure, networks, storage, and cloud services interact.

Build production-style projects.

Combine Terraform, AWS, Docker, Kubernetes, CI/CD, monitoring, and security.

Practise failure scenarios.

Simulate broken deployments, network failures, resource exhaustion, database issues, and configuration drift.

Develop incident-response skills.

Learn to investigate, communicate, recover, and prevent recurring failures.

Understand business impact.

Know why reliability, performance, security, and availability matter to the people using your systems.

Final Thoughts: Think Beyond YAML

DevOps is not about knowing the maximum number of tools.

It’s about understanding how systems behave, how failures happen, and how to build reliable solutions.

Linux helps you understand the operating system.

Networking helps you understand connectivity.

Kubernetes helps you orchestrate workloads.

Monitoring and observability help you investigate behavior.

Automation helps you improve consistency.

Databases and distributed systems help you understand application dependencies.

Rollback strategies and incident response help you recover when things go wrong.

Learn the tools. Build the projects. Practise the commands.

But learn why systems fail even harder.

Because the best DevOps engineers aren’t the ones who can write the most YAML. They’re the ones who can keep production running when everything starts breaking.

Think Beyond YAML. Build Beyond Tutorials. Engineer for Production.

Connect With Me

I share practical insights on DevOps, AWS, Kubernetes, cloud engineering, automation, production troubleshooting, and career development.

LinkedIn: https://www.linkedin.com/in/arvindverma021/

Medium: https://medium.com/@arvindverma021

Substack:

DevOps Learning Resources: https://topmate.io/arvindverma021/

Hashtags

#DevOps #ThinkBeyondYAML #Linux #Networking #Kubernetes #Docker #Terraform #AWS #SRE #Observability #Monitoring #ProductionEngineering #IncidentManagement #CICD #DevSecOps #PlatformEngineering #CloudComputing #Automation #Troubleshooting #DevOpsCareer

Arvind Verma is free today. But if you enjoyed this post, you can tell Arvind Verma that their writing is valuable by pledging a future subscription. You won't be charged unless they enable payments.

Pledge your support

Š 2026 Arvind Verma
548 Market Street PMB 72296, San Francisco, CA 94104