mailwal

ServerLabs Digest: November 2, 2026

Ivan Velichko

Practical server-side learning in your inbox • Feed cutoff: November 2, 2026


Skill Path: Troubleshoot Kubernetes Pods with kubectl debug

By Ivan Velichko • Published on November 1, 2026

Troubleshooting a running Kubernetes Pod isn't always as simple as kubectl exec. Many production containers have no shell, lack common debugging tools, or run as a non-root user with limited permissions. Ephemeral containers offer a way around these limitations: using kubectl debug, you can add a container with its own tools to a running Pod without restarting the application.

However, adding an ephemeral container doesn't automatically give you access to the processes and files of other containers in the Pod. Kubernetes security policies may also restrict what the debug container can do or prevent it from starting altogether.…

Troubleshoot Kubernetes Pods with kubectl debug

Challenge: Edit a File in a Running Kubernetes Pod: a Non-Root Container Case

By Ivan Velichko • Published on November 1, 2026

The Pod in the default namespace runs a simple web server that serves static files from the /var/www/html directory on port 8080. Unlike in the Simple Case of this challenge, the container runs as a non-root user, and the files in /var/www/html belong to root.

Your task is to edit the /var/www/html/index.html file in this container without restarting the app container or recreating the Pod:

  • Replace the phrase "Hello World" with the phrase "Hello Labs".
  • Add the phrase "Practice for the win!" anywhere in the file.

You can change resources only in the default namespace. You may change the…

Edit a File in a Running Kubernetes Pod: a Non-Root Container Case

Challenge: Edit a File in a Running Kubernetes Pod: a Distroless Container Case

By Ivan Velichko • Published on November 1, 2026

The Pod in the default namespace runs the same web server as in the Non-Root Container Case of this challenge, but this time from a distroless image. The image has no shell, no tar, and no other tools, and the server runs as a non-root user that owns the files in the /var/www/html directory. On top of that, the default namespace enforces the baseline level of the Pod Security Standards.

Your task is to edit the /var/www/html/index.html file in this container without restarting the app container or recreating the Pod:

  • Replace the phrase "Hello World" with the phrase…
Edit a File in a Running Kubernetes Pod: a Distroless Container Case

Challenge: Use a Kubernetes Job to Test NATS Pub/Sub

By Omkar Shelke • Published on November 1, 2026

Scenario

NATS is a messaging system. Applications use it to send messages to each other without connecting to each other directly. It works with publish/subscribe:

  • A publisher sends a message to a subject, a name such as payments.requests.
  • A subscriber listens on a subject and receives every message sent to it.
  • The NATS server sits between them. Clients connect to it on port 4222, and it delivers each message to every subscriber of the subject.

A NATS server already runs in this cluster.

More about NATS:

NATS is not…

Use a Kubernetes Job to Test NATS Pub/Sub

Challenge: Reach a Linux Service Through an Interface It Doesn't Listen On

By Ivan Velichko • Published on October 31, 2026

In 2002, a surprising Linux networking behavior was documented: on a multi-homed machine, a service listening on the IP address assigned to eth1 could still be reached through eth0, even with IP forwarding disabled.

This happens because Linux follows the weak end host model (described in Section 3.3.4.2 of RFC 1122). The kernel treats IP addresses as belonging to the host as a whole, not exclusively to individual network interfaces. As a result, binding a service to an IP address restricts which destination address it accepts connections on, but not which interface those connections can arrive through.

…

Reach a Linux Service Through an Interface It Doesn't Listen On

Shell Gym: Kubernetes Gym: Managing Container Images

By Kaloyan Preslavski • Published on October 31, 2026

This gym trains working with built container images: tags, containers, registries, image archives, and private images in Kubernetes.

Kubernetes Gym: Managing Container Images

Shell Gym: Kubernetes Gym: Building Container Images

By Kaloyan Preslavski • Published on October 31, 2026

This gym trains writing Dockerfiles and building images with docker build.

Kubernetes Gym: Building Container Images

Challenge: Fix a Deployment Stuck in ImagePullBackOff

By Omkar Shelke • Published on October 29, 2026

Scenario

The catalog Deployment in the shop namespace was just rolled out, and its Pods do not start. From cplane-01:

kubectl get pods -n shop
NAME                       READY   STATUS             RESTARTS   AGE
catalog-647b446c54-6l8bz   0/1     ImagePullBackOff   0          43s
catalog-647b446c54-jkmt5   0/1     ImagePullBackOff   0          43s

The catalog page at http://catalog.shop.local returns 503.

The team keeps the manifests and its notes on cplane-01:

/home/laborant/shop/
├── README.md
├── catalog-deployment.yaml
├── ingress.yaml             # do not change
├── namespace.yaml           # do not change
└── service.yaml             # do not change

Task

Find out why the catalog Pods do not start, and get them running so that the…

Fix a Deployment Stuck in ImagePullBackOff

Blog Post: How I added dynamic social preview images to my site for almost nothing

By Ivan Velichko • Published on October 27, 2026

Most people probably take this functionality for granted, but social preview images (og:image) for a site with dynamic content aren't trivial to build.

For quite some time, I had been putting off building them for iximiuz Labs, but adding author profiles, and then learner profiles, finally made me reconsider. After all, people would want their profile links to look nice when shared on social media.

Services that screenshot web pages certainly exist, but I've always been wary of unnecessary dependencies, especially if they want you to pay as you go. At the same time, running my own headless…

How I added dynamic social preview images to my site for almost nothing

Challenge: Connect a Container to a Service Bound to Localhost on the Docker Host

By Ivan Velichko • Published on October 25, 2026

This challenge focuses on Docker Engine running directly on a Linux machine. Docker Desktop runs Docker Engine inside a separate VM and provides some of the host connectivity out of the box, so the commands from this challenge may behave differently on your computer.

This challenge continues the story of Connect a Container to a Service Running on the Docker Host, and it assumes that you know the host-gateway trick from there.

Your company is moving its services into containers, and your team works on the cart service, which keeps the shoppers' carts in Redis. Historically, the development version…

Connect a Container to a Service Bound to Localhost on the Docker Host

Challenge: Connect a Container to a Service Running on the Docker Host

By Ivan Velichko • Published on October 25, 2026

This challenge focuses on Docker Engine running directly on a Linux machine. Docker Desktop runs Docker Engine inside a separate VM and provides some of the host connectivity out of the box, so the commands from this challenge may behave differently on your computer.

Your company is moving its services into containers, and your team works on the shop service. Historically, the development version of the shop service and all its dependencies have run on a single VM and talked to each other over localhost.

You have a developer workstation for experiments called dev-01. The team has packaged…

Connect a Container to a Service Running on the Docker Host

Happy hacking!

Ivan