🐮 DevOpsLinks | How Cloudflare cut its inference bill by 68%

Release night shouldn't need a prayer.

Same app, same people. One team ships once a month and holds its breath. The other ships every day and breaks less.

How CI/CD works is a short comic that shows why, from merge-day chaos to feature flags and rollbacks.

Read episode 1

Railway, the app hosting platform, disclosed that an attacker decrypted stored environment variables in at most 1.85% of its workspaces, about 100,000. The attacker gained higher access inside the build infrastructure, then took credentials that could decrypt stored variables, likely to collect AI provider keys. Railway cannot tell which keys were read, so every customer who signed up before 21 October should rotate secrets; all Railway access tokens were revoked. Some providers, GitHub among them, received every key Railway stored, so a revocation email alone does not prove a leak.

HashiCorp released Terraform 1.17.0, where variables and locals can set a provider's source and version in the provider requirements block. The variable must be declared with const = true and cannot depend on values computed during a plan, but it can still come from the command line, a variables file or an environment variable. A new -minimal-refresh plan option re-reads only the resources with proposed changes. Terraform Policy, the rule checks on plans, is now generally available and no longer needs an experimental build.

Quad9, the non-profit public DNS resolver at 9.9.9.9, will stop serving users in France on 24 November 2026. A French court ordered it to block domains that broadcaster beIN ties to pirated sports streams; Quad9 has appealed and faces a penalty of 1,000 euros per day. It refuses both a worldwide block and geoIP filtering (guessing location from an IP address), and says about 20 similar cases are pending in France. French networks will lose the whole service, malware and phishing blocking included, so users there have 30 days to switch resolvers.

A unit file that reads fine can still start your service before the network is up, or leave it dead after a crash. This quiz puts you in front of real systemd unit settings and asks what systemd actually does with them: how dependencies differ from ordering, when a restart policy kicks in, and what happens once the start limit is reached. Each answer explains the rule from the systemd documentation, so the next boot-time failure on your servers takes minutes to trace.

Jeremy Schneider shows how recovery_prefetch, the PostgreSQL setting that makes a replica read data ahead while it replays changes, can stop all writes on a primary that uses synchronous replication. When the primary does large sequential I/O, the setting turns it into 8 KB random reads on the replica, up to 16 times the IOPS. If a network glitch then breaks streaming, the lagging standby stays in WAL-restore mode until it catches up, and the primary waits: his reproduction on a 500 IOPS cloud disk hung writes for over 30 minutes. With recovery_prefetch turned off, the same test stayed caught up.

Datadog explains how it ranks thousands of internal services by how much an outage would hurt customers, instead of treating them all alike. The first signal came from an LLM that read several hundred SEV-1 and SEV-2 postmortems in under a day, for about $32, and named which services caused or suffered each incident. Traces, traffic and paging monitors are combined with it, and near-zero traffic can only lower a rating, never raise it. The result, served from one internal API with owner overrides, decides how fast a service deploys across data centers and whether a dependency update can auto-merge.

Cloudflare cut the cost of its internal AI code reviewer from $2.43 to $0.77 per review, and the median run dropped from 4m 09s to 1m 32s. The biggest single fix was in the prompt cache: the first request of a session found a cached prefix only 14% to 26% of the time, because the shared prompt and repository-specific details sat behind one cache breakpoint. Model and effort changes gave the rest, and Sonnet 5.5 reviewers cost up to 62% less per run by using fewer tokens. Fewer findings are posted now, but authors act on 59% of serious ones, up from 34%.

The AWS European Sovereign Cloud is a separate partition, not an EU Region with extra controls: it has its own IAM, Organizations, billing in euro, console and arn:aws-eusc: ARNs, and existing accounts do not carry over. Roles cannot be assumed across the boundary, and peering, Transit Gateway and S3 replication do not cross it, so a hybrid setup means two landing zones and partition-aware infrastructure code. DynamoDB Global Tables and CloudHSM are not coming, Bedrock has no Claude models, and Datadog's AWS integration does not accept the partition yet. In dollar terms, prices run about 8.5% to 18% higher than Frankfurt.

ON DELETE CASCADE in PostgreSQL turns one customer deletion into a single large transaction that holds locks on every table in the tree. A routine migration that needs an exclusive lock then queues behind it, and every new query on that table piles up until the delete ends; one cascade can also write hundreds of gigabytes of WAL. With logical replication, subscribers skip cascades, so child rows that exist only on a subscriber are left orphaned without an error. The proposed alternative is soft deletes: mark one row deleted, then remove the real rows later in small batches.

A short post on writing a container runtime for fun calls Docker weird: containers are ordinary processes, images are folders, and runc, written in Go, needs a C module and a re-exec trick because Go has no usable fork. Hacker News commenters answer that none of this is weird, and argue that Docker's real contribution was the Dockerfile, layer caching and a shared image registry rather than isolation. Others point to rootless Podman, with no daemon and no root, and to bocker, a Docker clone in about 100 lines of bash. Solomon Hykes, Docker's creator, answers questions in the thread, from Docker's early days to its relations with Red Hat.

DevOpsDays Barcelona 2026 runs on 13 and 14 November 2026 in Barcelona, Spain, organized by a local volunteer team. Both days combine a talk track, with invited speakers and talks from an open call, and open spaces, where attendees suggest topics and the room picks the ones that run. Recurring subjects are delivery pipelines, infrastructure automation, cloud and Kubernetes operations, observability, incident response and platform engineering.

X-Ops Conference runs on 20 and 21 November 2026 at Universidad Rey Juan Carlos, Campus Móstoles, near Madrid. The program groups DevOps, DevSecOps, AIOps, MLOps and platform engineering under one label, with talks built on implementation case studies plus hands-on workshops. Topics include security test automation, CI/CD, monitoring and secure architecture. It is aimed at engineers, SREs, platform engineers and tech leads, and the organizers report more than 500 attendees for the Madrid edition.

DevOpsDays Warsaw returns on 23 and 24 November 2026 in Warsaw, Poland. Like every edition of the series that began in Ghent in 2009, it is run by a local volunteer team. The two days pair a talk track, drawn from invited speakers and an open call for papers, with open space sessions where attendees propose, vote on and lead the discussions. Recurring subjects are CI/CD, infrastructure as code, monitoring, incident response and how dev and ops teams share responsibility for running systems.

Krkn, a CNCF sandbox project started by Red Hat's chaos engineering team, runs chaos scenarios against a Kubernetes or OpenShift cluster from outside it. It kills pods and containers, fails nodes, simulates power and zone outages on major clouds and bare metal, and injects network chaos and time skew. During a run it watches cluster health and Prometheus alerts, so each scenario ends with a pass or a fail. Node and zone scenarios need cloud credentials that can stop real machines, and faults inside a single application process are outside its scope.

GoAlert is an open source on-call tool from Target, the US retailer, covering rotations, schedules and escalation policies. A monitoring system such as Grafana or Prometheus creates an alert through an integration key, and GoAlert pages the person on call by SMS, voice, email or Slack, then moves up the escalation policy until someone acknowledges. It is self-hosted only: a single binary or container image plus a PostgreSQL database. It stops at paging, so incident timelines, retrospectives and status pages come from other tools.

Chaos Toolkit is a Python CLI that runs chaos engineering experiments written as one JSON or YAML file. Each file states a steady-state hypothesis, a method of probes and disrupting actions, and optional rollbacks; chaos run checks the hypothesis, runs the method, then checks the hypothesis again. Drivers for AWS, Azure, Kubernetes, Istio, Kafka, Prometheus, OpenTelemetry and others come as Python extensions, and as a plain CLI it fits in a CI/CD job. It has no web console or scheduler of its own and runs experiments only when called.

Costory pulls billing from AWS, Google Cloud and Azure plus usage-priced services such as Datadog, OpenAI and Anthropic into one model, then allocates spend to teams, services and environments. It correlates cost changes with GitHub deploys, price changes and traffic, and exposes the result through an MCP server, so an assistant in Claude Code, Cursor or VS Code can explain why a line of the bill moved. Pricing is a flat monthly fee, not a share of spend, starting at 250 euros a month. The service is closed source.

Fortio is a Go load testing tool that holds a fixed query rate and reports latency percentiles from a real histogram, not a running average. The same binary runs a web UI for comparing runs and an echo server that can inject latency or error codes, which is why it was built inside Istio to measure proxy hops and service mesh sidecars. It speaks HTTP/1.1, HTTP/2, gRPC and raw TCP. Fortio has no threshold check that fails a build on a missed percentile; results are saved as JSON files.

Did you know that Nagios, the open source monitoring and alerting system, began in 1996 as a small MS-DOS program that pinged Novell NetWare servers and sent numeric pages? According to the project's own history, Ethan Galstad released the Linux rewrite as open source in 1999 under the name "NetSaint", guessing that about a dozen people might want it. Trademark trouble led to a new name in 2002, a recursive acronym: "Nagios Ain't Gonna Insist On Sainthood".

"The basic thesis of this article is that organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations."

Melvin Conway, "How Do Committees Invent?", Datamation (April 1968), the line now known as Conway's law.

Load-bearing since 1983.