mailwal

Data Engineering Digest, October 2026

Data Engineering Community

Empowering data engineers through shared knowledge. Join the largest data engineering community in the world today.

Hello Data Engineers,

Welcome back to another edition of the data engineering digest - a monthly newsletter containing updates, inspiration, and insights from the community.

Thanks for reading the Data Engineering Digest! Subscribe for free to receive new posts and support the community.

Here are a few things that happened this month in the community:

  • Are we building semantic layers for chatbots now?

  • Why doesn’t the business care if the data is accurate?

  • Should transformations live in Python or SQL?

  • DuckDB 2.0 and Polars 2.0 ship with a few upgrade traps

  • A database breach and a poisoned Python package

  • Are AI coding tools breaking database benchmarks?

Community Discussions

Here are the top posts you may have missed:

1. My work has shifted from building pipelines to building semantic layers for chatbots

One data engineer shared how their job has moved from building pipelines to building semantic layers for chatbots. As companies put AI chatbots on top of the warehouse, more of our time goes into semantic layers (the shared, written-down definitions of metrics and business terms that sit between the raw tables and whoever is asking questions).

If your company is rolling out an AI assistant on its data, this work is probably headed your way if it hasn’t happened already.

💡 Key Insight

Some data engineers find it a lot less fun than building pipelines, while others find building these “context engines” is more interesting than pipeline work ever was.

As one data engineer put it, “We have always needed a semantic layer, it was just always ignored. Turns out the bot needs the same thing the people do.” Metric definitions that used to live in someone’s head (or in five slightly different dashboards) now have to be written down, because an AI agent can’t lean over and ask what “active user” means. If you’ve been asking for a semantic layer for years, the chatbot might be the incentive to finally get the investment.

📊 Related: Pinterest wrote up how they built a metrics layer that works for both people and AI agents, where each metric is defined once in Git and served to dashboards and agents from the same place.

2. Why does accuracy not seem to matter for the business?

Companies have huge budgets for their data teams to support business decisions, yet most of our output is never used to make a difference. Business seems to care more about a KPI 'looking good' than being actually correct, if they care at all.

Have you ever spent days getting a number exactly right, only to realize nobody would have noticed if it was a little off?

💡 Key Insight

To borrow a line from Douglas Adams, a lot of businesses would rather “set the record, if not straight, then at least firmly crooked.” Consistency usually beats accuracy, because a number that matches last month’s report is easier to trust than a “more correct” one that suddenly changes. A few other things are worth keeping in mind:

  • Plenty of decisions only need to be roughly right. If 80% accuracy points to the same choice as 99%, the extra work doesn’t change anything.

  • Many KPIs (key performance indicators) end up in a manager’s slide deck, so they get judged by how they look more than how they were calculated.

  • Customer-facing analytics is the exception. When customers see the numbers (or get billed by them), accuracy matters a lot, and that’s where companies will actually pay for it.

POLL
How often does your work change a real decision?

3. Data engineer insists on doing all transformations in Python instead of SQL

Is writing every transformation in Python actually better than SQL? A data engineer asked for help with a teammate who insists on writing every transformation in PySpark (the Python way to write Spark jobs), convinced it beats SQL.

This comes up on a lot of teams where engineers and analysts share the same tables, and it tends to get heated because both sides have a point.

💡 Key Insight

If the argument is speed, it doesn’t really hold up. PySpark and Spark SQL go through the same engine and the same query optimizer, so the same logic usually runs the same way whichever one you write it in.

SQL is the simpler choice the majority of the time while Python may be better for certain complex transformations. That being said, Python does make it easier to write unit tests and reuse code, which is a fair point in its favor. In reality, most teams settle this with a written standard that says which kinds of logic live in SQL and which live in Python, so everyone knows where to look.

Rule of thumb: use SQL for transformations until the complexity requires a different approach.

Poll Results From Last Time

Do you think most data engineers understand dimensional modeling well enough to guide an AI? (34 responses)

Industry Pulse

Here are some industry topics and trends from this month:

1. DuckDB 2.0 and Polars 2.0 are here

Two of our favorite tools hit major versions, and both come with an upgrade note you’ll want to know about before you bump the version.

DuckDB 2.0 is a long-term support (LTS) release, supported until at least October 2027. The headline is a finished version of Quack, a protocol that lets one DuckDB connect to another running as a server, so DuckDB can now act as a small shared database instead of only living inside one Python process. Before you upgrade, know that older DuckDB versions can’t read database files written by 2.0, so update anything that reads your files first.

Polars 2.0 makes the streaming engine the default and spills to disk when memory runs low. The change most likely to bite you is that row order is no longer guaranteed after joins, group-bys, and unpivots. If your code quietly relied on rows staying sorted, set maintain_order or add a sort where it matters (if you didn’t ask for an order, you aren’t guaranteed one!).

And if you’ve wondered what AWS plans to do with DuckDB after agreeing to buy DuckLabs, here’s a first hint: Aurora PostgreSQL can now use DuckDB to query Iceberg and Parquet files in S3, read-only.

Continue reading: DuckDB 2.0 | Polars 2.0 upgrade guide | Aurora PostgreSQL with DuckDB

2. A database breach and a poisoned Python package

Diagram showing compromised LiteLLM 1.107.0 installing Python startup files that automatically launch a credential collector. Stolen developer, cloud, browser, SSH, environment, and LiteLLM database data is encrypted and sent to an attacker-controlled endpoint. An optional Unix path can retrieve and

Two very different attacks this month went after the same thing: your credentials.

PingCAP, the company behind TiDB Cloud, confirmed that attackers used stolen service-account credentials to get into its internal systems for about a month. They took TiDB Cloud usernames and emails, text from the SQL editor, billing data, and a table of encrypted API keys. PingCAP says there’s no evidence that data stored in TiDB itself was touched. An extortion group claims it took much more and says engineers pasted customer credentials into Jira tickets, though none of that is verified. If you use TiDB Cloud, rotate your management API keys and watch for phishing emails.

Meanwhile, a poisoned release of LiteLLM (a popular Python library for calling different AI models through one interface) sat on the Python Package Index (PyPI) and Docker Hub for a few hours. It ran every time Python started in that environment, even if you never imported LiteLLM, and stole cloud, Kubernetes, Git, and SSH credentials. If you installed version 1.107.0, pin to 1.106.2, rebuild your environments, and rotate your secrets. As always, pinned versions and a lockfile are what keep a nightly build from grabbing a bad release during the few hours it’s live.

Continue reading: PingCAP security update | LiteLLM incident report

3. Are AI coding tools breaking database benchmarks?

GitHub - ClickHouse/ClickBench: ClickBench: a Benchmark For Analytical Databases

Every database says it’s the fastest, and this month gave us a good reason to be skeptical. Thomas Neumann, a database researcher at the Technical University of Munich, warned that AI coding tools are making benchmarks less useful. His example is pgrust, a newer engine posting fast times on ClickBench, a popular public speed test for analytics databases. Some of its tricks lean on knowing the queries ahead of time, like caching filters that repeat, and one groups text by its hash alone, which Neumann says can give wrong answers. As he puts it, “If you tell an AI to make the queries faster, it will make the queries faster.” His fix is to treat benchmarks like an exam you can’t study for, with hidden or freshly generated queries each round.

Keep that in mind the next time a vendor chart crosses your feed. Polars says 2.0 beat DuckDB on tests Polars built and ran itself, and ClickHouse says its own CostBench benchmark shows it’s 752 times better per dollar than Databricks. Neither has to be wrong, but the benchmark that counts is still your own data and your own queries.

Continue reading: The Benchmarking Crisis | Polars 2.0 benchmarks | ClickHouse CostBench

Thanks for reading! Share this post with someone who might find it valuable.

🎁 Bonus:

📅 Upcoming Events

Share an event with the community here or view the full calendar

Opportunities to get involved:

What did you think of today’s newsletter?

Your feedback helps us deliver the best newsletter possible.

Want more Data Engineering? Join our community.

Stay tuned next month for more updates, and thanks for being a part of the Data Engineering community.

All content and views expressed in this newsletter are those of the original authors and do not reflect the thoughts and views of the Data Engineering Community.

© 2026 Data Engineering Community
548 Market Street PMB 72296, San Francisco, CA 94104