🛠️ What's happening in data engineering? | DataTalks.Club Weekly

#301 · November 2, 2026

This week: an Astronomer webinar on data engineering trends, our new course on AI agents for generative media, the Project 1 peer review in AI Dev Tools Zoomcamp, the Machine Learning Zoomcamp midterm project, and upcoming community events.

SPONSORED

What's happening in data engineering? (Nov 18)

You’re deep in legacy data, broken pipelines, and surprise schema changes. Now there’s Jev, new open source models, and Airflow Dags with loops???

If you’re feeling overwhelmed, you’re not alone. Join data engineers Tamara and Volker on November 18 for an honest conversation about what’s worth your attention and what you can safely ignore.

Here’s what to expect:

• Two data engineers trying to decipher the current trends

• Where your role is headed (it’s not all bad!)

• Resources and code to take home

New Free Course: 2026 Cohort Registration is Open

When an agent's tools are GPU jobs, you have to handle async calls, polling, retries, resuming, and duplicate execution. This free course shows you how to:

• Run image, video, and audio models on serverless GPUs, without managing a cluster

• Give agents GPU models as tools (MCP) and build generate, evaluate, regenerate loops

• Fan out batch workloads, handle failures, and report cost per asset

To apply, tell us what you're building. We'll select a limited number of participants who'll join the live cohort with homework and a final project. The materials are public and MIT-licensed, so you can still follow along.

Project 1

Last call for Project 1: submissions close on October 27, 2026 at 00:00 CET (October 26 at 23:00 UTC). Check what to build and how it's scored before you submit.

After the deadline, you review three projects from other participants. Reviews are part of the project: without them, your project doesn't pass. They're due November 3, 2026 at 00:00 CET (November 2 at 23:00 UTC).

Not ready yet? You can submit on the second attempt, Project 2, until November 17, 2026 at 00:00 CET (November 16 at 23:00 UTC).

Midterm Project

We start the midterm project. For the next two weeks, you apply everything from Modules 1 to 6 to a problem of your choice: pick a new public dataset, train and evaluate a model, and deploy it as a web service. Check the project guidelines before you start.

Submissions are due November 17, 2026 at 00:00 CET (November 16 at 23:00 UTC). After that, you review three projects from other learners. Reviews are due November 24, 2026 at 00:00 CET.

Still finishing Homework 6? There's still time: you can submit it until November 3, 2026 at 00:00 CET.

EVENTS

Upcoming Events

See all upcoming events · Add our events to your calendar

RECORDINGS

Podcast Recording: Decoding the Open Lakehouse

On September 14, we talked with Sahil Walia, Senior Technical Architect at Snowflake, about how data platforms have changed, from writing simple SQL queries to running enterprise-scale Apache Iceberg lakehouses. We discussed:

• How Apache Iceberg differs from Hive metastores and compares to Delta Lake

• How Iceberg metadata helps with GDPR deletions and file compaction

• Whether you need a lakehouse for small data

• How to start contributing to open-source data projects

READ

From Alexey on Data: Trends in Agentic Engineering

Do coding agents write more of your code than you can review?

In his new newsletter, Alexey looks at agentic engineering, following his webinar with Hugo Bowne-Anderson:

• The implementer-reviewer approach: one agent builds, another tries to reject the result

• Books like Clean Code as shared rules for agents

• Organizing agents into teams and larger hierarchies

• Dark software factories, where humans never look at the code

DataTalks.Club is a global community of data enthusiasts. If you haven't had a chance to join our Slack yet, use this invite link. If you have problems with the invite link, fill in this form, and we'll add you to Slack manually.

Do you have any questions? Hit reply, and we'll be glad to answer.

See you in Slack!

Copyright (C) 2026 DataTalks. All rights reserved.
You're receiving this email because you registered in our Slack community DataTalks.Club or attended one of our events

Our mailing address is:
DataTalks Langhansstr. 70 Berlin 13086 Germany

Want to change how you receive these emails?