mailwal

How I Use Claude Code as a Data Engineer (The Toolkit)

Data with Baraa

How I Use Claude Code as a Data Engineer (The Toolkit)

Four tasks that cost me the most time as a data engineer, and how I do them with Claude Code now.

Hey friends, Happy Thrusday!

A few weeks ago, Anthropic reached out to partner on a video showing how I use Claude Code for data engineering.

I was already building tools with it for my projects, and today, that video is live 🙂

Let’s talk about what we actually do all day as data engineers.

Writing the code is maybe the easiest part of our job. Most of the time goes into understanding new sources, designing the data models and the architecture, and reviewing the work of others.

And a big part of every day is firefighting: fixing bugs, investigating incidents, and figuring out why the data is wrong.

So this is not about using Claude Code to write scripts, this is easy. It is about the real work, and here are four of those tasks, with the pages it gave me.

Let’s go.

Three habits first

Before any task, three habits that make all of this work:

  • CLAUDE.md: the onboarding document of the project. Claude Code reads it at the start of every session, so keep it clear and short.

  • Skills: every task that repeats becomes a skill, a text file with the steps. After that, the task is one command.

  • Web pages: for every big task, I ask for the result as a web page. It is easier to read and to follow than a wall of text in the chat.

For the model, I use Claude Opus 5.5 for almost everything. I switch to Claude Fable 5.1 only when the work gets really complicated, like re-engineering a complex concept or a big decision.

1. Profiling a new source

Before you connect a new source, you have to understand it: the format, the structure, the quality. You need this before you design or build any pipeline, and to define the data contract with the source system.

And detailed profiling takes a lot of time, so in reality we check two or three things and hope for the best.

Now Claude Code gets my checklist, the same one I do by hand:

  • File format: delimiter, encoding, null tokens, decimal separators, date formats.

  • Schema: tables, columns, data types, row counts, and the schema drift between the files.

  • Data quality: nulls, empty columns, constant columns, duplicates, and the uniqueness of the primary key.

  • Bad records and personal data: corrupt rows, outliers, values out of range, and the columns we have to flag.

The result is one page for the whole source: 10 tables, 87.7 million rows, 205 columns, and 21 findings.

The profile of the purchasing source: 10 tables, 87.7 million rows, 205 columns, and the row count of every table

And this is what one source can hide:

  • The file format: two delimiters, two encodings, and the dates in two formats, one with a point and one with a minus.

  • The nulls: sometimes NULL, sometimes an empty string, sometimes a dummy value.

  • Duplicate rows: the same rows delivered twice, on different days.

  • Schema drift: a column added and a column renamed in the middle of the monthly files.

  • Data gaps: business days without rows. Is this normal, or is something wrong with the delivery?

  • Deleted rows: the source also deletes rows.

Here are two of them on the page: the schema drift and the data gaps.

Two findings from the profile: the schema drift in the monthly files, and the business days without rows per table

You never catch all of this by hand. You find it in the middle of your pipeline, way too late.

Now the data contract starts from the findings: one encoding, one separator, or I handle it inside the pipeline. And I can plan all of this before I build anything.

And because every new source gets the same checks, the whole thing is a skill now. Next time it is just /profile and the name of the source.

The profile skill: a text file with the job and the steps

Just a text file with the steps, that is all a skill is.

2. Investigating an incident

The first thing we check in the morning is the pipelines. And if one failed, the day is gone, because everyone is waiting for fresh data.

Finding the cause can take more than a day. We spend half of our time on broken pipelines.

In every project I build the knowledge from the history. For incidents, this is a log file, because incidents repeat: the same source renames a column again, the same check fails again.

If the fix lives only in one head, next time you start from zero, or you wait until this person is back from vacation. So every incident goes into the log, Claude Code reads it on every new incident, and writes the new entry at the end.

Here is the report for one failed run:

The incident report: where it failed, when, the status, and the error message

It follows the order I always use, and this is what came back:

  • Where: the silver layer, with the job, the script, the table, the time, and the error message.

  • Root cause: the source renamed status_code to status, and added a new column. The rows of yesterday and today sit side by side, my proof when I ask the source team: did you change something?

  • Seen before: yes. In June, another source renamed a column, and the fix is in the log.

  • Workaround: ask the source to resend the file in the old schema, good luck with that. Until then, the gold layer holds the data of yesterday.

  • Two solutions: fix the schema in the data contract, 3 hours, or add a new check at the first gate, 2 days. Claude Code recommends the first one, but the decision is mine.

  • Future risks: 16 tables can fail exactly the same way.

  • Two email drafts: one to the consumers, one to the source owner. I read them, and I send them myself.

  • The log entry: it goes into the incident log, so next time Claude Code finds it directly.

Here is the root cause with the two rows, and the entry from June:

The root cause: the column status_code is called status in the new file, the row of the day before and the row of today side by side, and the entry of a similar incident in June

For the same error, a junior spends the whole morning and a senior needs one coffee, because the senior saw it before. And here, Claude Code checked it for me.

And because we check the pipelines every morning at the same time, this skill runs as a routine. Every day at 05:45 UTC it checks the last run, and if it failed, it investigates and leaves me the report.

A routine runs on Anthropic’s servers, connected to the GitHub repository. My machine can be off, and the report is waiting when I start my day.

3. Building the metadata

We use metadata for many things: the lineage, the data quality, the generators. And the painful part is the description of every table and every column.

A team that commits to it might spend a month or two. So I met many teams that skip it, and they miss out on a lot.

My rule for doing it with AI is simple: Claude Code is not allowed to guess anything.

  • The technical part comes from the data: Python scripts read the CSV exports and give the tables, the columns and the types.

  • The descriptions come from the documents: the PDFs of the source system, after you checked what your company allows.

  • Unknown stays empty: if the documents say nothing about a column, the description stays empty.

I write one example by hand, one source and one table. Claude Code extends it to the rest of the source, with a Python script that I run again when new tables arrive.

The filled metadata file: the source, the first table, and its columns with name, type and description

One of my projects had over 1,000 tables from 12 sources. At that size I fan out: in this project 10 agents, one per source, and the main session is the orchestrator.

With the core metadata in place, the generators can start, and the column descriptions go into the SQL of the silver layer by themselves.

4. The data engineering toolkit

In a complex task, most of the time goes into the analysis before the building: which functions to touch, what depends on them, where the data comes from, how the model connects.

And after a few years, nobody has the big picture of the project anymore.

So Claude Code built me a small website from the repo: an overview, the runs, and one page per tool, with six Claude Opus 5.5 agents in parallel. This is the architecture page:

The architecture page of the toolkit: the sources, the layers, the gates and the consumers, drawn from what runs

  • Architecture: the sources, the layers and the gates. When I enter a new project, I spend a lot of time drawing this by hand just to understand it.

  • Data lineage: click a gold table and see everything upstream and downstream. Before you change anything, you know what breaks.

  • Call graph: the same for the Python functions. Click one and see every function that depends on it.

  • Data model: the entities at the top, and the star schema of the gold layer below.

  • Data quality: every check per day, and when it failed.

  • Data catalog: every table and column. Type the name and see the inventory.

And this is the lineage page:

The data lineage page: the silver_clean and silver_core tables on the left, the gold tables on the right, and the lines between them

Every time I run it, the pages are up to date, because they are built from the code itself, not drawn by hand.

They might sound like a luxury, but they save a lot of time, and they make your life easier in a project.

Three more tasks

All three run the same way, a skill with the steps and a web page at the end:

  • Data products: we prepare the data as a product, as part of the data mesh. The consumers browse an overview, and inside each product they find everything from the guide to the contract.

  • The pull request review: Claude Code reads every line the way I do: the security, the correctness, the duplicates, and our standards.

  • The big review: Claude Code looks at the whole solution at once: the housekeeping, the costs, the complexity. I used to do it twice a year, now as often as I want.

This is the product page of Trading, with its tabs from the guide to the contract:

The data product Trading: the release, the grain, the tables, and the tabs from the guide to the contract

And the big review of one month, the numbers on top and the most read tables below:

The big review of one month: 3,032 queries, 11 unused tables, 11 unused functions, 0 checks in the gold layer, and the top 10 tables by query count

Every one of them was a task I did by hand, or did not have the time to do at all.

Quick recap

Four things to take with you:

  • Give it your checklist: the prompt is the list of what you check by hand, in your order.

  • Make it a skill: once the result is right, the task becomes one command.

  • Build the knowledge: CLAUDE.md, the incident log, the metadata, so next time nobody starts from zero.

  • Review everything: our work is built on trust, and once you lose it, it is hard to get it back.

So …

Data engineering is so much more than coding, and this is exactly why companies pay well for us.

And Claude Code is not only writing scripts for me anymore. It reads, it investigates, it profiles, it discusses with me.

This is the difference between a data engineer who uses AI to move faster and one who does not. But I always review, and I always make the final decision.

If you want to start, take one task you repeat over and over and give Claude Code the list of what you check by hand. And when the result is right, make a skill out of it.

Thanks for reading ❤️

Baraa

Also, here are 4 complete roadmap videos if you’re figuring out where to start:

📌 Data Engineering Roadmap
📌 Data Analyst Roadmap📌 AI Engineering Roadmap📌 Data Science Roadmap

Hey friends —

Hey, I’m Baraa, a Data Engineer with over 17 years experience, Ex-Mercedes Benz, where I led and built one of the biggest data platforms for analytics and AI.

Now I’m here to share it all through visually explained courses, real-world projects, and the skills that will get you hired. I’ve helped millions of students transform their careers.

Visit My Youtube Channel

© 2026 Baraa
548 Market Street PMB 72296, San Francisco, CA 94104