mailwal

Data Pipeline 101: Data Quality Checks

Vu Trinh

Data Pipeline 101: Data Quality Checks

A fresh look at data quality.

Nov 3
∙
Preview

To sustain my work, I’ve opened up paid membership. If you find value in what I do, I’d really appreciate it if you considered upgrading; it helps me keep this work going :d

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

Upgrade with 7$/month

  • If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

  • If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

The abstract goal of data engineering and software engineering is the same: deliver reliable solutions to some problems.

In software engineering, the solutions take the form of software

The solution's reliability depends on how well and safely the software performs.

In data engineering, the solution is a data foundation: a data architecture, a data set, a data pipeline, or … anything data users need.

The solution's reliability depends on data quality.

However, ensuring that is not easy.

This is because data is different from software. It's not just 0 or 1; it can have many states: correct but incomplete, correct but late, passing all tests but semantically wrong, or wrong without anyone realizing it…

Also, you can't "trial" data. You can't say, “Let's let 10 users try this first to see if there are any problems.” Data must be high-quality because it supports business decisions.

—

In this article, join me to learn and discuss the fundamentals of data quality checks in our field.

This article doesn’t try to give you every data quality check you should have or tutorials on how to implement them. Instead, I’m trying to give you the right mental model for thinking about data quality.

The rules

You might have heard about the famous data quality pillars: accuracy, completeness, consistency, timeliness, uniqueness…

They’re extremely helpful. They give you a kind of checklist.

There’s nothing wrong with them, but I prefer to think about data quality more abstractly so it ties more closely to the ultimate goal: serving the business.

—

The goal of the data quality check (or any check in the world) is to make sure the checked object meets some expectation.

The Great Expectation data quality platform did every good job here for picking that name :d

But for a dataset, where do the expectations come from?

To answer this question, we must think about the nature of the data:

  • It originates from business activities: a user visits a website, buys an item, or issues a refund

  • It’s captured so that the business owners can understand and optimize business activities: this month’s revenue, active users? Why do users churn so much? How did the marketing campaign perform? Does the new feature help?

So, the expectation of any dataset is that it must correctly reflect business activities so business users can trust it and make decisions based on it.

Data expectations are tied to business rules.

To me, the accuracy, completeness, consistency, timeliness, and uniqueness pillars are just categories:

  • Accuracy, uniqueness, completeness: rules that ensure data reflect the business correctly. Product price must be in range; email must be in the correct format; each user must have a unique identity; a user buying an item one time should have one record; or users must input their email…

  • Consistency: rules that ensure data reflect the business correctly, across all systems. If a customer’s address is updated in the operational database, it should be updated in the data warehouse.

  • Timeliness: rules that ensure data reflect the business in a timely manner so business users can make decisions based on fresh “reality“. The two-day-before, third-party data must be available at 7:00 daily.

Because the data expectations are tied to business rules, you must work with those who understand the business to materialize their understanding into rules. Yeah, you can argue that you don’t need this step to do the quality check.

You’re right, you can look at a table and write some rules yourself: email should have the correct format, unique fields must be unique, non-null fields should not contain null, …

The thing is, you're very likely to miss some “edge cases”; you interpret expectations from the table presentation, not how it’s expected and used.

To sustain my work, I’ve opened up paid membership. If you find value in what I do, I’d really appreciate it if you considered upgrading; it helps me keep this work going :d

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

Upgrade with 7$/month

  • If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

  • If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Subscribe to VuTrinh. to unlock the rest.

Become a paying subscriber of VuTrinh. to get access to this post and other subscriber-only content.

A subscription gets you:

Full article repository access + 35-page Spark PDF
Access to learning tools that boost your DE career: dbt, Spark, Airflow, ...
Vietnamese readers contact me for support. You will get 50% off.

© 2026 Vu Trinh
548 Market Street PMB 72296, San Francisco, CA 94104