Most data quality programs die the same way. Someone writes the rules, the rules pass, and eighteen months later a report is wrong, and nobody can explain why. The rules didn’t break. The data changed underneath them and closing that gap is the entire job of data quality management.
This page is the manual we wish the category had when we started. All of it comes from doing the work inside enterprises that were fifteen years into their ERP schema before we arrived. We kept the parts that broke.
What is data quality management?
Data quality management is the practice of defining what correct means for your data, enforcing those definitions automatically, and proving they held. It covers data profiling, rule definition, validation, monitoring and remediation. Running the checks is the easy part. Keeping the rules accurate while the business changes underneath them is the hard part.
Cleansing projects fix a snapshot. Management keeps the fix true next month, next quarter and next year.
A source system adds a field. A currency code changes. Somebody quietly changes what counts as an active customer. The rules don’t hear about any of it.
Think of it as brakes, oil and a service log for your data. You don’t buy them once; you keep them working.
In practice that means five moving parts. Data profiling tells you what’s actually in a dataset, and rules encode what should be in it. Validation checks the two against each other.
Monitoring watches results over time. Remediation routes whatever fails to someone who can fix it.
ISO 8000-61 describes this same loop as a formal process model. That helps when a budget holder asks why the project has no end date. It has no end date because it’s something you run, with a cost, like monitoring or backups.
Why do most data quality programs fail?
Most data quality programs fail for four reasons: rules get duplicated across every pipeline, static thresholds miss slow drift, errors surface in technical language the business can’t act on, and nothing remembers past decisions. All four look technical. All four are ownership problems.
We’ve hit each of these on real projects, usually in this order.
- Rules live inside pipelines, with no reusable data contract. The same completeness check gets rewritten in every ETL job that touches the customer table. Two teams end up enforcing two different versions of correct. We see this constantly in enterprise integration work: checks welded into pipelines that were never built to share them, so every new dataset starts from zero.
- Static thresholds miss silent drift. A null-rate alarm set at 5% in 2023 says nothing while the real rate creeps from 0.4% to 4.9% over a year. Schema drift behaves the same way. By the time anyone notices, the damage is three joins downstream, in a report someone already presented.
- Errors arrive in the wrong language. “Regex mismatch on CUST_TYPE_CD” means nothing to the person who owns customer data. If a business owner can’t act on an alert, the alert trains everyone to ignore alerts.
- There’s no governed memory. Every run starts cold. The reasoning behind last year’s threshold change lives in one engineer’s head. The day they leave, the program starts guessing.
The fix for all four is the same shape: shared rules, moving baselines, plain words, and a memory. Hold that thought for the last section.
Vendors rarely write this section. Writing it means admitting the category is harder than the brochure says. It’s also why buying a scanner and calling it a program keeps failing.
What are the six dimensions of data quality?
The six dimensions of data quality are completeness, uniqueness, validity, accuracy, consistency and exploratory. The first five are the classics you’ll know from DAMA-DMBOK. The sixth covers the checks nobody thought to write: patterns an automated profile surfaces that no human rule saw coming.
Each dimension earns its place by failing differently in the wild.
| Dimension | What it measures | How it fails in production |
| Completeness | Required values are present | A source starts sending empty strings instead of nulls, and every null check keeps passing |
| Uniqueness | One real-world thing, one record | The same customer exists three times under three spellings, and revenue counts them three times |
| Validity | Values match allowed formats and ranges | A date field holds 03/07 entered in two regional orders, and both parse |
| Accuracy | Data matches reality | The warehouse says the contract is active. The signed PDF says it ended in March |
| Consistency | The same fact agrees across systems | Finance and CRM disagree on customer count by 8%, and both teams are certain |
| Exploratory | Patterns outside any written rule | Order volume from one country doubles because a test account never got switched off |
Print the table and pin it near whoever owns the rules. It pays for itself the first time a check stays green while a report goes wrong.
DAMA-DMBOK treats dimension selection as a design decision, and we agree. Measure the dimensions your real decisions depend on. How you weight them belongs in a data quality framework. So does turning the weights into a score.
How do you know a data quality rule is still working?
A passing rule proves the data matched the rule, and nothing more. To know the rule still fits reality, you need a historical baseline for every metric it touches. Baselines show drift before it crosses a threshold. Rules rot quietly, and the pass rate stays green the whole time.
Baselines change the question from “did this pass” to “does this still look like itself.” A 3% null rate passes a 5% threshold. A null rate that sat at 0.2% for two years and hit 3% this week is an incident, whatever the threshold says.
What do you baseline? Start with the boring metrics: null rates, distinct counts, row volumes per load, value spreads on the fields your reports lean on, and arrival times.
None of it is exotic. All of it is cheap to store. The point is to give every rule a memory, so this week gets compared with the last hundred weeks rather than with a number someone picked in a meeting.
This is where hardcoded limits turn into the problem they were meant to solve. In our own live deployment, moving to anomaly-aware thresholds that track the historical baseline cut false alarms by 60%.
The mechanism matters more than the number. Thresholds that move with the data stop paging people about seasonal swings. The pages that remain get taken seriously.
Two other signals tell you a rule has expired. Data lineage shows its inputs now pass through a transform nobody planned for when the rule was written. And schema drift, a renamed column or a new enum value, breaks the rule’s assumptions without breaking its syntax.
A rule can survive both and keep passing. That’s the trap.
So check your own rules this week. Pick the ten oldest. Ask who last read them, when they last changed, and who signed the change off.
What does bad data quality cost before anyone notices?
Bad data costs rework first, trust second, and decisions third. Teams re-run reports, re-reconcile numbers and re-explain gaps long before anything reaches a KPI. In our production use, steady quality checks cut that rework by 40% and removed the downstream incidents that used to follow bad loads.
For a CDO the bill arrives as headcount: analysts who spend their week checking numbers rather than using them. For a CFO it arrives later and larger, as a restated figure or a write-off traced back to duplicate records. Master data management failures compound fastest. One bad customer or product record touches every process that uses it.
The quietest cost sits in analytics. A wrong number in a dashboard doesn’t announce itself. Half the time the number is technically valid and semantically wrong. The semantic layer and the warehouse hold two definitions of the same metric.
That’s why we build quality checks into data and analytics delivery from the first sprint. A cleanup phase after go-live always arrives too late.
Want the business case? Count the hours your team spent last month checking figures they already produced once. That’s the budget line this program replaces. Most teams guess low, sometimes by half.
One honest limit. Data quality management surfaces bad data. It can’t prevent bad decisions, because people still make those. What it removes is the excuse.
What does agentic data quality actually change?
Agentic data quality adds a discovery layer on top of deterministic validation, then wraps both in governance and memory. Agents profile datasets, propose new rules and explain failures in business language. Humans approve. Validation still runs deterministically in Python. The AI never gets to decide what’s true.
This is the architecture we built ThemisData around, and it answers the four failure modes above directly. Three layers:
- Deterministic validation. Versioned rules run as code against data contracts, with results you can repeat. Nothing probabilistic touches the pass or fail decision.
- Agentic discovery. Agents profile new datasets and flag patterns no rule covers yet. That’s the exploratory dimension at work. They draft rules with a plain-language reason attached.
- Governance and memory. Every proposed rule goes through data steward approval. Every change is versioned, and every result carries an evidence chain. The rule lifecycle is 100% auditable, which is the difference between an alert and something you can show an auditor.
The effect shows up at onboarding. Wiring a new dataset into the program used to take days of hand-written checks. With agents drafting and stewards approving, it takes hours.
Days to hours sounds like a sales line, so here’s the honest version: agents remove the drafting time, and only the drafting time. The stewards still read every rule. That part doesn’t shrink, and shouldn’t.
The approval step is the whole point of the design. Discovery without governance means a machine can promote its own guesses into policy. If you’re building a data governance framework, the approval workflow is exactly where the two programs meet.
Common questions about data quality management
These four questions come up in nearly every data quality conversation we join. The answers below are short on purpose. Each one is the version we’d give a colleague who asked in a corridor. Caveats kept in, vendor gloss left out.
What is the difference between data quality and data governance?
Data quality measures whether the data is right; data governance decides who defines right and who proves it. Quality is the measurement, and governance is the authority behind it. You can’t do the first well without the second, because a rule nobody owns is an opinion with a cron job.
Who owns data quality in an organisation?
The data steward owns the rules, the CDO owns the outcome, and the business owns the definitions. “Active customer” means whatever the business says it means. No engineer can read that from a schema. That’s why steward and business roles survive even when most checks run themselves.
What are the six dimensions of data quality?
The six dimensions are completeness, uniqueness, validity, accuracy, consistency and exploratory. The first five test data against what you already expect, and the exploratory dimension covers what profiling finds beyond them. Think anomalies, odd patterns, and outliers nobody thought to write a rule for.
Can AI fix data quality automatically?
AI can find problems you didn’t know to look for and explain them in business language. It shouldn’t be allowed to promote its own findings into policy. Agents propose. Stewards approve. Validation runs deterministically. That separation keeps an automated program auditable while it speeds up.