Every AI project starts with a pitch. This one started with a question: are these numbers actually right? 

A data pipeline is the chain of steps that moves data from source systems into the reports a business relies on. When a pipeline breaks, it rarely looks broken. It still runs on schedule and the dashboard still refreshes, so the number on the screen looks plausible even when it is wrong. 

The cause is usually several steps back. A join (the step that combines two tables) may quietly duplicate rows, or a filter may drop part of the business. Nothing fails, so nobody is alerted, and the report drifts away from reality. 

Our client, a global enterprise with hundreds of pipelines feeding data from dozens of source systems into business reports, knew this well. Validation was mandatory on paper, but in practice it was manual, slow and done by the same person who had built the pipeline. There was no shared standard approach for what “validation” meant across the different teams. 

Impact: a pipeline validation that used to take approximately 3 days now takes around 1 hour. 

For 10 pipelines, that is about 240 working hours (10 × 3 days × 8 hours) → about 10 hours (10 × 1 hour). The saving grows with every pipeline that is created or changed. 

These figures are averages based on the validations completed so far, so they may not apply to every case. The actual time depends on the complexity of the pipeline and can be lower or slightly higher. The 10-pipeline example is an illustration based on those averages. 

The trust came first, the certainty came later 

What made this project possible wasn’t a technology decision. This project was possible because the client was willing to let us learn in front of them. They gave the team room to experiment and to make mistakes, and the patience to let a real solution take shape. 

There is one detail we do not usually share with the audience: nobody on the team had an AI engineering background. Our experience was in pipelines and data models, and prompts and model behavior were new to us. At the start, that gap was the biggest risk on the table. 

Q&A 

You’re data engineers, not AI engineers. How did you end up building AI agents? 

Gabriela Doneva, Data Engineer: Honestly, we never planned to build AI agents. We had a problem, and AI turned out to be part of the answer. 

The question was simple: does this pipeline really do what it claims? We needed that answer for every pipeline, without someone spending three days on each one. 

Our first version was very simple: one long instruction file telling an AI assistant how to validate a pipeline. In a demo it looked great. In real work it wasn’t reliable at all. 

And I’m actually grateful for that, because it showed us very quickly where the hard parts were. 

What surprised me was how much of our experience we could keep. As data engineers we work every day with sources we don’t fully trust, like a file that suddenly changes format or a system that sends the same record twice. So you get used to checking everything and believing nothing at first sight. 

An AI model is exactly that kind of source. It’s fast and it sounds confident, but sometimes it’s simply wrong. We already knew how to handle that. 

The instinct was there, we just had to learn the vocabulary. It felt like being a good driver in a new country: you know how to drive, you only need to learn the road signs. 

I really started to trust the tool about three months in. We ran it side by side with validations people had done by hand, and on every pipeline in the trial it came back with even better verdicts. That was our go decision. 

Making it solid enough to hand over to other developers took another couple of months, and that’s where the interesting part starts. 

What we built, in plain terms 

Every number in a report has travelled through a chain of steps: it is extracted from a source system, cleaned, joined with other data and summarized. The framework checks that journey one step at a time. 

For each step, it rebuilds what the result should be by following a recipe written from the pipeline’s code. It then compares the rebuilt result with what the pipeline actually produced. If the two agree, the step does what it claims, and if they differ, something in the pipeline is wrong. 

In other words, what the framework tests is the pipeline’s logic: the way each step transforms the data. 

Figure 1. How every step gets checked: the framework rebuilds what a step should produce, following the recipe, and compares it with what the pipeline actually produced. 

Figure 2. Built once, checked every day. A person approves before anything is checked, and a person decides what to do when something is wrong. 

Figure 3. What it checks. Four kinds of checks decide whether a pipeline passes; oddities in the data are reported but never fail a run. 

Q&A 

Why not just point an AI model at the whole problem? 

Gabriela Doneva: Because an AI model can sound completely right and still be wrong, and in validation that’s exactly the mistake you’re there to catch. Asking the model to try harder won’t fix it. The design has to. 

So we drew a clear line: if something can be written as code, it stays in code. The model only works where code really can’t help, which is understanding what a pipeline’s code is trying to do and explaining a failure in plain language. 

Even when it describes a pipeline, it can’t write whatever it wants. It picks from a fixed menu of building blocks, so it can only describe what’s already there. If it reaches for something that isn’t on the menu, the recipe is rejected before a single row of data is read. 

Everything else is normal, predictable code: the self-checks, the rehearsal and the engine that does the calculations. 

And the model never sees the client’s data. The engine reads the data where it lives, and the only thing that comes back is a scorecard with numbers, like counts, totals and pass or fail. Anything that could carry real records is removed on the way out. 

So the AI works with the recipe, and the data stays with the client. For us that boundary is part of the design itself, and it’s the reason people can trust the whole thing. 

Figure 4. Where the data lives, and what is allowed to leave. 

The moment it almost didn’t work 

We did not start with that design. In the first version the model did most of the work, and that nearly ended the project. 

The main problem was inconsistency. Given the same pipeline twice, the model described it in two slightly different ways. It might read a join one way on Monday and another way on Tuesday, or treat a number as an amount in one run and as a category in the next. 

Some variation is acceptable in many AI products, but in a validation tool it is fatal. A checker that gives two different verdicts on the same pipeline is worse than having no checker at all. 

We stopped adding features and focused on the inconsistency. We ran the same pipelines repeatedly, recorded every place where the descriptions disagreed and sorted each disagreement by its root cause. 

Where code could compute the answer, we moved the decision into code. Where the answer was genuinely ambiguous, we wrote explicit decision rules. It was slow, unglamorous work, and it saved the project. 

Q&A 

What would have killed this project, and why didn’t it? 

Gabriela Doneva: Two things, in this order: inconsistency first, then false alarms. A validation tool really only has one thing going for it, and that’s people trusting it when it says something is broken. Either of those problems could have killed that trust. 

The inconsistency is the story you just read, so I’ll skip it here. 

The false alarms came later. In our early runs on real data, the tool flagged a pipeline as broken when it was actually fine. The real cause was how the framework handled data shared between different parts of the business, and the pipeline itself was perfectly healthy. 

We caught it early and fixed it. If it had happened at scale, people would have learned to ignore the red light very quickly, like a car alarm that goes off every time a cat walks past. Once that happens, any monitoring tool is finished. 

What it does today 

Today the framework is part of how pipelines are built. When engineers create a new pipeline or change an existing one, they request a validation with a single line in the AI assistant they already use for coding. The framework then completes the steps described above and returns the recipe to the engineer in plain language. 

The next step is deliberately human. The engineer reviews the recipe as they would review a colleague’s work, and decides whether to approve, correct or decline it. 

Once a recipe is approved, the framework continues to check the pipeline. When everything matches, the engineer receives a short confirmation email. When something does not match, the engineer receives a ticket that starts with the cause: which step failed, which column is affected and why, all in plain words. 

The framework never changes a pipeline and never approves its own work. Those decisions stay with people. 

Q&A 

What would you tell the next client thinking about automating part of their work? 

Gabriela Doneva: First, pick one real problem you can’t solve today and explain it properly, from start to finish, before you start thinking about scale. Be strict where the AI goes, use the models only for the parts that genuinely need reasoning. 

Second, give the tool to the people who will use it as early as you can, and let them try to break it. When colleagues ran it on their own pipelines, they found things we would never have found on our own. 

And the most important one: be honest with each other. Work with a partner who tells you openly when something didn’t work, and do the same for them. In our case that honesty went both ways, and it’s the reason this works today.