Every production system has a quiet window, and that window is exactly where the worst problems grow. Nikola spent years watching failures build up between 2am and 10am, then land on engineers at 9:05, in front of everybody. So he spent his evenings building Argus, an AI agent that investigates alerts the way an engineer would: it pulls the logs, traces, deploys and past incidents, proposes a cause, and shows its evidence so you can check the work yourself.
We sat down with him to talk about the problem behind it, the mistakes along the way, and why he built a tool that deliberately shows you where not to trust it.
Meet the Builder
What’s the day job, when you’re not building Argus?
I’m a Senior Technical Consultant in Application Operations. In plain terms, I keep other people’s production systems visible and healthy, and I’m one of the people who gets called when they aren’t.
Day to day I’m in AppDynamics, Datadog, Azure Monitor, CloudWatch and Grafana. That list reads like showing off, but it isn’t a choice. Every client runs something different, so I have to be comfortable with all of them. Around that there’s cloud and infrastructure work, integrations, automation, and a lot of ordinary troubleshooting.
I’m also team lead on my main project, so a decent chunk of my week goes on other people’s problems rather than my own.
There’s no typical day. It depends whose problem is loudest that morning. Some days I’m doing QE support or debugging something for a team I’m not even on.
The tools I open every single day, as opposed to the twenty I touch now and then: VS Code, the terminal (kubectl, cloud CLIs, git), Postman, Slack and Teams, plus whichever ticketing system the client happens to use.
That constant switching is also why I kept Argus small. I never had a clear month to build something big, so I built it in pieces that each had to stand up on their own.
The 2AM Problem
Let’s start at 2am. What’s actually happening on the systems you look after?
For most of the clients I look after, the risky hours aren’t the busy ones. They’re roughly 2am to 10am, when the nightly batch runs. Data pipelines, scheduled jobs, maintenance tasks. All the work that gets pushed to the middle of the night because nobody’s using the system.
Two things make that window hard for me.
Nobody’s watching closely. And the traffic at 4am looks nothing like the traffic at 2pm, so I don’t have a reliable picture of what normal even is at that hour. I can’t confidently say something looks wrong when I’m not sure what right looks like.
Then at 8 or 9 the real users log in, and the system is supposed to already be fine. No warm-up. No grace period.
That sequence is why I built Argus. A connection pool slowly filling up since 3am, meaning the limited set of database connections an app is allowed to hold open, doesn’t break at 3am. It breaks at 9:05, in front of everybody.
Why don’t those failures show up during the day?
Because at 2am it’s basically a different system.
A nightly job might be the only thing touching a particular table or queue at that hour. If it ends up fighting something else for the same resource and slows down, I can’t reproduce that at 2pm. At 2pm the conditions that caused it don’t exist. Different jobs, different load, different everything.
The other kind builds up slowly. A memory leak. Connections filling up a few at a time. Nothing looks wrong all night, then the morning rush pushes it over the line.
Either way I’m chasing something that only exists in a window I was asleep for.
What Argus actually is
Forty minutes a week isn’t a crisis. Why did it bother you enough to spend your own evenings on it?
Forty minutes is the wrong way to measure it.
Break it up and it’s four or five separate times a week where somebody gets woken up, opens five different tools, and burns the first few minutes just working out what’s going on before they can start fixing anything. Nobody’s losing forty minutes of output. They’re losing the ability to properly switch off, and that’s a much more expensive thing to lose.
I kept watching it happen to people around me, not only to me. That’s what moved it from something I complained about to something I spent evenings on.
I want to be straight about one thing. On-call pain isn’t new, and I don’t think AI is the only honest answer to it. Better alerts, better runbooks, fewer pages in the first place. Those are all real fixes. What changed is that a model can now work through several steps across different tools without falling apart: ask a tool, read what came back, decide what to check next, keep going. That wasn’t dependable a couple of years ago. Google’s own SRE team published on this same direction in “AI Engineering for Reliable Operations”, so it isn’t only my hunch.
Old problem. New opening.
Going after it was something we landed on as a team. The architecture and the building were mine. A few colleagues kept checking in, throwing ideas at me and telling me honestly when something wasn’t good enough yet, and that shaped it more than they realize.
Explain Argus to an on-call engineer. Three sentences, no product language.
When an alert fires, Argus goes and collects the logs, traces, recent deploys and past incidents for that service, works out what’s most likely wrong, and shows you the evidence it used so you can check it yourself.
You can also point it at a service manually, before or during a deploy, or tell it to sweep everything looking for problems that don’t have an alert yet.
It won’t change anything without you approving it first.
Building it the hard way
What did the first version get wrong?
There was never a clean version one. It started as something that could receive alerts and nothing else. It couldn’t go and look anything up. That came later.
My real mistake came with tool use. Argus has to pick which tool to call: fetch the logs, check the recent deploys, query the metrics. The model makes that choice, and that part is never going away. But my first attempt had the model write its choice out as ordinary text, and then I parsed that text to work out what it wanted. That holds up until the model phrases it slightly differently, and then it doesn’t work at all, and it fails quietly instead of loudly. I rebuilt it on structured function calling, where the model hands back the tool name and its arguments in a fixed format I can depend on. The model makes the same decision. I just stopped guessing at what it said.
The confidence one bothers me more, because I should have known better. I tried to stop it being overconfident by telling it not to be, in the prompt. That works right up until conditions get strange, and then it stops working. So I moved the limits into code. Hard caps the model doesn’t get a vote on.
Which took longer: teaching Argus to read the data, or teaching it to write something worth reading at 3am?
Reading took more hours. There’s a long line of integrations behind this, one per platform: Kubernetes, Grafana, Prometheus, Datadog, GitHub, GitLab, Jira, Confluence, AWS, Azure, Honeycomb, Jenkins, and more. Each one had quirks that only turned up once real data was moving through it.
But that work ends. I build the integration, it works, I move on.
Getting it to write something an engineer will trust never ends. Three things keep needing attention there. Making the confidence scores mean something, so 40% and 85% actually feel different to the person reading them. Catching it when a tool says one thing and the timeline says another. And making sure the evidence is genuinely part of the reasoning, rather than pasted underneath the conclusion to look convincing.
None of that transfers cleanly. Every client’s data looks and behaves differently, so I retune it each time.
Reading the data was breadth. Trust is depth, and depth doesn’t finish.
Who was the first colleague to use it, and what did they say that changed the design?
What I can say: people in my own practice looked at it and immediately started mapping it onto their projects rather than mine. “This would help with X.” That told me the idea worked past the one case I built it for.
It still happens. Nearly everyone who sees it brings an angle I hadn’t thought of, and that’s driving the roadmap now, not just at the start.
The comment I got most was that they could work the interface out without me explaining it. So I kept it simple instead of exposing every setting, which I was tempted to do, because I know what all the settings mean and I forget that nobody else does.
Argus on shift
Walk us through a shift handover, with and without Argus.
Without it: whoever’s going off shift has to write up or explain everything still in progress, and whoever’s coming on rebuilds the picture from scratch. They ask follow-up questions, then go and check the logs themselves anyway, because a secondhand summary isn’t something you want to bet on. All of it usually happens while the first person is trying to log off.
With it: that’s what I built the On-Call Briefing for. You open one page that’s been assembled from what Argus actually did overnight, not from someone’s memory of it. What’s still being investigated. Patterns that keep coming back. Which deploys line up with which incidents. What to look at first. Every line opens into the full investigation behind it, with the proposed cause, the confidence score and the evidence.
Three smaller things matter a lot at that exact moment.
If a problem comes back or changes across the shift boundary, I track it as a continuation of the earlier investigation. You see “this is the same thing from six hours ago, here’s what was already tried” instead of a fresh ticket for a repeat.
There’s a team view for what’s assigned or escalated right now. The briefing tells you what happened. The team view tells you what’s yours.
And when a recurring problem has an approved runbook, I link it from the investigation. The “here’s how we normally fix this” knowledge gets written down, instead of living only in the head of the person who just went to bed.
A NOC operator doesn’t know what to do next. What does Argus give them?
A NOC operator usually has three moves available: acknowledge it, escalate it, or close it as nothing. The hard part was never doing the action. It’s knowing which one, and who to hand it to.
So I built them their own view. Plain language, no stack traces, no raw metric names. When something needs an engineer, it names the team that owns the service and hands over the exact message to pass on. No guessing who to page, no guessing how to describe it.
What about when paging goes wrong, and an alert reaches the wrong team or reaches them too late?
I made the routing configurable. Rules match on service, alert type or severity, and each match goes to its own channel, ticket project or team.
The alternative, which is quietly what most setups are, is one fixed pipeline sending everything to the same place regardless of what it actually is.
How does it spot something before there’s an alert for it?
The proactive sweep is what serves that purpose. It runs across services with no alert needed, looking for things no existing monitor covers: error patterns, response times slowly getting worse, anomalies the observability platforms flag on their own.
Every alert rule anyone has ever written has gaps around the edges of it. That’s not a criticism of the rules, it’s just what a rule is. The sweep goes and checks those gaps instead of waiting for one of them to fire.
Where the human stays in charge
Where does Argus stop and the engineer start?
Argus proposes a cause and, where it applies, a fix broken into steps. Every step needs a person to approve it. Nothing runs on its own.
I’ll flag that the fix part is the newest and least proven thing in there. I designed it cautiously on purpose, but it hasn’t been through enough real incidents for me to call it settled. That’s honestly part of the answer to where it stops. It goes further than pulling the logs together and proposing a cause, and it stops well short of doing anything unsupervised.
It doesn’t get the last word on whether something is fixed either. It rechecks the original signal and reports back. A person decides.
And when it doesn’t know, I made it say so. Low confidence or no clear cause shows up plainly, rather than as a confident-sounding guess.
How do I check its work?
You get a ranked list of possible causes, each with its own confidence score, so you can see what else it considered and why the others lost. The cause and the suggested fix link straight back to the exact logs, traces, deploys and commits behind them.
Click through. Don’t trust the summary. I built it that way on purpose, because a summary you can’t check is just a nicer-looking guess.
Which part of Argus are you quietly proud of that nobody notices?
I built a whole section whose only job is to show you where not to trust it. Low confidence scores, repeated failures, negative feedback people have left. All of it deliberately visible.
Most tools don’t advertise their weak spots. Mine does, on purpose. A tool that only shows you when it’s right is more dangerous than one that admits when it’s guessing, and I’d rather people came to trust it slowly.