Incident diagnosis time, the gap between an alert firing and an engineer forming a working hypothesis, is the largest hidden phase of MTTR in most enterprise operations. That’s where the on-call engineer is awake at 2 AM, jumping between six tools, trying to figure out what’s actually wrong. In our 20 years building enterprise IT systems, we’ve watched this gap consume 30 to 45 minutes on almost every serious incident, every shift, across every customer environment. This is the part of incident response nobody benchmarks. It’s also the part costing your customers the most.
If you’ve ever been the on-call engineer who stared at PagerDuty at 2:17 AM and asked yourself “okay, but where do I even start?”, this blog post is about you. It’s also about why that moment isn’t a personal failing. It’s a structural one.
What actually happens when an alert fires at 2 AM?
A typical on-call engineer touches six to eight separate tools before they have a working hypothesis. Monitoring (Datadog or Grafana). Logs (Elastic or Splunk). Deployments (GitHub, Jenkins, GitLab). Topology (a service catalog that may or may not be current). Ticketing (Jira or ServiceNow). Chat (Slack or Teams). Each one needs context, credentials, and a mental model. None of them are talking to each other.
Here’s how that plays out in real time, from real incident timelines we’ve reviewed across telco, banking, and CPG clients:
- 00:00 PagerDuty fires. The alert says “Provisioning Flow 5xx error rate elevated.” The engineer is in bed.
- 00:02 Laptop open. VPN connecting. Coffee not yet started.
- 00:04 Monitoring tool opened. Confirming the alert is still firing. Checking the time range. Is it spiking or steady?
- 00:09 Switch to GitHub. Was there a deploy in the last 24 hours? Which commit? Who wrote it?
- 00:14 Switch to Jenkins. Did the deploy succeed? Are there pipeline failures hiding upstream?
- 00:19 Back to monitoring. Pull up Order Manager and Symphonica. Are they seeing errors too? Is this contained or spreading?
- 00:26 Switch to logs. Filter to the right service. Filter to the right time window. Find the stack trace.
- 00:33 Open Slack. Search the channel for past mentions of this service. Did this happen last month?
- 00:41 Form a hypothesis. Probably a timeout misconfiguration introduced in last night’s deploy. Probably.
- 00:45 Type the first command.
Forty-five minutes from page to first action. Customers have been seeing 500s for forty-five minutes. The engineer hasn’t done anything wrong. The system around them has.
Why is this 45-minute gap so consistent across companies?
Because modern operations isn’t a visibility problem anymore. It’s a connection problem. Every team we work with has the data they need to diagnose any given incident. It just lives in six different places, owned by six different teams, accessed through six different interfaces, and humans are the only thing connecting it. The bottleneck isn’t tooling. The bottleneck is the connective tissue between tools.
Twenty years ago, the answer was “buy better monitoring.” That worked when there was one place to look. Today, an enterprise on-call engineer routinely needs to correlate signals from APM, log aggregation, distributed tracing, CI/CD systems, service registries, and incident history, plus interpret them through the lens of which downstream services depend on which. No single dashboard contains all of that. Building one is a years-long platform project most teams can’t afford.
So, the work falls to humans, in the middle of the night, with adrenaline as their main correlation engine.
What does this cost beyond the engineer’s sleep?
The diagnosis phase is the largest controllable portion of Mean Time To Resolve, and almost no one measures it separately. Most MTTR dashboards collapse the entire incident into a single duration. That hides where the time actually goes. When we instrument it properly with customers, the pattern is consistent: detection and resolution are usually fast. The middle, the part where someone is figuring out what to do, eats most of the clock.
Three concrete costs come out of that middle:
- Customer-facing minutes burned. Every minute of diagnosis on a P0 service is a minute customers feel something is wrong. For a payments flow, a streaming service, an order management system, those minutes are revenue.
- Wrong escalations. NOC operators who don’t yet know whether an alert is a blip or a major incident either escalate too early (waking people unnecessarily) or wait too long (delaying the fix). Both come from the same root cause: lack of context at decision time.
- Senior engineer attrition. The same three or four senior people get woken up over and over because they’re the only ones who can navigate the tool maze quickly. They burn out. They leave. Their tribal knowledge leaves with them.
The 45 minutes isn’t just slow. It’s expensive in ways the dashboard doesn’t show you.
Could this be fixed without replacing the tools?
Yes, and it has to be, because no enterprise is replacing Datadog plus Grafana plus GitHub plus Jenkins plus ServiceNow in a quarter. The tools aren’t the problem. The fact that humans are the integration layer between them is. The fix is a thin operational intelligence layer that reads from all of them, pulls the timeline, walks the topology, checks recent deployments, compares against past incidents, and hands the engineer a structured briefing before they pick up the page.
Not a replacement. A connection layer. Built to give engineers a head start, not to make decisions for them. Sources cited, reasoning visible, engineer still in control.
That’s the bet behind Argus, our operational intelligence layer for enterprise IT. It reads from the monitoring, deployment, and topology tools you already run, and produces a 90-second structured briefing the moment an alert fires. Same engineers. Same tools. Same authority. Forty fewer minutes of fumbling in the dark.
Where to start if you want to close the gap
Three things you can do this quarter, even before evaluating any tool:
- Measure the diagnosis phase separately. Split your MTTR dashboard into detection, diagnosis, and resolution. You’ll find that diagnosis is two to three times longer than the other two combined.
- Document the “first 15 minutes” playbook for each P0 service. Which tools, in which order, with which queries. Most teams have this in someone’s head. Get it on paper.
- Audit your tool sprawl. Count the systems an on-call engineer touches in a typical incident. If it’s six or more, you have a connection problem, not a tooling problem.
The 45-minute gap is not the cost of doing enterprise IT. It’s the cost of leaving the connective tissue between tools to humans at 2 AM. You can close it. Most teams just don’t, because they’re benchmarking the wrong part of the incident.
For a closer look at how teams are closing the diagnosis gap without replacing their existing monitoring stack, see our breakdown of where MTTR time actually goes, or book a 30-minute walkthrough and bring one real alert from your environment. We’ll show exactly what a structured briefing would surface for it.
FAQ
Detection is the time from problem occurring to alert firing. Resolution is the time from first remediation action to service restored. Diagnosis is the middle: the time an engineer spends figuring out what’s actually wrong and what to try first. It’s almost always the longest of the three, and the easiest to compress.
AIOps platforms typically focus on alert correlation and noise reduction, which is a different problem. The 45-minute gap isn’t caused by too many alerts. It’s caused by the human work of stitching together signals from monitoring, deployments, topology, and history once a single alert fires. That work needs context across tools, not better filtering inside one.
No. A properly designed operational intelligence layer reads metadata from your tools through their APIs. It doesn’t sit in the data path, doesn’t add latency to your monitoring, and doesn’t store customer data. Your stack runs exactly as it does today. The layer just makes the connections that humans were making by hand
Most teams see measurable reduction in diagnosis time within the first few weeks, once two or three of their highest-value data sources are connected. The compounding benefit comes from history: every investigation adds context for the next one.