This started with an Unknown Known we hit while benchmarking a new system at work: a problem every senior SRE already knows how to solve, that nobody noticed was happening.
A bill that could cover a few engineers' annual salaries
We were building a large distributed data system on Cloud Kubernetes. During benchmarking, persistent disk (PD) IOPS kept falling short, so we rebuilt the entire distributed database over and over. Kubernetes PDs have a quirk, though: deleting the database doesn't delete the PD, because the system has no way of knowing whether there's anything important left on it.
Round after round of testing quietly piled up a stack of high-IOPS PDs. They're expensive, and nobody noticed until weeks later — that one month's bill alone could have covered a few engineers' annual salaries.
Anyone who's run Kubernetes in production for a few years recognizes this instantly, and the fix is trivial: a script that finds every PD not attached to an instance. But the real question isn't "why didn't anyone write that script" — it's what this exposed about how fragmented and thin the monitoring actually was.
Two models of infrastructure governance
Most companies land somewhere between two models.
Centralized: a central IT team maintains every system and sets a single standard, covering everything with the smallest possible headcount. The tradeoff is a permissions bottleneck — every deployment goes through the same team, which becomes a serious drag whenever things are urgent or teams are spread thin. There's also a technical lock-in problem: even across multiple clouds, operating models differ wildly — GCP Kubernetes and AWS ECS aren't remotely alike — so a small central team tends to converge on one stack and pour its effort into hardening that one thing, which isn't always the right answer for a specialized service. An internal leave-request system, for instance, can run for single-digit dollars a month on AWS Lambda + API Gateway + DynamoDB; force it onto Kubernetes and the baseline cost jumps to tens or hundreds of dollars. The upside of centralization is that one set of rules covers everyone, which minimizes audit, access-control, and stability risk — because the systems are consistent, monitoring is reusable too.
Decentralized: each team spins up its own AWS account or GCP project, with full control over it. Early on this is fast and unconstrained by any particular stack — every team operates like a small startup. But the problem is just as obvious: every team now needs its own competent operations staff, every project uses different technology, and monitoring, management, and auditing rarely carry over from one project to the next. Each one is effectively built from scratch.
Why this only happens in the decentralized model
The billing incident above happened in exactly this kind of decentralized setup. Even though the team knew immediately what the problem was the moment it surfaced, monitoring simply couldn't be built as fast as resources were being created, and that gap is where the problem lives. Add high engineer turnover and project teams that spin up ad hoc — some staffed by people moving over from backend or QA roles who are drawn to new technology and want to build their management tooling on an unfamiliar stack — and monitoring coverage falls even further behind.
From the vantage point of a leadership team overseeing dozens of teams, the same category of incident keeps recurring: a dozen-plus P0s a year, many embarrassingly basic — a forgotten credential rotation, autoscaling that silently stopped working, a broken rate limiter or cache, a resource nobody was watching until it ran out and took the system down with it. New teams wanting to try new technology isn't itself the problem. The problem is the same failure mode repeating across teams that never learn from each other.
The fix isn't a better RCA — it's monitoring that appears with the resource
My take is that the real fix here isn't a more thorough RCA, or bolting on one more monitor as a surface-level patch. It's enforcing a policy: the moment a resource is created, monitoring for it exists too.
On Kubernetes this already has a working model to borrow from — Prometheus-style resource discovery, where a new pod or service shows up and monitoring picks it up automatically. For non-Kubernetes resources on GCP or AWS — RDS instances, cache servers, queues — you need a script to detect them instead. It doesn't need to be real-time: there's usually at least a 24-hour gap between a resource being created and actually going live, so a daily scan that back-fills the monitoring system with anything new is enough. The same logic applies to DNS — the moment a new HTTPS endpoint gets a DNS record, certificate-expiry monitoring and a health check should get created for it automatically.
Not glamorous, but it actually works
None of this is exciting, but it's practical: the moment one team using this system hits a P0 and a general-purpose monitor gets added to cover it, every other team benefits from it going forward. There's a condition attached, though — every monitor you add has to account for compatibility with every other team using the system.
The moment each project goes back to doing its own thing, this falls apart, and specialized, one-off monitoring systems reappear. Managing this technology was never about locking everyone into one stack. It's about whether the management system you're building can account for every project, whether it considers how other teams' setups fit into it, and whether all the monitoring that was painstakingly built up before is still covered when you're done.