By the time I got into this line of work, nobody was installing an operating system on bare metal anymore — but we also weren't yet in the era where you push code and someone conjures a machine for you. I landed right in the middle, and worked my way through virtual machines, cloud, containers, and serverless. Looking back, every time the infrastructure grew another layer of abstraction, I had one less layer I could actually touch, and what monitoring cared about drifted up a layer along with it.
A painful bit of ops grunt work
Back then, standing up an internal service went like this: buy physical machines, install vSphere to virtualize the hardware, carve out virtual machines, then install the operating system and the application on each one by hand. Networking was your problem too — if you wanted load balancing, you stood up HAProxy or something like it and split traffic across the machines behind it. As for deployment automation, that's a separate story for another day — but the two are more related than they look, because without monitoring you can trust, automated deployment is just shipping with your eyes closed.
In that environment, what monitoring covered was entirely up to you, because nobody else was going to cover it.
Monitoring, stacked layer by layer
The first thing you watch is, naturally, OS-level resources: CPU usage, memory free space, disk free space, disk IOPS, network bytes and packet count. This layer answers a simple question — does this machine still have the strength to do work.
But a machine with strength doesn't mean the service is still there. So one layer up, you look at the application itself: is the process count right, did one quietly vanish or did a pile of them appear, are the memory and cpu usage of individual processes sane, is the port the service is supposed to be listening on actually open. This layer answers — is the thing still there.
And the thing being there doesn't mean it can still do work. So the layer above that is the health check probe: fire a request at it and see whether it answers properly. This layer answers — is it still willing to talk to anyone.
The genuinely valuable layer, though, is the one above that: digging through the application's own telemetry for the metrics that mean trouble once they cross a certain value, and monitoring those. This layer means something completely different from the ones below it — everything below only lights up once the problem has already happened, while this one hands you a warning before anything actually blows up.
Last is the end-to-end interface facing the outside world: write a script that simulates what a user would really do, then check whether what comes back is correct. Every layer below can be green and this one can still be red — and this layer is the only one a user ever feels.
Thresholds are hard to get right
The genuinely awkward part of all this isn't collecting the numbers, it's deciding where the thresholds go.
The same single rule — alert when CPU usage goes over 90% — means completely different degrees of urgency on a machine with many cores versus one with few. Disk IOPS is the same; the hardware underneath differs enormously in what it can do. So the hardware's own specs have to be treated as an input parameter to the monitoring, rather than applying one set of numbers across the entire machine room.
Another approach is to look at averages instead, and there's a trap here that's very easy to step into. Averaging CPU usage by core sounds perfectly reasonable — but suppose an eight-core machine is actually running only two processes, and those two saturate their own two cores while the other six sit idle. The average still comes out low, and the dashboard looks entirely normal. As far as that service is concerned, though, it's full. CPU load is more accurate in a case like that, but load has its own room for misreading, and you still have to read it against the process count and the core count together. One number on its own is never enough.
Then there's splitting thresholds into warning and critical, so the alert itself carries its urgency, instead of everything dragging someone out of bed with the same level of panic. And one very practical thing: momentary spikes cause false alarms constantly, so pick a mechanism with flapping detection and only fire a notification after several consecutive breaches — a few nights of being woken up over a single spike and you'll know exactly what that feature is worth.
The two-minute boot era
Moving into the cloud virtual machine era, the most immediate change was boot time. A system that comes up in two minutes means a full scale-out can finish in two or three.
That changed the nature of alerting entirely. A CPU spike alert used to exist to wake up whoever was on call, and that person got up and started machines. Now the alert becomes the trigger event for scale-out directly — the machines grow on their own and nobody has to wake up. That's how a large chunk of on-call time got cut.
The focus of a release shifted with it. The most important part of shipping back then was turning a validated virtual machine into an image, so the auto scaling system could deploy new machines straight from it, and then making sure those new machines got added to the monitoring system automatically once they came up. Skip that step and machines are created and destroyed faster than monitoring can cover them — which is exactly where the bill I wrote about in Unknown Known: When Monitoring Can't Keep Up With Resource Creation came from.
Containers arrive
Maintaining virtual machines still cost real people real time, though. The machines could grow on their own now, but what ran on them — which package versions were installed — was still unsolved. Application dependencies were routinely the reason a system went haywire after an upgrade, and it usually went haywire at the least convenient possible moment.
Then containers showed up. They pack the application and every one of its dependencies into a single image, starting an image takes seconds, and a large share of both maintenance cost and dependency pain disappears at once. Auto scaling became the mainstream approach from then on — because taking two minutes to add a machine and taking two seconds to add an instance are two completely different operating models.
The unit of monitoring shifts
Monitoring followed, moving to the container as its unit rather than the virtual machine, because a single virtual machine can run several containers and the machine is no longer the smallest unit that means anything.
Container management grew into a discipline of its own — Kubernetes, ECS, Cloud Run — abstracting auto scaling and CPU and memory resource management into one layer, then separating virtual-machine-level CPU, memory, and disk scaling from container-level scaling: overall resource monitoring decides whether to add a virtual machine, while the container level has its own HPA (horizontal pod autoscaling) that scales on the real load of each individual container, which is to say each individual application. Inside one cluster, the two layers scale independently of each other.
And it's precisely because container management got abstracted — because you can pull the list of container instances straight from an API — that monitoring at this layer became feasible at all. Prometheus lives at exactly this layer: it pulls container resource metrics through the metric server, and defines a uniform telemetry output format that standardizes collection itself. The values it collects can be run through a query combined with functions, and from those results you set warning and critical thresholds and wire them to notifications, or wire them straight to scaling conditions.
The birth of SLI, SLO, and SLA
Further along, even instance-level management became more trouble than it was worth. So the cloud providers wrapped this now-mature technology in one more layer and turned it into managed services — Fargate for ECS and EKS, Cloud Run — pushing developers as close to the application layer as possible so they could concentrate on the application itself.
This is exactly where SRE's SLI, SLO, and SLA framework comes from. When none of the machines underneath are yours to manage, the only thing you can define — and should define — is the health of the service itself. First the SLI (service level indicator), things like API latency and API response status code. Then the SLO (service level objective) you want to hold, like keeping average response time under 500ms. And only then does that service quality go into a contract as an SLA (service level agreement), where daily, monthly, and yearly average downtime converts into the percentage you commit to publicly — 99.99%, say, which works out to under 53 minutes of downtime a year, with compensation clauses if you go over.
Monitoring's role changed here too. It's no longer watching the machines on your behalf; it's continuously proving that the promise you made to the outside world is being kept.
Lambda and how it feels to the user
Later still, even container-level maintenance became too much trouble, and stateless services could just run as Lambda functions, with AWS scaling for traffic on its own. The developer can't even get at the concept of a machine anymore. What's left worth caring about is the user's actual experience. SLI and SLO are still the standard way to measure it — those standards just move as close to the user's perception as they can, with metrics like Apdex that convert response time into a satisfaction score.
What monitoring tracks is control
From that vSphere virtual machine you had to install an operating system on yourself, to a Lambda today that scales itself the moment you push, every era abstracted away the most painful thing about the era before it: the creation and destruction of machines went to auto scaling, dependencies went to the container image, resource scheduling went to orchestration, node maintenance went to managed services.
And the center of gravity of monitoring drifted upward the whole way, following whatever the developer could still touch. You used to watch disk IOPS because that disk was sitting physically in front of you, and because nobody else was going to look at it. You don't watch it now — not because it stopped mattering, but because it stopped being yours. And on the day none of it is yours, the only thing monitoring has left is whether the user is having a good day.