Sentinel: Out-of-Band Monitoring on Google Cloud

A watchdog that lives on the machine it watches can only report problems the machine survives. Lose power, lose the network, or have a reboot land somewhere unexpected, and the work stops and the alarm stops together. Nothing rings. From the outside, a studio humming along and a studio that died at two in the morning look exactly the same: quiet.

The machine does not log in by itself, and most of the automations need a logged-in session to run. So the shape of a bad night is this: the power flickers, the machine reboots the way it is supposed to, and it comes back sitting at the login screen. The hardware is fine. It answers pings, the fans are spinning, every simple up-or-down check in the world says green. And eighty-odd jobs are dead and will stay dead until somebody walks over and types a password. An ordinary heartbeat would have sailed straight through that. So ours does not just say the machine is alive. It says whether anyone is actually logged in and how many jobs are loaded, and if that number falls through the floor while the machine keeps answering, that is its own alarm.

From the cloud, a dead machine and a dead internet connection look identical. There is no way to tell them apart from the outside, so the alert does not pretend to. It says there has been no heartbeat from the studio in the last fifteen minutes, which is exactly true, and it never says the studio is down, which would be a guess. Small thing, but it decides whether you trust the next one.

Here is the uncomfortable part of building a thing that alerts you: working perfectly and being completely broken both look like silence. If the webhook gets revoked on a Tuesday, nothing happens, and nothing keeps happening, and you feel fine about it. So every Monday morning it posts proof of life to the same channel a real alert would use, down the same path, through the same credentials. If that message does not arrive, the alerting is broken and you know within a week instead of finding out during an actual outage. Underneath that sits a second alarm run by Google's own monitoring, on infrastructure we do not control and code we did not write, watching for our watcher going quiet. It sends email rather than chat, on purpose. If chat is the thing that broke, an alert about it over chat never arrives.

Testing a fire alarm by waiting for a fire is a bad plan. There is a command that deliberately makes the studio go quiet for a set window, in production, so you can sit and watch a real alert land on a real phone and then watch the recovery message follow it. The mute is capped, so a drill somebody forgot to end cannot silence the machine indefinitely, and it clears itself and starts reporting again on its own.

When the machine comes back, the useful question is not that it went down, it is what did not run while it was gone. Each job reports its runs outward as it goes, so the recovery message can hand you the actual list of names: what should have run during the window and did not. That used to be an hour of reading logs and guessing.

The whole thing is defined as code and rebuilt from that definition rather than clicked together in a console, and it deploys without a stored password existing anywhere for someone to find. It costs a couple of dollars a year to run. The original watchdogs all still run, because they catch different things. Sentinel exists for the failure none of them could ever report: the machine going quiet, and nobody hearing about it.