The 03:37 alert that was never sent

At 03:35 on 20 September, the public URLs of both my Railway-hosted apps started returning 522 errors. Uptime Kuma logged Request failed with status code 522 for each of them within a second of each other, held them as pending for one retry, marked both down at 03:37 and saw them back at 03:38. This site, on Cloudflare Pages, never blinked.

Both apps down at 03:37 and back at 03:38. Their names are painted out.

This is exactly the event every site is monitored twice was built for, and the design earned its keep the first time it met a real one. A 522 is Cloudflare saying it couldn’t reach the origin, which on its own reads as “Railway was down”. But the origin monitors, which go straight to each app’s Railway address and skip Cloudflare entirely, stayed green throughout: 715 checks in twelve hours, every one up. That points at the path between Cloudflare and Railway rather than the apps going down, though checks once a minute from one house can miss a brief failure, so it’s a strong lead and not a proof.

Then I wrote up the morning, and one line of it was wrong.

I described the alert from the configuration

“The alert went out at priority 5, at 03:37.” Priority 5 is ntfy’s highest, and I went on to wonder whether a three-minute blip in the small hours deserved it.

It didn’t go out. Nothing had, in two days. I only noticed because it had been suspiciously quiet: no message about the 522, none when Immich was down for an hour during its move that same day, none for two reboots. ntfy.sh keeps a short history for each topic, and mine held exactly one message for the previous 24 hours: the test I sent after the fix.

I hadn’t seen an alert arrive. I’d seen a notification configured and a monitor go red, and I filled in the step between them because the configuration said it would happen. The published post makes the same leap: “Alerts go to an ntfy topic.” That sentence was false on the day it went up.

Seventeen monitors, attached to nothing

The notification existed and was configured correctly. It was attached to 0 of 17 monitors.

My monitors are declared in a Python script rather than clicked into the web UI, and this is how it created the notification:

created = api.add_notification(
    name=name,
    type=NotificationType.NTFY,
    isDefault=True,
    applyExisting=True,
    ntfyserverurl="https://ntfy.sh",
    ntfytopic=NTFY_TOPIC,
    ntfyPriority=5,
    ntfyAuthenticationMethod="none",
)

isDefault and applyExisting both read like “use this everywhere”, and neither did anything here. “Apply to existing” ran once, when the notification was created, before a single monitor existed. “Default” only pre-ticks a box in Uptime Kuma’s web form; a monitor created over the API gets no notification unless one is passed in explicitly. The monitors themselves were created with a plain api.add_monitor(**spec) that never mentioned a notification. So every one was silent from the start, and the dashboard showed nothing wrong, because a monitor with no notification is a perfectly valid monitor.

Monitoring that has never been seen to alert isn’t monitoring yet. The configuration describes what should happen. Only a message on the phone shows that it does.

The check that was missing

The fix is one line where each monitor is created:

spec["notificationIDList"] = [notification_id]
created = api.add_monitor(**spec)

The existing 17 were repaired the same way, and then the part that counts: a check at the end of every run that refuses to finish while any monitor is silent.

def assert_alerting(api: UptimeKumaApi, notification_id: int) -> None:
    """Fail loudly if any real monitor still cannot alert."""
    silent = [m["name"] for m in api.get_monitors()
              if m["type"] != "group" and notification_id not in (m.get("notificationIDList") or [])]
    if silent:
        raise SystemExit(f"these monitors have no notification attached: {silent}")
    print("  every monitor has the notification attached")

Groups are left out on purpose: a group goes down whenever a child does, so it would only ever send duplicates. The next time I add a monitor and forget, the script fails instead of quietly succeeding, and the gap can’t reopen through the same door.

Its docstring claims more than it checks, though. It confirms the notification is attached, not that anything arrives, which is exactly the gap this post is about. Attachment is the part a script can check on every run. Delivery took a different test.

Then I tested each link on its own, because each one can fail on its own. Kuma’s test reported Sent Successfully, the message was sitting on ntfy.sh, and it reached the phone. The first real alert came that same afternoon, when Paperless went down from 13:52 to 13:58 during its move, and this time my phone knew about it.

This isn’t the first post here to be undercut within a day by the very thing it described. The SQLite trap caught me the same day I wrote about it, and the fix there was the same shape: stop trusting the setting, and check the thing it was supposed to cause.

If you run monitoring at home, when did it last actually reach you? Not the dashboard turning red, but a message on the device you’d have in your hand at 3am. If you can’t name the time, find out on purpose: point a temporary monitor at an address that can’t answer, confirm the alert reaches your phone, then fix the address and check the recovery message arrives too.