The Flask app I maintain runs on Railway, and the rest of my monitoring lives in Prometheus and Grafana on a ZimaBoard at home. Railway’s dashboard shows request counts, latency, CPU and memory, and its GraphQL API will hand you the same figures if you ask. But there’s no /metrics endpoint to point Prometheus at, so the one service I most wanted on the dashboard was the one that wasn’t there.

There is a community Prometheus exporter for Railway, and it covers CPU, memory and network. Requests aren’t in it. So on Saturday 10 October I wrote railway-bridge: one Python file, standard library only, that polls Railway’s GraphQL API every five minutes and serves what it learns at /prometheus.

Most of it is plumbing. The part worth writing about is the request counter, because the obvious version is wrong in three ways.

Railway gives you minutes, Prometheus wants a counter

Railway answers httpMetricsGroupedByStatus with per-minute buckets, one list per status code, for whatever window you ask about. Prometheus wants a counter: a number that only goes up, so that increase() and rate() can work out the traffic between any two scrapes. The obvious bridge asks for the last few minutes on each poll and adds them to a running total.

  • The windows overlap. Each poll asks for the last ten minutes: five for the interval, three for the lag below, and two to spare for a poll that runs late. That means every minute arrives in two polls, and adding up whatever comes back counts most of them twice.
  • The current minute is still running. I checked this on Sunday morning by reading the same quarter-hour a few times over. The minute starting at 08:56 came back at 08:56:17 with 31 requests in it. Read again once it had ended, it had 78. Count it at 31 and either the other 47 are never counted or, if the next poll adds the minute again, the first 31 are counted twice.
  • A restart replays the window. Starting from zero after a restart is fine, because Prometheus reads a counter that drops as a reset and carries on. Counting the whole look-back window on the first poll is not: ten minutes of traffic land in one scrape, and increase() shows a burst that never happened.

A high-water mark per status code

The counter keeps one thing per status code: the newest minute it has already taken.

def add(self, answer: list[dict], now: int) -> None:
    cutoff = now - self.lag
    for group in answer:
        code = group.get("statusCode")
        seen = self._seen.get(code, self.start - 1)
        newest = seen
        for sample in group.get("samples") or []:
            ts = sample["ts"]
            if seen < ts and ts >= self.start and ts + self.step <= cutoff:
                self.totals[status_class(code)] += int(sample["value"])
                newest = max(newest, ts)
        self._seen[code] = newest

Each condition answers one of the three problems. seen < ts skips a minute this status code has already counted, so the overlap costs nothing. ts >= self.start skips anything older than where this process begins counting. And ts + self.step <= cutoff takes a minute only once it has ended and a three-minute lag has passed, so a half-full minute is left for a later poll.

It’s per status code rather than per minute for a reason the tests spell out. A minute whose 200s were counted on one poll can show its first 500 on the next, and a single high-water mark for the whole minute would drop it.

Where a process begins counting is set when it starts, three minutes back:

self.counter = RequestCounter(start=int(time.time()) - LAG)

Three minutes back is the line the previous process could never have crossed. It only ever took minutes older than the lag, so everything from there on is new, and anything before it is skipped rather than replayed. That’s also why the comparison further down starts at 11:59 for a restart at 12:01.

The restart rule has a price. The minutes between the last ones the old process took and that line are never counted at all. Running the real counter over a simulated minute-by-minute stream, at every point in the poll cycle, that came to one to six minutes, depending on how long after its last poll the bridge restarted, and more if it stays down past the poll it would have made next. I’d rather lose a few minutes than invent ten.

The fixed window has a price too. Each poll asks for the ten minutes before now, not for everything since the last minute it took, so a failed poll leaves a hole: in the same simulation, one failed poll lost three minutes and two lost eight. The bridge doesn’t backfill.

A per-minute figure from an API is a reading, not an event: count each minute once, and only after it’s over.

The lag is a margin I chose, not one I measured. It delays every count, and I don’t know how late Railway can publish a minute. In my reads, finished minutes didn’t change. If one ever did after the counter had moved past it, the change would be ignored. Separate marks let a status code that turns up late add an older minute, as long as its own mark hasn’t already passed that minute. They don’t recover a revision to a minute that’s already been counted.

The counter agreed with Railway to the request

The tests check the logic on made-up minutes. The question that matters is whether it agrees with Railway on real ones, so on Sunday I compared the two.

The bridge had been redeployed at 12:01 UTC on Saturday, which Prometheus recorded as the counter’s only reset in the stretch compared here: the 2xx count went from 8 to 0, as designed. From the first minute the restarted bridge would count, 11:59, to the last minute its 08:52 poll could take the next morning, 08:48, Railway’s own per-minute figures add up to 9,344 requests. The counter read 9,344.

No poll failed in that time: railway_bridge_up never dropped to 0. So it’s one service over 21 quiet hours, not a proof for every case. But a counter that double-counted the overlap would have read nearly twice that, and one that took half-full minutes would have been off by the difference. It was off by nothing.

What matches is the total, not the shape. The counter moves in steps, once a poll, three to nine minutes behind the traffic. rate() over five minutes is a staircase, not the minute-by-minute pattern Railway’s own graph shows. Longer windows make the steps less noticeable, but they don’t recover when the requests actually happened. The exact match above compares running totals over the same minutes of Railway’s data, not a PromQL increase() over an arbitrary hour, which extrapolates from its samples and can give a fraction from a counter of whole requests.

Uptime Kuma already tells me whether the app answers. This is the other half: how much it served, and how much of that was errors.

The real bug was in the loop around it

A security review of the code before it went public found a bug I’d missed. The poll loop caught a fixed list of errors:

except (urllib.error.URLError, RuntimeError, KeyError, ValueError, TimeoutError) as error:

urlopen wraps most network failures in URLError, but not all of them. If the server closes the connection without answering, the response is read outside that wrapper, and Python raises http.client.RemoteDisconnected, which is none of the five. One dropped connection would have ended the polling thread for good, while the server thread carried on serving the last figures it had, with railway_bridge_up still at 1.

The fix catches everything and keeps polling:

# Anything, not a list: urlopen lets a dropped connection through as
# RemoteDisconnected, and one uncaught error would end polling for good.
except Exception as error:  # noqa: BLE001

It wouldn’t have stayed hidden long. /health checks the age of the last good poll rather than the up flag, and Uptime Kuma watches it, so the frozen figures would have shown up as a 503 within fifteen minutes. But a metric that says everything is fine while it’s frozen is exactly the wrong way to fail. The fix keeps the bridge polling; it doesn’t recover the minutes a failed poll skipped, for the reason above. The fix went onto the board on Sunday evening, after the comparison, so the 21 hours above ran the version before it, with the same counter.

If you’re turning someone else’s per-minute figures into a counter, check it the way you’d check money. Add up their numbers over the same stretch and see whether yours match. Mine did, to the request, over one stretch. The tests show the logic does what I meant; that check is what showed I’d meant the right thing.