I wanted one screenshot of a new dashboard row. Taking it stalled Grafana twice, for about four to five minutes each time. Uptime Kuma checks its health endpoint once a minute, and those checks normally answer in a few milliseconds. During the stalls they took up to 37 seconds, and one gave up after 48 and marked Grafana pending. Each time it came back on its own.

The number I came to suspect was one I’d written the night before.

The limit I gave it

Grafana, Prometheus and node-exporter ran as one compose app on a ZimaBoard with 7.5 GiB of RAM, until Grafana and Prometheus moved to a machine of their own. When I set the stack up on 19 September, I gave the three of them about 960 MiB between them, and this is Grafana’s share, from its service in the compose file:

        deploy:
            resources:
                limits:
                    memory: "402653184"

That’s 384 MiB, written in bytes. Prometheus got 512 and node-exporter 64. It looked like a sensible budget for a small board. Grafana’s own install docs recommend at least 512 MB.

Every clue said disk, and the host had memory

The numbers that explained it weren’t in Grafana. They were on the host, from node-exporter:

  • The NVMe was reading close to 1.7 GB a second, with under half a megabyte a second of writes.
  • Major page faults rose to about 950 a second on the dashboard’s five-minute average, from a usual level under 200. A major fault is the kernel going to disk for a page a program needs.
  • Nothing was swapping in. So it wasn’t memory spilling into swap.
  • More than 4.3 GB of RAM was available the whole time.

A busy disk and a host with gigabytes free reads like a disk problem, not a memory one. But a stalled process making the disk read hard with no writes at all looked to me like something re-reading what it already had.

My working explanation was repeated page-cache reclaim inside a tight container limit, and it goes like this.

A container’s memory limit counts the files it reads as well as the memory it allocates, so a tight limit can look like a disk problem long before anything crashes. The kernel charges the page cache a container fills to that container. When it hits its cap, the kernel reclaims file pages to stay under it, and those include pages of the program’s own code. The program needs them again straight away and faults them back in from disk. Clean file-backed pages can be discarded and read again when needed, so repeated reclaim and refaulting can make a container painfully slow without killing it. If reclaim can’t free enough memory, a hard limit can still trigger an OOM kill. Grafana came back on its own each time, but its compose file restarts it automatically, so that doesn’t rule a kill out, and I didn’t check for one.

That explanation rests on host-level numbers. I had no per-container metrics, so I can’t show which container was doing the reading.

What tipped it over was my screenshot

The stall wasn’t random. My notes from that morning record the capture script opening the dashboard in a browser window 3,200 pixels tall, so Grafana rendered the whole thing at once, 29 panels in five rows, instead of the handful a normal window shows. Two of those loads in a row were enough: each stall shows up in Prometheus as two bursts of reads. The screenshot was the first workload I’d tried that asked for everything at once.

One number, and the same test again

The fix was the limit, from 384 MiB to 1 GiB:

         deploy:
             resources:
                 limits:
-                    memory: "402653184"
+                    memory: "1073741824"
         depends_on:
             - prometheus

On ZimaOS a compose app’s limits are applied through its API, a PUT of the compose file, not by my deploy script, which only copies files. I saved a copy of the live compose first. ZimaOS recreated Grafana on its own, and Prometheus kept running and lost nothing.

Then I repeated what had broken it: the same two full-height loads, back to back. The commit message I wrote at the time says each finished in 15 seconds. The records that survive agree nothing stalled: in the hour after the change, Kuma’s checks answered in 3 to 8 milliseconds, and the disk never read more than 18 MB a second.

That supports the diagnosis without proving it. Raising the limit also recreated Grafana, so this wasn’t a limit-only experiment: I didn’t isolate the restart’s effect, and I still have no per-container evidence.

What I’d check first next time

Two things, and the second matters more.

  1. When a container is slow and the host has memory free, look at major faults and disk reads before CPU. High reads, no writes, no swap and memory available is a reason to look at the container’s memory pressure, not proof of it. On cgroup v2 the container’s own memory.events and memory.stat show it directly: how often it hit its limit, whether anything was OOM-killed, and how many file pages it had to read back in (workingset_refault_file). I’d look there before changing anything.
  2. Set a limit from the container’s worst load, not from a budget. 384 MiB was a share of 960, chosen before I’d ever watched Grafana render a full dashboard. The limit should come after that measurement, not before it.