Resource requests are load-bearing

Add a node to a Kubernetes cluster and it can fill up almost immediately. Not with the workloads you were planning to move onto it, but with a scattering of small pods that had been running happily elsewhere for months. Nothing had been rescheduled on purpose. Nothing had failed. The scheduler had simply decided that the new, empty node was the best place for a set of pods it believed were free.

They were free, as far as the scheduler was concerned. None of them declared resource requests.

The scheduler does not look at usage

This is the part that surprises people, and it surprised me for longer than I would like to admit. When Kubernetes decides where a pod goes, it does not measure how much memory or CPU that pod actually consumes. It reads spec.containers[].resources.requests and treats that number as the truth.

A container with no requests declares nothing, so it is scored as costing nothing. It fits anywhere. It fits on a node that is already running at ninety percent of its real memory, because as far as the scheduler’s arithmetic goes, that node still has its full allocatable capacity available.

Now combine that with the default scoring behaviour. Kubernetes prefers the least allocated node, on the reasonable theory that spreading work out is better than piling it up. But “least allocated” is computed from requests, not from usage. A brand new node has zero requests against it, so it wins every scoring round until something with real requests lands on it.

Put those two behaviours together and you get that outcome. Request-less pods gravitate toward whichever node is emptiest on paper. In practice that means they gravitate toward the node you just added, which is exactly the node you were trying to keep free.

What it costs

The count is usually higher than expected. On one cluster, fifteen pods had no requests declared at all. Their actual combined usage was roughly 1.4 GB of memory. That was 1.4 GB the scheduler did not know about, distributed across nodes according to a model that said it did not exist.

None of it had caused an outage, which is what makes it easy to leave alone. The cluster looks balanced in every dashboard that reports requests, and is quietly not balanced at all.

The second problem: QoS class

Requests do not only drive scheduling. They also decide which of three quality of service classes a pod lands in, and that class decides who gets killed when a node runs out of memory.

• Guaranteed: every container sets requests and limits, and they are equal. Evicted last.
• Burstable: requests are set, and are lower than limits. Evicted in the middle, ordered by how far above its request the pod is running.
• BestEffort: no requests and no limits at all. Evicted first, before anything else on the node.

Every one of those fifteen pods was BestEffort. So the same omission that caused them to crowd onto the emptiest node also guaranteed they would be the first thing killed when that node came under pressure. The failure mode is self-inflicted on both ends: you concentrate them where the headroom is thinnest, and then you make them the first casualties of the thing you caused.

Sizing requests without guessing

The advice to “just set requests” is easy to give and annoying to act on, because the obvious question is what number to use. Two rules got me most of the way.

Size requests from observed usage, not from fear. Look at what the workload actually consumes in steady state and set the request near that, with a little headroom. Requests are a reservation. Every megabyte you request is a megabyte no other pod can be scheduled against, whether or not you ever touch it. Inflating requests to feel safe is how you end up buying a node you did not need.

kubectl top pods -A --sort-by=memory

# and the inverse: everything that declares nothing
kubectl get pods -A -o json | jq -r '
  .items[] | select(
    [.spec.containers[].resources.requests // {}] | map(length) | add == 0
  ) | "(.metadata.namespace)/(.metadata.name)"'

Limits can be generous. Requests cannot. A limit costs nothing at scheduling time. It is a ceiling that only matters when a container tries to exceed it. So there is little downside to setting a limit at two or three times the request, which gives a workload room to handle a burst without letting a runaway process take the node down with it.

There is one important exception. If a process has its own internal memory ceiling, setting the container limit equal to that ceiling will get it killed before its own eviction logic ever runs. That is a specific enough trap that it deserves its own post, and I have written one.

The rule I settled on

Every deployment declares requests. Not because every workload is important, but because a workload that declares nothing is invisible to the only model the scheduler has. An unimportant pod that lies about costing zero does more damage to placement decisions than an important one that tells the truth.

initContainers are the easiest thing to miss. They are short lived, but they are scheduled against the same node as the pod they belong to, and while they run they are just as invisible as anything else.

If you want one thing to check after reading this, run the second command above. The output is usually shorter than you expect and more interesting than you want.