Never set a Redis memory limit equal to its maxmemory

A Redis instance was OOMKilled twelve times in a row. It had a 256 MB memory limit and a maxmemory of 256 MB, an eviction policy that should have kept it comfortably inside that, and no obvious reason to die. It died anyway, restarted, filled up, and died again.

The two numbers being equal is the entire bug.

Why matching them is wrong

Setting maxmemory equal to the container limit looks like the careful thing to do. You are telling Redis exactly how much room it has, and telling the kernel the same number. Nothing is wasted.

The problem is that maxmemory is an eviction threshold, not a ceiling on the Redis process’s resident memory. Redis accounts for memory allocations, with exclusions such as replication and AOF buffers. Resident memory can be higher because of fragmentation and other overhead.

Replication and AOF buffers, allocator fragmentation, and copy-on-write memory during persistence can all affect the headroom needed. The amount depends on the workload and configuration; dataset size alone is not enough to choose a container limit.

So the sequence goes like this. Redis fills toward maxmemory. Its own bookkeeping says it is approaching the ceiling and eviction should begin shortly. But the resident set size the kernel sees crossed the cgroup limit some time ago, because of everything in the paragraph above. The kernel does not wait politely for Redis to finish its own accounting. The OOM killer fires, the container dies, and it restarts empty.

Then it fills up again. Twelve times, before anyone looked at it properly.

Eviction never gets a turn

The detail worth sitting with is that the eviction policy is not broken. It is never reached. Whatever you configured, allkeys-lru or volatile-lru or anything else, is a strategy for staying under maxmemory, and the process is being killed before that strategy has any work to do.

This is why the symptom is confusing. You configured eviction. You can see the policy is set. The instance still dies as if it had noeviction. The logs show a clean start and then nothing useful, because being OOMKilled is not something a process gets to write a log line about.

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# OOMKilled

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount}'
# 12

If you see OOMKilled with a rising restart count, compare the container limit, maxmemory, and observed memory usage. Equal limits are one likely cause, but bursts, persistence, fragmentation, and other memory pressure also need investigation.

The fix

Give maxmemory real headroom under the container limit. I use half, which is more conservative than strictly necessary and costs nothing I care about on a cache.

commonConfiguration: |-
  maxmemory 128mb
  maxmemory-policy allkeys-lru
  save ""
  appendonly no

master:
  resources:
    requests:
      memory: 128Mi
    limits:
      memory: 256Mi

The 50% setting is my choice for this cache, not a universal sizing rule. Measure memory under representative load and during persistence or replication activity. This example disables persistence and is suitable only where cached data can be rebuilt.

Watch both Redis and container metrics. used_memory reports Redis allocation accounting; used_memory_rss reports resident process memory. Container memory accounting determines enforcement of the cgroup limit and can include more than that process RSS.

redis-cli INFO memory | grep -E 'used_memory_human|used_memory_rss_human|mem_fragmentation_ratio'

A fragmentation ratio meaningfully above 1 tells you how much of a gap you are dealing with. That is the gap your headroom has to cover.

It is not really about Redis

The general shape of this applies to anything that manages its own memory ceiling inside a cgroup limit. A JVM with -Xmx set to the container limit will be killed before it can garbage collect its way out of trouble, because heap is not the only thing in a JVM process. The same reasoning covers PHP-FPM worker counts multiplied by memory_limit, and any runtime with a configurable arena.

The rule is the same in each case. The number the application uses to decide when to start managing its own memory has to be below the number the kernel uses to decide when to kill it. If they are equal, the application’s own protection never runs.

Worth checking your chart defaults, incidentally. A packaged chart will happily let you set both values to the same number, because it looks tidy.

See Redis’s eviction documentation for how maxmemory and excluded buffers interact.