Never set a Redis memory limit equal to its maxmemory

A Redis instance was OOMKilled twelve times in a row. It had a 256 MB memory limit and a maxmemory of 256 MB, an eviction policy that should have kept it comfortably inside that, and no obvious reason to die. It died anyway, restarted, filled up, and died again.

The two numbers being equal is the entire bug.

Why matching them is wrong

Setting maxmemory equal to the container limit looks like the careful thing to do. You are telling Redis exactly how much room it has, and telling the kernel the same number. Nothing is wasted.

The problem is that maxmemory does not mean “the amount of memory this process will use”. It means “the amount of memory Redis will count toward its own accounting of stored data”. Those are different numbers, and the second one is always larger.

What sits outside that accounting includes client output buffers, replication buffers if you have them, the copy-on-write pages during a fork for persistence, allocator fragmentation, and the process itself. Fragmentation alone routinely runs at ten to twenty percent above the reported dataset size, depending on your workload’s key and value sizes.

So the sequence goes like this. Redis fills toward maxmemory. Its own bookkeeping says it is approaching the ceiling and eviction should begin shortly. But the resident set size the kernel sees crossed the cgroup limit some time ago, because of everything in the paragraph above. The kernel does not wait politely for Redis to finish its own accounting. The OOM killer fires, the container dies, and it restarts empty.

Then it fills up again. Twelve times, before anyone looked at it properly.

Eviction never gets a turn

The detail worth sitting with is that the eviction policy is not broken. It is never reached. Whatever you configured, allkeys-lru or volatile-lru or anything else, is a strategy for staying under maxmemory, and the process is being killed before that strategy has any work to do.

This is why the symptom is confusing. You configured eviction. You can see the policy is set. The instance still dies as if it had noeviction. The logs show a clean start and then nothing useful, because being OOMKilled is not something a process gets to write a log line about.

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# OOMKilled

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount}'
# 12

If you see OOMKilled with a rising restart count on something that has an eviction policy, this is almost always what happened.

The fix

Give maxmemory real headroom under the container limit. I use half, which is more conservative than strictly necessary and costs nothing I care about on a cache.

commonConfiguration: |-
  maxmemory 128mb
  maxmemory-policy allkeys-lru
  save ""
  appendonly no

master:
  resources:
    requests:
      memory: 128Mi
    limits:
      memory: 256Mi

If halving feels wasteful, the usual guidance is to leave at least twenty five percent, and more if you fork for snapshots, because copy-on-write during a background save can add a great deal depending on write volume. On a cache instance I disable persistence entirely, which removes the fork from the equation and makes the remaining overhead much easier to reason about.

The other half of the fix is watching the right number. used_memory is what Redis counts. used_memory_rss is what the kernel counts, and it is the one that gets you killed.

redis-cli INFO memory | grep -E 'used_memory_human|used_memory_rss_human|mem_fragmentation_ratio'

A fragmentation ratio meaningfully above 1 tells you how much of a gap you are dealing with. That is the gap your headroom has to cover.

It is not really about Redis

The general shape of this applies to anything that manages its own memory ceiling inside a cgroup limit. A JVM with -Xmx set to the container limit will be killed before it can garbage collect its way out of trouble, because heap is not the only thing in a JVM process. The same reasoning covers PHP-FPM worker counts multiplied by memory_limit, and any runtime with a configurable arena.

The rule is the same in each case. The number the application uses to decide when to start managing its own memory has to be below the number the kernel uses to decide when to kill it. If they are equal, the application’s own protection never runs.

Worth checking your chart defaults, incidentally. A packaged chart will happily let you set both values to the same number, because it looks tidy.