RabbitMQ in Kubernetes: a memory alarm that can never fire, and a config that never loads
Three quiet defaults of a RabbitMQ Helm chart: the memory watermark is computed from the node, not the container, so the OOM killer always wins; a ConfigMap change does not restart the broker; and the volume is deleted together with its PVC.
I was reviewing a RabbitMQ broker that runs inside a Kubernetes cluster from a community Helm chart, before the environment went live. Nothing was broken yet. That is the best moment to look, because all three findings below are invisible until the day they cost you the message bus.
The memory alarm that cannot fire
RabbitMQ has a memory high watermark: when the broker reaches it, it blocks publishers so that consumers can drain the queues. It is the safety valve that should act before memory runs out.
In this chart memoryHighWatermark is disabled by default. Without it, RabbitMQ takes 0.6 of the memory it can see — and inside a container it sees the node’s memory. On an 8 GB node that is about 4.9 GB, while the container limit was 2 GB.
So the valve is set above the ceiling. The kernel OOM killer always gets there first, and the broker dies instead of slowing publishers down.
config:
memoryHighWatermark:
enabled: true
type: relative
value: 0.6
Two details make or break this:
- The key lives under
config, not at the top level. In the chart’svalues.yamlit sits under a section comment that makes it look like a root key. With the wrong nesting Helm silently ignores the block: the render does not change, Argo CD showsSynced, behaviour stays the same. Only a diff of the rendered manifests tells the truth. - With
type: relativethe chart also writestotal_memory_available_override_valuefromresources.limits.memory. No limit, no override line, and the threshold drifts back to node memory.
Verify on the broker itself:
kubectl exec rabbitmq-0 -c rabbitmq -- rabbitmq-diagnostics status | grep -i 'high watermark'
A config change that does not apply
The StatefulSet has no checksum annotation for the ConfigMap and no reloader watching it, and rabbitmq.conf is read only when the process starts. After a sync the file on disk is new, the broker runs with the old one — and it looks exactly like “my change did not apply”.
kubectl rollout restart sts rabbitmq
With a single replica that restart is a bus outage: publishers and consumers reconnect. On an environment without traffic the cost is zero, which is why such changes belong before launch. Durable queues survived the restart without loss; messages in non-durable queues do not.
A volume that leaves with its claim
The broker’s PVC was created on a StorageClass with reclaimPolicy: Delete. Deleting the PVC — a careless helm uninstall, a prune in Argo CD — deletes the disk with the bus on it.
The policy can be changed on the existing PV, without downtime and without recreating anything:
pv=$(kubectl get pvc data-rabbitmq-0 -o jsonpath='{.spec.volumeName}')
kubectl patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
The other side of it: a retained volume stays in the cloud and keeps costing money until someone deletes it by hand. For a message bus and for logs that is the right trade, but keep a list. I did the same for the other stateful volumes of that environment.
What this does not solve
One replica with classic queues is still a single point of failure. Moving to quorum queues is a job for the application — the queue type is declared by the client, not by the chart. That went to the developers as a separate task.
Takeaways
- A memory threshold computed from the node is no threshold in a container. Set it explicitly and tie it to the container limit.
Syncedin Argo CD only means the render matches Git. A misplaced values key renders to nothing and stays green. Check the rendered diff.- If a config file is read only at startup and nothing restarts the pod on change, every config change needs a deliberate restart.
- Check
reclaimPolicyon every stateful volume before go-live.Retaincan be set on a live PV in one command. - Do these changes while the environment has no traffic: the same restart is free today and an incident next month.