Profiling a Python memory leak in production with memray — and the outage we caused on the way

On a product platform at Gambchamp one Python backend kept getting OOM-killed. The limit was 1 GiB, the pod lived 10–15 hours, crept up smoothly and died with exit code 137. The developers needed actual lines of code, not “it grows”, so we decided to profile it with memray on the live production pod — one replica, real traffic.

Measuring the slope first

Before attaching anything we measured the leak from the outside. Retained memory grew about 28 MB per hour, anonymous memory about 48 MB per hour (310 → 503 MB in four hours). From a start of roughly 350 MB to a 1 GB limit that is about ten hours — exactly the interval between OOMs. So this was one steady leak, not a spike.

The first attempt took production down

We patched the Deployment to start the app under memray run, borrowing a trick from an old runbook: keep the container alive with sleep infinity and drop the liveness probe so the profiler would not be killed.

memray crashed with a segfault (--trace-python-allocators does not get along with greenlet and asyncpg). The application died with it — but the container stayed “alive” in sleep, so kubelet saw nothing to restart. Production was down for about seven minutes.

The fix was simple and is now the rule: the wrapper ends with exec python main.py, and the liveness probe stays. If the profiler dies, the container dies, and Kubernetes does its job.

More traps from the same evening:

• --aggregate mode keeps the aggregation in process memory, so after the segfault the output file was zero bytes — twelve minutes of profiling lost. On production services use the plain mode only.
• The default startup probe killed the first pod: under memray the start is slower.
• The HPA added a second replica because of the CPU our own snapshot uploads used, and later removed a pod on scale-down. controller.kubernetes.io/pod-deletion-cost on the profiled pod protects it without touching autoscaling.

Hourly snapshots instead of one long run

A truncated memray file is still readable, so there is no need to wait for the run to finish. We took a snapshot every hour and diffed them by allocation site. The diff removes everything allocated at startup and leaves only what keeps growing. Five snapshots over four hours showed three linear sources:

• json.loads inside prometheus_client multiprocess mode — 67 MB and growing about 13.6 MB per hour; the whole Prometheus group was about 94 MB, 22% of all retained memory;
• the brotli ASGI middleware creating a compressor per request — about 4 MB per hour;
• our own payload-masking function in the logging middleware, copying whole request bodies before logging — about 1 MB per call.

The punchline was the first one. Multiprocess mode re-reads and re-parses its mmap files on every scrape — and the service runs as a single process. It did not need multiprocess mode at all.

None of the sources was an application endpoint. That is why the growth was smooth and did not correlate with any particular route.

Cleaning up was harder than profiling

We redeployed through CI and assumed the patch was gone. It was not. helm upgrade only resets fields the chart sets. The memray command, the extra volume, the grace period and an eviction annotation had been added with kubectl patch and were never in the chart. The service ended up still running under the profiler, now with the normal 1 GiB limit — worse than before. A second cleanup patch and a field-by-field comparison with the backup fixed it. Now we check PID 1 and the absence of the profile volume, not grep memray over /proc (your own grep matches itself).

Takeaways

• Never keep a container alive with sleep while the real process is under a profiler. End the wrapper with exec and keep liveness.
• Snapshot periodically and diff by allocation site: it separates startup allocations from the leak.
• Check what a library mode costs before enabling it: multiprocess metrics in a single-process service were 22% of the memory.
• After any rollback, compare the live spec with the backup. helm upgrade does not remove what kubectl patch added.
• A 350 MB baseline under a 1 GB limit leaves little room even after the fix — revisit the limit through the pipeline, not with a patch.