Long-running Python background processes consume all available RAM until the operating system steps in with an out-of-memory killer. Standard tools like cProfile only track CPU time. Memory leaks remain completely invisible until production clusters drop requests. You need a profiler that traces both Python objects and native C allocations down to the exact line of code. Memray handles this cleanly.
Before tracking down elusive allocations, install the package in your worker environment. Avoid installing heavy profiling tools directly inside minimal production images. Build them in a staging pipeline instead. You'll need the library available to run your script. Run the installation command in your terminal.
pip install memrayUnmanaged resources quietly exhaust system memory over time in asynchronous codebases. Read our guide on Fixing Async Python Memory Leaks from Unclosed Sessions for more details on that specific failure mode. But standard async issues are only half the battle. C extensions, database drivers, and serialization libraries allocate memory off the Python heap entirely. Standard garbage collection ignores these allocations. They become impossible to find with standard inspection tools.
Tracing Native Allocations in Production Workers
Memray hooks directly into the allocator. It tracks every single block of memory requested from the operating system. It doesn't just watch high-level Python objects. It sees underlying C malloc calls too. When a worker process swells from fifty megabytes to four gigabytes over a weekend, guessing where the leak lives fails. You must capture a binary trace file of the execution.
Run an existing worker script under Memray by passing the module or script path straight to the CLI tracer. Imagine a worker script that processes queue jobs in an infinite loop. Launch it with native tracking enabled like this.
memray run --native worker.pyThat command generates a massive binary file containing every allocation event. Because it records everything, don't run this indefinitely on a high-traffic server. Run it for a representative workload. Process a fixed number of tasks before letting the script exit. Once the file is written, generate a flame graph to visualize the memory footprint.
The flame graph gives you an interactive HTML report of memory usage. Open it in any browser. See immediately which function calls allocated the largest blocks of memory. If a third-party library leaks raw buffers in a C extension, it sticks out. Spot the culprit by looking for wide blocks at the top of the call stack that never shrink.
Isolating Reference Cycles and Garbage Collection Failures
Python reference counting handles most memory cleanup automatically. Circular references break simple reference counting entirely. Object A points to object B, and object B points back to object A. Their reference counts never hit zero when they go out of scope. They wait for the cyclic garbage collector to run. If your application creates these cycles faster than the collector cleans them up, RAM usage climbs steadily.
Inspect live memory stats or parse the generated trace using Memray summary commands to find where objects accumulate. Look at a simple worker class that generates lingering references if you aren't careful about clearing internal state handlers.
import sys
import time
class JobProcessor:
def __init__(self):
self.history = []
def process_payload(self, data):
self.history.append((data, sys.getrefcount(data)))
if len(self.history) > 1000:
self.history.pop(0)
if __name__ == "__main__":
worker = JobProcessor()
for i in range(5000):
worker.process_payload({"id": i, "payload": "data"})
time.sleep(1)Run that script under Memray and parse the output. Check memory stats grouped by line number. Execute the summary command with the recorded trace file to inspect top memory consumers.
memray summary memray-worker.py.binThe output lists exact source files and line numbers responsible for retaining the most bytes. When you see a list or dictionary growing inside a long-lived instance, add explicit cleanup or context managers. Sometimes the fix is deleting a reference or breaking a closure capturing a large scope.
Integrating Profiling into Automated Test Pipelines
Waiting for a worker to fail in production is a terrible way to catch memory bugs. Incorporate memory tracking into test suites or staging deployments. While you won't run full Memray traces on every unit test, use it to assert that specific background jobs don't exceed defined memory thresholds before merging code changes.
Generate statistics reports in JSON format during your CI pipeline to track memory regressions over time. If a new dependency introduces a heavy C-level allocation on every request, your CI script catches the delta before it hits main. Run the stats command to output a structured summary for your CI runner.
memray stats memray-worker.py.binAnalyzing these stats helps establish baselines for how much memory your application actually needs. If your workers require five hundred megabytes to idle comfortably, provisioning a smaller container guarantees an out-of-memory crash during peak hours. Use the data from your profiles to right-size infrastructure and keep background processes stable under real-world loads.
