Long-running Python daemons have a dirty habit of growing in resident set size until the kernel's out-of-memory killer steps in and terminates the process. If you run a worker pool or an async consumer that stays up for days, standard reference counting won't save you from slow creep caused by cyclic references, global caches that lack eviction policies, or unclosed file descriptors. You need to inspect heap allocations directly while the worker runs.
You can find related resource management patterns in Fixing Python Subprocess Resource Leaks and diagnostic techniques in Profiling Python Memory Usage with Memray.
A worker that processes thousands of queue items per hour will eventually surface every hidden allocation quirk in your codebase. Memory doesn't just vanish. It gets attached to module-level dictionaries, event handlers, or orphaned references that refuse to go out of scope. Finding these issues requires a systematic approach to heap inspection rather than guesswork. Most developers only look at logs when the process crashes, which means they miss the gradual climb that happens hours before the crash.
Tracking Allocations With Tracemalloc
Python includes a built-in module named tracemalloc that records where memory blocks are allocated. It slows down execution slightly, so you don't want it running in every production thread permanently, but it is invaluable for staging environments or targeted debugging sessions. You configure it at startup to start taking snapshots of block traces. Memory tracing tells you the exact line of code that triggered the allocation, cutting down debug time from days to minutes.
Here is a minimal worker script that hooks into tracemalloc to dump the top memory consumers to standard output every few cycles. It keeps track of a growing internal list to simulate an unmanaged cache.
import time
import tracemalloc
tracemalloc.start(25)
class DataProcessor:
def __init__(self):
self.cache = []
def ingest(self, payload):
self.cache.append(payload)
processor = DataProcessor()
def run_worker():
for i in range(10000):
processor.ingest("x" * 1024 * 100)
if i % 1000 == 0:
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics("lineno")
print(f"--- Top 3 Memory Blocks (Iteration {i}) ---")
for stat in top_stats[:3]:
print(stat)
if __name__ == "__main__":
run_worker()
When you run that script, the output tells you the exact file path and line number responsible for allocating the bulk of the bytes. You will often find that a seemingly innocent list inside a singleton class grows indefinitely because nobody calls a clear method on it. Fixing it requires explicit bounds or a different data structure like an OrderedDict with a maximum size limit.
Developers frequently miss the cumulative effect of small allocations. Allocating a few kilobytes per request seems harmless. When that request handler loops infinitely inside a background worker, those kilobytes compound into gigabytes. Tracemalloc exposes these accumulation points instantly.
Resolving Cyclic Garbage Collection Deadlocks
Reference counting handles most objects instantly when their reference count drops to zero. However, Python's reference counter fails completely when two objects reference each other. An object with a reference count of one that only points to another object in a closed cycle will never be freed by simple reference decrementing. Python relies on a generational garbage collector to find these cycles, but certain structures confuse the detector completely.
If you define custom classes with a __del__ destructor method, older Python runtimes could leave those cyclic objects in the gc.garbage list because the interpreter didn't know which destructor to run first. Modern Python handles this better, but defining __del__ on application models in a worker daemon is still a bad idea. It stops the garbage collector from reclaiming that memory automatically.
You can inspect the state of the garbage collector programmatically to see if uncollectable objects are piling up in your worker process. Here is how you can set up a controlled test for reference cycles.
import gc
class Node:
def __init__(self, name):
self.name = name
self.peer = None
def create_leak():
a = Node("alpha")
b = Node("beta")
a.peer = b
b.peer = a
if __name__ == "__main__":
gc.disable()
create_leak()
print(f"Uncollected objects before collection: {len(gc.garbage)}")
collected = gc.collect()
print(f"Collected {collected} objects in cycle.")
gc.enable()
Disabling the garbage collector entirely is a common anti-pattern that people try when they think they are squeezing out micro-benchmarks. It guarantees that any cyclic reference created during normal request handling or background message processing stays in memory until the process dies. Leave the collector enabled and invoke gc.collect() manually at safe boundaries, such as after processing a massive batch job or rolling through a heavy queue migration task.
Debugging these cycles without instrumentation leads to endless frustration. You end up guessing which objects are locked together. Using the gc module to dump referrers helps you map out the graph and break the reference loop before it hits production.
Fixing Third-Party Library Leaks
Sometimes your code isn't leaking memory at all. C-extensions and database drivers often allocate buffers that aren't tracked accurately by Python's standard tooling. If you use connection pools that retain idle sockets or cache prepared statements indefinitely, the resident memory size will climb even though sys.getsizeof() reports tiny numbers for your application objects.
You should inspect the worker's open file descriptors using standard Linux utilities to verify whether database connections or pipe handles are leaking across task boundaries. If a background worker opens a new SQLite connection or HTTP client session inside a loop without using a context manager, those underlying sockets stay open and consume system resources until the OS enforces limits. You can cross-reference these issues with insights from Fixing Async Python Memory Leaks from Unclosed Sessions to ensure your cleanup logic is airtight.
Always wrap network calls and heavy file operations in explicit context managers. If you use async frameworks, ensure that tasks are awaited properly and canceled tasks don't leave lingering references in the event loop scheduler. Cleaning up background tasks prevents stale closures from holding onto entire object graphs in memory.
Monitoring RSS alongside heap metrics gives you the full picture. When RSS climbs while tracemalloc shows stable heap usage, look straight at native C libraries, thread stacks, or unreleased database cursors.
