How Python decides what to free,
and when to go looking.

Most Python developers never think about garbage collection, which is a compliment to its design. Anyone running Python in a latency-sensitive service eventually does, because the collector is the one part of the runtime that can stop everything for a user-visible interval, and because the rules that decide when it runs are not the ones people usually assume.

This is a walkthrough of those rules: the two mechanisms Python uses to reclaim memory, how the cyclic collector decides what is unreachable, what actually triggers a collection, and how the free-threaded build in Python 3.13 counts allocations now that threads run in parallel. That last piece is the least documented, and it is where the arithmetic gets interesting.

The two-part system

Python manages memory with two mechanisms that solve different halves of the problem.

Reference counting frees an object the instant the last reference to it goes away. It is immediate, it is deterministic, and it handles the overwhelming majority of objects in any real program. It also has one blind spot.

Cyclic garbage collection exists to cover that blind spot: groups of objects that reference each other, where every count stays above zero even though nothing outside the group can reach them. This is the part that runs periodically, and the part with pauses, so it is the part worth understanding.

Reference counting

Every Python object carries a count of the references to it in ob_refcnt. Assigning it to a name, passing it to a function, or storing it in a container increments the count; letting any of those go decrements it. When the count hits zero the object is freed on the spot.

SIMPLIFIED FROM Include/refcount.h

/* Every reference gained */
static inline void Py_INCREF(PyObject *op)
{
    op->ob_refcnt++;
}

/* Every reference lost */
static inline void Py_DECREF(PyObject *op)
{
    if (--op->ob_refcnt == 0) {
        _Py_Dealloc(op);   /* free it now, no collector involved */
    }
}

This is why Python's memory use is usually so predictable, and it is worth appreciating before we get to the collector: the collector is the exception path, not the main one.

The problem with cycles

Reference counting fails on exactly one shape, and it fails silently.

A CYCLE THAT REFERENCE COUNTING CANNOT FREE

a = []
b = []
a.append(b)      # b's refcount is now 2: name b, plus a[0]
b.append(a)      # a's refcount is now 2: name a, plus b[0]

del a, b         # both names gone, both objects still at refcount 1
                 # neither is reachable, neither is freed

Two objects, each holding the only remaining reference to the other. No name refers to either, nothing in the program can reach them, and both counts sit stubbornly at one. Reference counting has no way to notice this, because from each object's local point of view it is still in use.

In an inference server this shape is not exotic. Request context objects that point at their own response builders, exception tracebacks that reference the frames that raised them, caches that hold back-references, and any nontrivial async machinery all produce cycles routinely.

Cyclic garbage collection

The cyclic collector examines container objects, the only ones that can participate in a cycle, and asks a question that sounds harder than it is: which of these are referenced only by each other?

The algorithm, in gc_collect_region(), is elegant. Copy every tracked object's reference count into a scratch field. Then walk every reference held by objects inside the region and decrement the scratch count of whatever it points at. When the pass finishes, each scratch count holds the number of references from outside the region. Anything sitting at zero is reachable only from within the group, which means it is not reachable at all.

When does garbage collection run?

Here is the part that surprises people, and the part everything later in this post depends on: collection is triggered by allocation, not by time and not by memory pressure. CPython keeps a running count of container allocations minus deallocations, and when that count crosses a threshold, it collects.

It also assumes that most objects die young, so it sorts objects into generations and scans the youngest one far more often than the others.

The important consequence is that a young-generation pass is only cheap because it runs frequently. Every collection it skips is objects that pile up for the next one to traverse, and the cost of a pass scales with how much survived to be walked. A collector that falls behind does not degrade gracefully, it degrades quadratically in perceived pain: longer gaps, bigger heaps, longer pauses.

Thread-local allocation counting

Python 3.13 shipped a free-threaded build, which removes the global interpreter lock and lets threads execute Python bytecode genuinely in parallel. That is exactly what you want in an inference server, where the alternative is process-per-worker and a copy of the model per process.

It also complicates the allocation counter. A single global count incremented on every allocation would be a contended atomic in the hottest path in the interpreter, which reintroduces by the back door the serialization the GIL removal was meant to eliminate. CPython's answer is batching: each thread keeps its own local count and only touches the shared one when the local count crosses LOCAL_ALLOC_COUNT_THRESHOLD, which is 512.

SIMPLIFIED FROM Python/gc_free_threading.c

#define LOCAL_ALLOC_COUNT_THRESHOLD 512

/* One of these per thread, no atomics in the common case */
static void record_allocation(PyThreadState *tstate)
{
    struct _gc_thread_state *gc = &((_PyThreadStateImpl *)tstate)->gc;
    gc->alloc_count++;

    if (gc->alloc_count >= LOCAL_ALLOC_COUNT_THRESHOLD) {
        /* flush the batch into the shared counter */
        add_to_shared_count(gc->alloc_count);
        gc->alloc_count = 0;
    }
}

static void record_deallocation(PyThreadState *tstate)
{
    struct _gc_thread_state *gc = &((_PyThreadStateImpl *)tstate)->gc;
    gc->alloc_count--;

    if (gc->alloc_count <= -LOCAL_ALLOC_COUNT_THRESHOLD) {
        add_to_shared_count(gc->alloc_count);   /* a NEGATIVE batch */
        gc->alloc_count = 0;
    }
}

This is a good design. Allocation stays cheap, the shared counter is touched once per 512 events per thread instead of once per event, and the count it holds is accurate to within 512 per thread, which is plenty for deciding when to run a collection.

Notice the asymmetry in that pair of functions, and compare it with what the build that still has the GIL does when an object goes away. There, the decrement is guarded:

SIMPLIFIED FROM Python/gc.c, THE BUILD WITH THE GIL

if (gcstate->young.count > 0) {
    gcstate->young.count--;      /* never goes below zero */
}

With one thread and one counter, that guard is all it takes to keep the number that gates collection non-negative. With per-thread batching there are two counters, and the question of where to put the floor becomes a real design decision.

Why the shared count can go negative

You can ask the interpreter for its own view of that state at any time, and the young generation's number is the one to read first:

INSPECTING THE COUNTERS

>>> import gc
>>> gc.get_count()
(1843, 0, 0)      # young, old-pending, old-visited

>>> gc.get_count()
(-2048, 0, 0)     # a deficit: 2048 allocations before zero,
                  # and 2000 more before a collection runs

The young generation's count is meant to be the number of objects allocated since the last collection, compared against a threshold of 2000. A negative value is therefore not a small inaccuracy, it is a deficit that has to be paid off in allocations before the threshold is even in reach. Collection is delayed by exactly that much.

The arithmetic that produces one takes several threads, which is why it is specific to the free-threaded build.

Two details make a deficit compound rather than self-correct. A collection zeroes the shared counter but not the per-thread ones, so the accounting starts each interval already skewed. And because the deficit makes the next collection later, there is more time to accumulate the next one before anything resets. Enough rounds of that and the shared count is in the millions rather than the thousands.

Where to put the floor

The thread-local count going negative is fine and should stay legal: a thread that frees more than it allocates is doing real work and its local number is honest bookkeeping. What matters is that a negative value not land in the shared counter that gates collection. So the floor belongs on the flush, not on the decrement.

Because several threads can flush at the same moment, the clamp cannot be a read followed by a write. It needs a compare-and-exchange loop, so that a thread which loses the race retries against the value that actually landed:

CLAMP THE SHARED COUNT, NOT THE LOCAL ONE

/* Flush a thread's local batch into the shared count, floored at zero */
Py_ssize_t old = _Py_atomic_load_ssize_relaxed(&shared_count);
Py_ssize_t new_count;

do {
    new_count = old + local_batch;
    if (new_count < 0) {
        new_count = 0;        /* the gate never goes negative */
    }
} while (!_Py_atomic_compare_exchange_ssize(&shared_count,
                                            &old, new_count));

/* On failure, `old` holds the value another thread wrote, so the
   loop recomputes against reality instead of clobbering it. */

A few lines, and worth noticing what the compare-and-exchange buys. With a plain load, clamp, store, two threads flushing simultaneously can each read the same starting value, and one write wins while the other thread's batch disappears. The counter stays non-negative and stays wrong.

Why collection frequency is the number to watch

The practical consequence of everything above is visible with gc.set_debug(gc.DEBUG_STATS), which prints a line per collection. Compare a collector running on schedule with one that has fallen a long way behind on the same workload.

A COLLECTOR THAT HAS FALLEN BEHIND

gc: collecting generation 0...
gc: objects in each generation: 4892745 0 0
gc: done, 3.127s elapsed

THE SAME WORKLOAD, COLLECTING ON SCHEDULE

gc: collecting generation 0...
gc: objects in each generation: 2145 0 0
gc: done, 0.001s elapsed

This is the whole argument for the generational design, and its one condition. A young-generation pass is cheap because it runs often; whatever delays the trigger converts that cheap frequent pass into an expensive rare one.

Four things worth remembering

Read gc.get_count() before profiling a latency problem. If any of the three numbers is negative or implausibly large, the scheduling of collection is the thing to look at first; a profiler will otherwise just confirm that time went into whatever happened to run during the pause.

Watch collection frequency, not only pause duration. A collector that has stopped being triggered looks healthy on a pause-time dashboard, right up until the pause that does happen is measured in seconds. Frequency is the leading indicator.

Allocation is the clock. Not wall time, not resident memory. A workload that allocates in bursts and then goes quiet can hold garbage indefinitely, and one that allocates constantly collects constantly.

Per-thread bookkeeping has different arithmetic. Removing the GIL means every counter that assumed one thread of execution gets reimplemented as a batched, per-thread version. Those are correct in isolation and interact in ways the single-threaded version never could, which is exactly where a floor on one counter and a reset on another have to be reasoned about together.

Where this fits at Numerata

Runtime behavior at this level is what NinetyFive, the serving layer in the Numerata stack, is built on top of, and it is the sort of detail that only becomes visible when you own the serving path down to the interpreter build, the same way owning the CUDA graph capture is what makes warmup time addressable. It is also why the sizing arithmetic holds up in practice: a p99 target is only meaningful if the runtime underneath it is collecting on a schedule you can predict.

The original writeup, with more of the CPython source detail, is on the NinetyFive engineering blog at ninetyfive.gg/blog/python-gc ↗.