sglang/python/sglang/srt/mem_cache

mem_cache/

Everything that owns KV / SSM-state memory: who hands out slots, who holds the bytes on the device, who mirrors them to host and disk, and which radix cache decides what to keep. The layout is specified in #25371.

Layers

        scheduler / model_runner / attention backend
                          |
                          v
        allocation.py                  per-batch allocation policy
                          |
                          v
        hybrid_cache/                  multi-pool router (layer_id -> pool)
                          |
                          v
        allocator/                     "give me N slots"  (need_size -> indices)
                          | holds a reference to
                          v
        pool/ (device, L1)  --hicache-->  pool_host/ (host, L2)  -->  storage/ (L3)
        (layer_id, indices)               device_indices <->
             <-> tensor                   host_indices

Layer	Cares about	In -> Out
allocation.py	per-batch allocation policy	batch -> out_cache_loc
hybrid_cache/	per-layer routing across pools	layer_id -> pool
allocator/	which slots are free	need_size -> indices
pool/	physical KV / SSM state layout	(layer_id, indices) <-> tensor
pool_host/	host mirror + H2D/D2H	device_indices <-> host_indices
storage/	L3 backends (file, NIXL, HF3FS, Mooncake, ...)	hash -> bytes
radix cache	what to keep and what to evict	token prefix -> node

Two groups sit outside that stack:

• Radix cache is its own axis. The per-model variants (radix_cache.py, swa_radix_cache.py, mamba_radix_cache.py, hiradix_cache.py, chunk_cache.py) are converging onto the Unified Radix Cache (unified_cache/, #20415), whose Full/SWA/Mamba component model is documented in unified_cache/components/README.md.
• Construction cuts across every layer rather than sitting in it: kv_cache_configurator.py, kv_cache_builder.py, cache_init_params.py, allocation_sizing.py, kv_cache_dtype.py, kv_vmm_backing.py, and hybrid_cache/hybrid_pool_assembler.py decide the shapes and build the objects above.

Where does my class go?

By base class, never by name:

Inherits from	Home
BaseTokenToKVPoolAllocator	allocator/<family>.py
KVCache, BaseSWAKVPool, ReqToTokenPool, MambaPool	pool/<family>.py
HostKVCache	pool_host/<family>.py
HiCacheStorage	storage/<backend>/
BasePrefixCache	one module at the mem_cache/ root

<family> is the attention or state family: mha, mla, dsa, mamba, swa, hisparse, deepseek_v4. A new quantization or layout variant of an existing family is a new file in that family's module, not a new class in a catch-all one.

Allocator means two different things and they do not share a directory:

• slot allocator -- a BaseTokenToKVPoolAllocator subclass, hands out KV slots, lives in allocator/.
• host tensor allocator -- HostTensorAllocator and its subclasses, hands out pinned host memory, lives in pool_host/common.py and storage/.

Conventions

• Names drop affixes that do not differentiate. If every file in a directory shares the role the directory already names, the affix carries nothing: pool_host/mha.py, not pool_host/mha_pool_host.py. Keep a role affix only where same-directory siblings have different roles.
• A family is a module; a module may be a package. One file per family by default; past ~1500 lines the family becomes a package.
• Layers do not import upwards. pool/ and pool_host/ must not import allocator/, hybrid_cache/, or allocation.py; allocator/ may hold the pool it allocates into, not the reverse; none of the three may import the construction layer.