docs: state that HiCache L2 is instance-private and only L3 is shared (#37050)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

docs/docs/advanced_features/hicache_best_practices.mdx

@@ -7,6 +7,10 @@ metatags:
 
 SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems.
 
+<Note>
+L1 and L2 are private to a single inference instance; only L3 can be shared. Host memory cannot be pooled across instances or across hosts, not even for two instances on the same node. Raising `--hicache-ratio` or `--hicache-size` only enlarges the instance's own private L2. Cross-instance reuse is the job of L3, which needs `--hicache-storage-backend` and a backend configured for the scope you want: the `file` backend defaults to the node-local `/tmp/hicache`, whereas `mooncake`, `hf3fs`, `nixl` and `aibrix` reach cluster scope when every instance shares the same namespace. See [Tier Sharing Scope](/docs/advanced_features/hicache_design#tier-sharing-scope).
+</Note>
+
 ## Configuration Guidelines
 
 ## Core HiCache Parameters

docs/docs/advanced_features/hicache_design.mdx

@@ -18,6 +18,52 @@ Inspired by the classic three-level cache design of modern CPUs, HiCache organiz
 
 In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster.
 
+### Tier Sharing Scope
+
+Which tier is private and which is shared decides where a cache hit can come from, so it is worth stating explicitly:
+
+| Tier | Medium | Scope | Shared across instances? |
+|---|---|---|---|
+| **L1** | GPU memory (HBM) | One inference instance | No |
+| **L2** | Host memory (CPU DRAM) | One inference instance, on its own node | **No** |
+| **L3** | Storage backend (`file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, or a custom one) | Whatever the backend is configured to span | **Only if the backend is set up for it** |
+
+```mermaid
+flowchart TB
+    subgraph N0["Node A"]
+        subgraph I0["Instance 0"]
+            L1A["L1 - GPU memory<br/>private"]
+            L2A["L2 - host memory<br/>private to this instance"]
+            L1A --- L2A
+        end
+        subgraph I1["Instance 1"]
+            L1B["L1 - GPU memory<br/>private"]
+            L2B["L2 - host memory<br/>private to this instance"]
+            L1B --- L2B
+        end
+    end
+
+    subgraph N1["Node B"]
+        subgraph I2["Instance 2"]
+            L1C["L1 - GPU memory<br/>private"]
+            L2C["L2 - host memory<br/>private to this instance"]
+            L1C --- L2C
+        end
+    end
+
+    L3["L3 - storage backend<br/>shared by the instances the backend is configured to span"]
+
+    L2A --> L3
+    L2B --> L3
+    L2C --> L3
+```
+
+**L2 is node-local and instance-private.** It is host memory owned by one inference instance's process, so two instances never read each other's L2, not even two instances on the same node. A KV cache produced by instance 0 becomes visible to instance 1 only after it reaches L3.
+
+This is the answer to a common question: HiCache cannot pool the host memory of several machines into one larger L2. Growing `--hicache-ratio` or `--hicache-size` only makes each instance's own private L2 larger. Cross-instance reuse is the job of L3, so it needs `--hicache-storage-backend`.
+
+L3 is the tier that *can* be shared, but whether it actually is depends on the backend and how it is configured. The `file` backend writes to `/tmp/hicache` by default (`SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR` overrides it), which is node-local unless that path is a shared mount. Distributed backends such as `mooncake`, `hf3fs`, `nixl` and `aibrix` give cluster-wide scope, but only when every instance is pointed at the same namespace and configuration.
+
 ### HiRadixTree: Metadata Organization in HiCache
 
 For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage.