[Cookbook] DeepSeek-V4.1: add the HiCache L2 knob to the Playground (#38844)
Co-authored-by: Claude Code <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Code
parent
dc2157dcd6
commit
a37ded1693
@@ -192,3 +192,15 @@ Prefill/decode disaggregation is validated token-identical against a single serv
|
||||
One deployment note: Mooncake needs the RDMA fabric visible inside the container, so launch with `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1`. Without it Mooncake selects its NVLink transport, which only serves buffers from its own allocator and fails with `Requested address ... not found`. If you hit that, force TCP with `MOONCAKE_PROTOCOL=tcp` and `MC_FORCE_TCP=1` — the Playground's Mooncake option sets both.
|
||||
|
||||
PD and speculative decoding cannot be combined.
|
||||
|
||||
### 3.5 HiCache (Hierarchical KV Caching)
|
||||
|
||||
HiCache extends RadixAttention with a hierarchy of KV cache tiers, significantly expanding effective context capacity for long-context and multi-turn scenarios.
|
||||
|
||||
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**: the Playground emits `--enable-hierarchical-cache` on top of the recipe's flags, and cold KV pages spill to CPU pinned memory (L2, GPU + CPU) instead of being dropped from the GPU pool. The host pool is sized by ratio (`--hicache-ratio 2`), not by a fixed `--hicache-size`.
|
||||
|
||||
The Write policy knob controls the GPU → CPU write and defaults to `write_through` (the upstream default): every page is mirrored to the CPU tier as it is written. `write_through_selective` backs up only hot data and `write_back` defers the copy to eviction, trading cache freshness for host-side I/O.
|
||||
|
||||
The card is not offered on MI350X: the ROCm recipes run `--disable-radix-cache`, and the server rejects that alongside `--enable-hierarchical-cache`.
|
||||
|
||||
Only the L2 tier is exposed here. For the storage (L3) tier and the canonical flag set, see the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices) and the [HiCache documentation](../../../docs/advanced_features/hicache).
|
||||
|
||||
@@ -114,6 +114,19 @@ export const config = {
|
||||
},
|
||||
},
|
||||
|
||||
// GPU → CPU KV offload (L2 only; no storage tier). Hidden on MI350X: both
|
||||
// ROCm cells run `--disable-radix-cache`, which the server rejects alongside
|
||||
// `--enable-hierarchical-cache`.
|
||||
hicache: {
|
||||
excludesHw: ["mi350x"],
|
||||
writePolicies: [
|
||||
{ id: "auto", label: "Auto" },
|
||||
{ id: "write_through", label: "Write-through" },
|
||||
{ id: "write_back", label: "Write-back" },
|
||||
{ id: "write_through_selective", label: "Write-through (selective)" },
|
||||
],
|
||||
},
|
||||
|
||||
flagSelects: [
|
||||
{
|
||||
id: "dsparkBlockSize",
|
||||
|
||||
Reference in New Issue
Block a user