[docs] DeepSeek-V4: MI355X PD disaggregation recipes for all three strategies (#39396)

docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx

@@ -391,8 +391,8 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000.
 - **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
 - **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
 - **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
-- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
-- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Do not use EAGLE on 0813. Enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path.
+- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The 0813 Pro Official checkpoint keeps its MTP head too, so the same flags are the speculative option under PD disaggregation — see the DSpark note below.
+- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Prefer it over EAGLE on 0813 for ordinary serving: enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. The one exception is **PD disaggregation**, where DSpark cannot run at all and the Playground strips it — there, fall back to the MTP 3-1-4 shape above, as in [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io).
 - **Kernels** — uses the Unified KV attention and the flydsl MoE.
 - **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe.
 - **Agentic long-context (MI355X Pro Official FP4)** — TP-only serving adds `--prefill-decode-interval 10` for scheduler stability. The DP path additionally needs `--enable-dp-lm-head` (required for DSpark under DP attention), `--enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7` so a single long prefill does not monopolise the engine, plus the flags in the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark).
@@ -655,7 +655,7 @@ For AMD devices,
 
 The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow.
 
-For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache).
+For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). For the alternative that skips the host tier entirely, see [§3.9 UMBP](#3-9-umbp-direct-external-kv-store) — it is a sibling of HiCache, not a storage backend for it, and the two cannot be enabled together.
 
 ### 3.4 DSpark (Speculative Decoding)
 
@@ -887,3 +887,77 @@ python3 -m sglang.launch_server \
 ```
 
 `--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention.
+
+### 3.8 PD Disaggregation on MI355X (MORI-IO)
+
+On ROCm the prefill and decode roles talk over **MORI-IO** rather than Mooncake or NiXL, so the **PD Disagg** card in the [Playground above](#playground) hides the CUDA-only transports once MI300X / MI355X is selected and defaults the IB device list to the node's `rdmaN` NICs instead of ConnectX `mlx5_*` names.
+
+Both roles need the RDMA NICs passed into the container, so the **Docker** output adds the fabric flags below on top of the usual ROCm device access from the [Install panel](#install). `/dev/infiniband` covers `rdma_cm` and the per-NIC `uverbs0`…`uverbs7` nodes, and the memlock/`IPC_LOCK` pair is what lets MORI-IO pin the buffers it registers with the NICs:
+
+```bash Command
+docker run \
+    --device=/dev/kfd --device=/dev/dri --device /dev/infiniband \
+    --group-add video --cap-add IPC_LOCK \
+    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
+    --ulimit memlock=-1 --ulimit stack=67108864 \
+    --ulimit nofile=1048576:1048576 \
+    --network host --ipc=host --shm-size 32g \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911 \
+    python3 -m sglang.launch_server <role args below>
+```
+
+`--network host` replaces the single-node `-p` mapping because the two roles rendezvous across hosts. Long agentic runs on the 1.6T checkpoint may need more than `--shm-size 32g`; raise it if the workers die during weight load.
+
+Selecting either role with **MORI** on MI355X emits the shared MORI-IO transport env — `SGLANG_MORI_COMBINE_DTYPE` plus the RDMA send-queue tuning `MORI_IO_SQ_BACKOFF_TIMEOUT_US` / `MORI_IO_QP_MAX_SEND_WR` — together with that role's sizing. `--max-running-requests` is deliberately small on both: the pair streams KV between workers rather than holding a large running batch.
+
+The roles are sized per **Strategy**, so the Deploy panel's operating point carries through to the PD command:
+
+- **Low-Latency** — TP-only, `--mem-fraction-static 0.86`, and a running-request ceiling of 8. Decode captures graphs for batches 1–8. No offload tier: the KV pool is all there is.
+- **Balanced** — still TP-only, with the ceiling at `--max-running-requests 96` and a 16384 chunked-prefill budget. Decode ladders all the way to 96, since without DP the server-wide ceiling *is* the per-rank batch. Prefill runs a **HiCache** host tier (`--hicache-ratio 2.5`, `page_first`, write-through, best-effort prefetch); decode does not, because HiCache is not recommended on the decode role with MORI.
+- **High-Throughput** — inherits TP8 / DP8 and the 65536 chunked-prefill budget from the base cell, raises the ceiling to `--max-running-requests 256` and `--mem-fraction-static 0.92`, and ladders decode to 32 — the per-rank batch 256 leaves across DP 8. It adds `--enable-cache-report`, which surfaces the prefix hit rate you need to judge whether an offload tier is paying for itself at that concurrency.
+
+The offload tier is what actually separates balanced from high-throughput: balanced pairs prefill with HiCache's tiered GPU → host cache, while high-throughput pairs it with UMBP, which links the radix tree straight to MORI with no host tier. They cannot both be on, and the Playground enforces that.
+
+<Note>
+The low-latency and balanced roles are TP-only, and that is exactly why their decode ladders run to the full ceiling. Selecting either role forces DP-Attention off and grays the control: `--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so switching DP on would cut the per-rank batch below the captured graphs without changing either flag in the command. With PD off, the balanced cell stays a DP recipe for aggregated serving.
+</Note>
+
+<Warning>
+**Speculative decoding (MTP) changes how you size the ceilings.** For a target concurrency of N, set `--max-running-requests N*2` on **both** roles. Sizing them at N caps the served batch below the concurrency you are aiming for. PD disaggregation requires EAGLE/MTP (DSpark cannot run under it), so this applies to every PD deployment.
+
+`--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so the decode graph ladder should cover the **per-rank** batch that leaves — `N*2 / dp_size`. That is why the two points above ladder to different places from ceilings that differ by less: balanced is TP-only, so 96 slots are 96 per rank, while high-throughput spreads 256 over DP 8 and only ever sees 32 at a time.
+</Warning>
+
+The high-throughput prefill role is where [UMBP](#3-9-umbp-direct-external-kv-store) belongs — DP attention is already on, which is UMBP's hard requirement, and the concurrency is high enough for the pool to matter. Turn on the **UMBP** card alongside the role; it is a separate card rather than part of the role because the linker flags are its to own, and leaving them implicit would hide the dependency.
+
+Whichever point you pick, the two roles diverge in two places:
+
+- **Cuda graphs.** Prefill runs eager (`--disable-cuda-graph`) because the FP4 indexer's prefill path is not captured. Decode captures the small batch ladder that matches its running-request ceiling (`--cuda-graph-bs-decode 1 2 3 4 5 6 7 8`).
+- **MORI dispatch budget.** `SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK` is `16384` on prefill, sized for a whole chunked-prefill batch, and `128` on decode, where a step never dispatches more than a token per running request.
+
+Note that DSpark is incompatible with PD disaggregation, so picking a role strips the inherited DSpark flags (see [§3.4](#3-4-dspark-speculative-decoding)). For a speculative PD recipe use the bundled MTP head through the EAGLE path instead:
+
+```bash Command
+--speculative-algorithm EAGLE --speculative-eagle-topk 1 \
+--speculative-num-steps 3 --speculative-num-draft-tokens 4
+```
+
+Run the two roles on separate nodes with the same transport and IB device selection, then front them with the router the card prints. On MI355X that router is cache-aware rather than round-robin: `--policy consistent_hashing` keeps a conversation on the worker that already holds its prefix, which is what makes the agentic reuse pay off, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` keep that affinity from starving the peer given how small the running-request ceiling is. Health checks stay on with a long timeout so a wedged worker is still caught without a multi-minute prefill being mistaken for a failure.
+
+`--dp-aware` only does something when the workers run DP attention — turn on the Playground's **DP-Attention** card if you want it to take effect, otherwise it is inert on the TP-only recipe above.
+
+### 3.9 UMBP (Direct External KV Store)
+
+UMBP is the second way to spill KV off the GPU, and it is an **alternative to [HiCache](#3-3-hicache-hierarchical-kv-caching), not a tier inside it**. HiCache is a hierarchy — GPU, then pinned host memory, then optionally a storage backend. UMBP links the unified radix tree straight to MORI's buffer pool with no host cache tier in between, so pages load and offload against the external store directly. SGLang rejects the two together, so enabling the **UMBP** card in the [Playground above](#playground) strips the whole HiCache flag family from the command:
+
+```bash Command
+--enable-unified-cache-external-linker \
+--unified-cache-external-linker-backend mori
+```
+
+The **Store** knob picks the linker backend. `mori` is the UMBP pool; `mooncake` drives the same direct-linker path against a Mooncake store instead.
+
+**DP Attention is required**, and the card's Enable chip stays greyed out until you turn it on. This is not a soft preference. The linker keys its objects by rank, and MLA KV is replicated across TP, so a TP8 prefill worker under pure TP opens eight separate keyspaces holding eight copies of the same tokens — the pool's effective distinct-token capacity would be an eighth of what its byte budget suggests, which is easy to mistake for a real measurement. DP attention collapses the keys onto one shared keyspace.
+
+Two further limits are worth knowing before you size a run. The pool is a per-node process, so a prefill worker that spans nodes would shard its keyspace by node. And only the prefill worker offloads KV — the decode side is untouched.