Update DeepSeek-V4 Pro for B200 FP4 agentic HiCache DSpark (#38026)
This commit is contained in:
@@ -692,7 +692,7 @@ Larger blocks can improve decode latency when acceptance stays high, but they al
|
||||
|
||||
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
|
||||
|
||||
DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
DSpark currently requires CUDA and `pp_size == 1`. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The MI355X Flash Official recipes therefore run target-only, and so do the DP-Attention recipes in the Deploy panel — for a DP-Attention configuration that does run DSpark, see the [agentic recipe below](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
|
||||
### 3.5 Vision (Image Inputs)
|
||||
|
||||
@@ -734,3 +734,64 @@ Pending update...
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.6 Agentic Long-Context with HiCache DRAM Offload (B200 FP4, DSpark)
|
||||
|
||||
**TP8, concurrency 8–16:**
|
||||
```bash Command
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--moe-runner-backend flashinfer_mxfp4 \
|
||||
--enable-deepseek-v4-fp4-indexer \
|
||||
--disable-flashinfer-autotune \
|
||||
--mem-fraction-static 0.90 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--tool-call-parser deepseekv4 \
|
||||
--reasoning-parser deepseek-v4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
--speculative-dspark-block-size 6 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 2.75 \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-mem-layout page_first_direct
|
||||
```
|
||||
|
||||
DSv4 HiCache sizes the host tier with `--hicache-ratio` (host/device token ratio), not `--hicache-size`. Concurrency 1–5 runs the same command without the HiCache flags.
|
||||
|
||||
**DEP8 (DP Attention), concurrency 64–160:**
|
||||
```bash Command
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--enable-dp-lm-head \
|
||||
--ep-size 8 \
|
||||
--moe-a2a-backend megamoe \
|
||||
--enable-w4a4-mxfp4-megamoe \
|
||||
--enable-deepseek-v4-fp4-indexer \
|
||||
--disable-shared-experts-fusion \
|
||||
--disable-flashinfer-autotune \
|
||||
--mem-fraction-static 0.88 \
|
||||
--swa-full-tokens-ratio 0.02 \
|
||||
--chunked-prefill-size 49152 \
|
||||
--tool-call-parser deepseekv4 \
|
||||
--reasoning-parser deepseek-v4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
--speculative-dspark-block-size 6 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 8 \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-mem-layout page_first_direct
|
||||
```
|
||||
|
||||
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 6144 tokens per rank.
|
||||
|
||||
Reference in New Issue
Block a user