Update Qwen3.5 H200 FP8 for AgentX HiCache MTP (#35194)
This commit is contained in:
@@ -989,6 +989,45 @@ Max ITL (ms): 1220.68
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.3 Agentic Long-Context with HiCache DRAM Offload (H200 FP8, MTP)
|
||||
|
||||
For agentic workloads, here is how to enable HiCache and MTP.
|
||||
|
||||
Container image (pinned for reproducibility):
|
||||
`lmsysorg/sglang:nightly-dev-cu13-20260815-a5ba081f`.
|
||||
|
||||
Server Launch Command:
|
||||
```bash Command
|
||||
SGLANG_ENABLE_SPEC_V2=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
--served-model-name Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--data-parallel-size 1 \
|
||||
--expert-parallel-size 1 \
|
||||
--quantization fp8 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
--attention-backend flashinfer \
|
||||
--enable-flashinfer-allreduce-fusion \
|
||||
--mem-fraction-static 0.8 \
|
||||
--stream-interval 50 \
|
||||
--scheduler-recv-interval 10 \
|
||||
--tokenizer-worker-num 6 \
|
||||
--enable-metrics \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--page-size 64 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-size 77 \
|
||||
--hicache-io-backend kernel \
|
||||
--hicache-mem-layout page_first \
|
||||
--hicache-write-policy write_through_selective
|
||||
```
|
||||
|
||||
### 5.3 Vision Speed Benchmark
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.
|
||||
|
||||
Reference in New Issue
Block a user