Update Qwen3.5 H200 FP8 for AgentX HiCache MTP (#35194)

This commit is contained in:
Faradawn Yang
2026-08-17 17:12:07 -07:00
committed by GitHub
parent c0b6474b43
commit 91144797c5
@@ -989,6 +989,45 @@ Max ITL (ms): 1220.68
==================================================
```
#### 5.2.3 Agentic Long-Context with HiCache DRAM Offload (H200 FP8, MTP)
For agentic workloads, here is how to enable HiCache and MTP.
Container image (pinned for reproducibility):
`lmsysorg/sglang:nightly-dev-cu13-20260815-a5ba081f`.
Server Launch Command:
```bash Command
SGLANG_ENABLE_SPEC_V2=1 \
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3.5-397B-A17B-FP8 \
--served-model-name Qwen/Qwen3.5-397B-A17B-FP8 \
--trust-remote-code \
--tensor-parallel-size 8 \
--data-parallel-size 1 \
--expert-parallel-size 1 \
--quantization fp8 \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--attention-backend flashinfer \
--enable-flashinfer-allreduce-fusion \
--mem-fraction-static 0.8 \
--stream-interval 50 \
--scheduler-recv-interval 10 \
--tokenizer-worker-num 6 \
--enable-metrics \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-size 77 \
--hicache-io-backend kernel \
--hicache-mem-layout page_first \
--hicache-write-policy write_through_selective
```
### 5.3 Vision Speed Benchmark
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.