diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx index 11b10d0f0..f5cfe5f0b 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx @@ -989,6 +989,45 @@ Max ITL (ms): 1220.68 ================================================== ``` +#### 5.2.3 Agentic Long-Context with HiCache DRAM Offload (H200 FP8, MTP) + +For agentic workloads, here is how to enable HiCache and MTP. + +Container image (pinned for reproducibility): +`lmsysorg/sglang:nightly-dev-cu13-20260815-a5ba081f`. + +Server Launch Command: +```bash Command +SGLANG_ENABLE_SPEC_V2=1 \ +python3 -m sglang.launch_server \ + --model-path Qwen/Qwen3.5-397B-A17B-FP8 \ + --served-model-name Qwen/Qwen3.5-397B-A17B-FP8 \ + --trust-remote-code \ + --tensor-parallel-size 8 \ + --data-parallel-size 1 \ + --expert-parallel-size 1 \ + --quantization fp8 \ + --kv-cache-dtype fp8_e4m3 \ + --mamba-ssm-dtype bfloat16 \ + --attention-backend flashinfer \ + --enable-flashinfer-allreduce-fusion \ + --mem-fraction-static 0.8 \ + --stream-interval 50 \ + --scheduler-recv-interval 10 \ + --tokenizer-worker-num 6 \ + --enable-metrics \ + --speculative-algorithm EAGLE \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --page-size 64 \ + --enable-hierarchical-cache \ + --hicache-size 77 \ + --hicache-io-backend kernel \ + --hicache-mem-layout page_first \ + --hicache-write-policy write_through_selective +``` + ### 5.3 Vision Speed Benchmark We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.