[Performance] Decode Offload improves the long texts performance 100% through dynamic block offload. (#17216)

Co-authored-by: zhangheng <hzh0425@apache.org>
2026-03-08 17:16:53 +08:00
parent 97a2a9be0f
commit 230fb55899
8 changed files with 290 additions and 33 deletions
@@ -164,6 +164,8 @@ SGLang supports various environment variables that can be used to configure its
 | `SGLANG_WAIT_WEIGHTS_READY_TIMEOUT` | Timeout period for waiting on weights | `120` |
 | `SGLANG_DISABLE_OUTLINES_DISK_CACHE` | Disable Outlines disk cache | `true` |
 | `SGLANG_USE_CUSTOM_TRITON_KERNEL_CACHE` | Use SGLang's custom Triton kernel cache implementation for lower overheads (automatically enabled on CUDA) | `false` |
+| `SGLANG_HICACHE_DECODE_OFFLOAD_STRIDE` | Decode-side incremental KV cache offload stride. Rounded down to a multiple of `--page-size` (min is `--page-size`). If unset/invalid/<=0, it falls back to `--page-size`. | Not set (uses `--page-size`) |
+

 ## Function Calling / Tool Use