Kernels community fa3 (#20796)
This commit is contained in:
@@ -19,6 +19,7 @@ SGLang supports various environment variables that can be used to configure its
|
||||
| `SGLANG_FORWARD_UNKNOWN_TOOLS` | Forward unknown tool calls to clients instead of dropping them | `false` (drop unknown tools) |
|
||||
| `SGLANG_REQ_WAITING_TIMEOUT` | Timeout (in seconds) for requests waiting in the queue before being scheduled | `-1` |
|
||||
| `SGLANG_REQ_RUNNING_TIMEOUT` | Timeout (in seconds) for requests running in the decode batch | `-1` |
|
||||
| `SGLANG_CACHE_DIR` | Cache directory for model weights and other data | `~/.cache/sglang` |
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
@@ -47,6 +48,7 @@ SGLang supports various environment variables that can be used to configure its
|
||||
| `SGLANG_CUSTOM_ALLREDUCE_ALGO` | The algorithm of custom all-reduce. Set to `oneshot` or `1stage` to force use one-shot. Set to `twoshot` or `2stage` to force use two-shot. | `` |
|
||||
| `SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR` | Skip-softmax threshold scale factor for TRT-LLM prefill attention in flashinfer. `None` means standard attention. See https://arxiv.org/abs/2512.12087 | `None` |
|
||||
| `SGLANG_SKIP_SOFTMAX_DECODE_THRESHOLD_SCALE_FACTOR` | Skip-softmax threshold scale factor for TRT-LLM decode attention in flashinfer. `None` means standard attention. See https://arxiv.org/abs/2512.12087 | `None` |
|
||||
| `SGLANG_USE_SGL_FA3_KERNEL` | Use sgl-kernel implementation for FlashAttention v3 | `true` |
|
||||
|
||||
|
||||
## DeepGEMM Configuration (Advanced Optimization)
|
||||
|
||||
Reference in New Issue
Block a user