Compare commits

..
Author SHA1 Message Date
sglang-bot 3d8df4b127 chore: update CI test est_time from recent run data 2026-09-28 00:00:30 +00:00
3220 changed files with 50036 additions and 179050 deletions
-21
View File
@@ -1,21 +0,0 @@
# Serving a model? Read its cookbook page first
Before running, deploying, benchmarking, or reproducing a specific model, read
its page under `docs/cookbook/`:
- `docs/cookbook/autoregressive/` — LLM and VLM serving
- `docs/cookbook/diffusion/` — image, video, 3D, robotics
- `docs/cookbook/omni/`, `docs/cookbook/vla/`, `docs/cookbook/specbundle/`,
`docs/cookbook/base/`
That page carries the deployment we recommend to users — GPU count, parallelism
degrees, quantization, and the flags that matter for that model — so it is the
answer to "how should this be served", and the baseline any tuning starts from.
Configs under `test/` are a different thing: pinned, reproducible CI setups for
correctness and regression checks, not deployment advice. Reach for them when
you need an exactly reproducible run, and say which cookbook flags you diverged
from — a measurement taken on a configuration nobody deploys describes nothing.
Keep it true in both directions: when a change alters a model's recommended
deployment, update that model's cookbook page in the same PR.
-20
View File
@@ -1,20 +0,0 @@
---
paths:
- "**/bench_*.py"
- "**/benchmark/**/*.py"
- "benchmark/kernels/**"
---
# Kernel benchmarks
How a benchmark is written is up to you. The requirement is only that the numbers are valid.
## Tools
In SGLang we recommend `marker.do_bench` (`python/sglang/kernels/jit/benchmark/marker.py`). It works for any callable, not only JIT kernels, and it measures with a cold L2 by default. It is not mandatory. CUPTI (e.g. CUDA 13 `cupti-python`), the torch profiler / `bench_kineto`, Triton's `do_bench` / proton, `flashinfer.testing`, Nsight Systems / Nsight Compute, or a hand-written loop are all fine. Each pitfall below must be handled, whichever tool you use.
## L2 cache reuse
Timing the same kernel on the same inputs in a loop measures an L2-hot kernel, not the one serving runs. `triton.testing.do_bench_cudagraph` with fixed arguments and bare `for` loops between two events both do this. Measure cold L2 by flushing between calls or by rotating enough input copies to exceed L2 several times over. Hot L2 is acceptable only when the working set far exceeds L2 or when hot L2 is the real serving pattern, and the reason must be stated.
Host-memory (zero-copy) loads do not go through device L2. Peer-GPU (NVLink) reads still see reuse (on remote GPU).
+3 -35
View File
@@ -1,14 +1,10 @@
---
name: add-jit-kernel
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang.kernels JIT infrastructure and public operator groups
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
---
# Tutorial: Adding a New JIT Kernel to SGLang
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
## Goal
@@ -501,32 +497,6 @@ def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) ->
---
## Register the public entry point (required)
Add metadata in `python/sglang/kernels/ops/elementwise/__init__.py` so the
operator appears in the registry without importing its implementation:
```python
from sglang.kernels.registry import register_kernel
from sglang.kernels.spec import CapabilityRequirement, KernelBackend, KernelSpec
register_kernel(
KernelSpec(
op="elementwise.scale",
backend=KernelBackend.JIT,
target="sglang.kernels.ops.elementwise.scale:scale",
capabilities=frozenset({CapabilityRequirement.CUDA}),
)
)
```
Record the devices the implementation actually supports; the CUDA-only scale
example above does not imply that every JIT kernel is CUDA-only. Runtime and
correctness tests import `scale` from `sglang.kernels.ops.elementwise.scale`.
Do not import the implementation eagerly in the group initializer. When an
existing `BaseFusedOp` owns the operation, add the implementation there instead
of registering a second conflicting op/backend pair.
## Step 3 (optional): Tune JIT build flags
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
@@ -641,8 +611,6 @@ if __name__ == "__main__":
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
@@ -651,7 +619,7 @@ Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/j
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"`; if you narrow it, keep every read, in-place, and output tensor.
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning).
- `use_cuda_graph=False` for kernels that can't be captured.
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
@@ -756,7 +724,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
- `python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh` — minimal runnable reference
- `python/sglang/kernels/jit/csrc/add_constant.cuh` — minimal runnable reference
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
+43 -46
View File
@@ -5,10 +5,6 @@ description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
## Goal
@@ -29,7 +25,7 @@ Add a new operation that scales each element of a tensor by a scalar factor:
In addition, every new kernel must ship with:
- **Tests** (pytest)
- **A benchmark script** (`marker.do_bench` from `sglang.kernels.jit.benchmark` — it works for any callable, not only JIT kernels)
- **A benchmark script** (triton.testing)
---
@@ -198,21 +194,6 @@ Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
---
## SGLang integration (required for runtime use)
After exposing the AOT wheel symbol, add a lazy wrapper and a `KernelSpec` under
`python/sglang/kernels/ops/<group>/`. Set `backend=KernelBackend.AOT`, record
actual device/architecture support, and keep `sgl_kernel` imports inside the
implementation path. SGLang runtime and integration tests import that wrapper.
For this example, use `sglang.kernels.ops.elementwise.scale`.
The wheel-level tests below validate its standalone API/build. They do not
replace CI-registered SGLang correctness tests in
`test/registered/kernels/ops/elementwise/test_scale.py` and benchmarks in
`test/registered/kernels/benchmark/elementwise/bench_scale.py`. Follow
`write-sglang-test` for their registration and CI budget; do not register tests
under the shipped `python/sglang/` package.
## Step 6: Write tests (required)
Create `python/sglang/kernels/aot/tests/test_scale.py`:
@@ -260,46 +241,62 @@ if __name__ == "__main__":
## Step 7: Add a benchmark (required)
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
```python
import itertools
import torch
import triton
import triton.testing
import sgl_kernel
from sglang.kernels.jit.benchmark import marker
from sglang.utils import is_in_ci
IS_CI = is_in_ci()
dtypes = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
sizes = [4096] if IS_CI else [2**n for n in range(10, 20)] # 1K … 512K
factors = [2.0]
configs = list(itertools.product(dtypes, sizes))
def sglang_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
sgl_kernel.scale(input, factor, out=out)
def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
return input * factor
def torch_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
torch.mul(input, factor, out=out)
FN_MAP = {"sglang": sglang_scale, "torch": torch_scale}
@marker.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32], [torch.float16])
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096]) # 1K .. 512K
@marker.benchmark("provider", ["sglang", "torch"])
def benchmark(dtype: torch.dtype, size: int, provider: str):
input = torch.randn(size, dtype=dtype, device="cuda")
out = torch.empty_like(input)
return marker.do_bench(
FN_MAP[provider],
# Pass every tensor through input_args (not a closure) so marker rotates
# them across CUDA-graph calls to defeat L2 reuse.
input_args=(input, 2.0, out),
# Bandwidth = bytes(input) + bytes(out), both already in input_args.
memory_output=None,
@triton.testing.perf_report(
triton.testing.Benchmark(
x_names=["dtype", "size"],
x_vals=configs,
line_arg="provider",
line_vals=["sglang", "torch"],
line_names=["SGL Kernel", "PyTorch"],
styles=[("green", "-"), ("red", "--")],
ylabel="µs (median)",
plot_name="scale-performance",
args={},
)
)
def benchmark(dtype, size, provider):
input = torch.randn(size, dtype=dtype, device="cuda")
out = torch.empty_like(input)
factor = 2.0
if provider == "sglang":
fn = lambda: sgl_kernel.scale(input, factor, out=out)
else:
fn = lambda: torch_scale(input, factor)
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
fn, quantiles=[0.5, 0.2, 0.8]
)
return 1000 * ms, 1000 * max_ms, 1000 * min_ms
if __name__ == "__main__":
benchmark.run()
benchmark.run(print_data=True)
```
---
+2 -2
View File
@@ -136,8 +136,8 @@ than restating.
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
column must be produced by the SAME harness — flag a page that, say, measures one
platform's GSM8K with a historical raw completion harness and another's with
`sgl-eval run gsm8k --num-examples 1319` and shows both as one "GSM8K %"
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
`run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
(the scores aren't comparable). Either standardize on one harness (matching
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
+1 -1
View File
@@ -162,7 +162,7 @@ Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only f
For a rename where the old key must keep working with a warning:
```python
SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME")
SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK")
```
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
+3 -1
View File
@@ -46,11 +46,13 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
### Step 3: Validate accuracy (sanity check)
```bash
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
```
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- This is a quick sanity check, not a rigorous benchmark.
- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
### Step 4: Generate the profile
@@ -1,82 +0,0 @@
---
name: kernel-organization
description: Apply the SGLang kernels RFC when adding, moving, splitting, or reviewing kernel APIs, registry metadata, kernel tests, benchmarks, and model-specific implementations. Use with add-jit-kernel, add-sgl-kernel, and write-sglang-test for placement and migration checks.
---
# Kernel organization
Read `python/sglang/kernels/README.md` and `test/README.md` before choosing a
location. [RFC #29630](https://github.com/sgl-project/sglang/issues/29630)
establishes the namespace; subsequent migrations #32148 and #40922 clarify
logical grouping and remove the old `_jit_` filename prefix.
## Choose the owner
- Put callable SGLang kernel APIs under `sglang.kernels.ops.<group>`. Runtime
and integration tests import from that namespace, including its submodules.
The `sgl_kernel` wheel retains its own public API and packaging tests.
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
routing in `moe`, attention index selection in `attention`, sampling in
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
quantization alone belongs in `quantization`.
- Model-specific files/subpackages and tuning data are allowed **inside** a
logical group. Do not add a model bundle or an implementation file directly
under `ops/`. Propose a new logical group only for a distinct responsibility,
and update `ops._GROUPS`, documentation, and tests together.
- Classify a fused operator by its complete contract. Do not split a fused
kernel into separate launches just to separate norm, RoPE, or quantization.
Split unrelated public entry points that happen to share a source file.
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
wrappers call it from their logical group. Keep process groups, communicator
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
`srt/layers/communication/` need not pretend to be generic interfaces.
## Register the public entry point
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
existing `BaseFusedOp` registration when the operation has interchangeable
backends. Group imports must not import GPU implementations or compile kernels.
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
existing backend dispatch and describe actual device/architecture restrictions
with `CapabilityRequirement`; JIT/AOT are not device types. Input shape/dtype
checks remain part of the entry point's contract. Registering a torch custom op
does not register it in the SGLang kernel inventory. Predicates, private JIT
factories, reference helpers, and runtime classes are not separate kernel APIs.
## Place and preserve tests
- Kernel numerical tests: `test/registered/kernels/ops/<group>/`.
- CI microbenchmarks: `test/registered/kernels/benchmark/<group>/`.
- Inventory/selector tests: `test/registered/unit/kernels/`.
- Runtime unit tests: `test/registered/unit/<subsystem>/`, following the tested
runtime module. A CUDA allocation does not make a runtime test a kernel test.
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
standalone test/benchmark entry points in production operator modules.
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
execution entry points, and CI stages/runners. Apportion existing time estimates
across split files; do not duplicate the original budget for every output file
or silently drop a registration. Follow `write-sglang-test` for CI registration.
## Verify a migration
1. Search all runtime, test, benchmark, documentation, patch-string, and lazy
registry references before deleting the old path. Check relative imports and
package re-exports as well as direct imports. Do not leave forwarding shims.
2. Keep tuning files with the GEMM that loads them and preserve relative lookup
behavior. Check wheel/package inclusion as well as source-tree execution.
3. Preserve module singleton state: every runtime caller must import the same
new communication adapter, not a second copy of its buffer registry.
4. Separate relocation commits from semantic changes. Use
`mechanical-refactor-verify` to reproduce moves, and compare test inventories
and CI registration coverage before/after. Only delete an experimental path
after checking its call sites, flags, source, and dedicated tests together.
5. Run the namespace/dispatch CPU tests, the registered-test validation hook,
pre-commit, and relevant GPU tests when available. State exactly which GPU
checks ran; import/AST checks do not prove numerical or performance parity.
Do not infer violations solely from a model name inside a group, a missing
`_jit_` prefix, use of a runtime utility, or absence of `BaseFusedOp` inheritance.
@@ -46,8 +46,8 @@ in-flight row as shipped.
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
| --- | --- | --- | --- | --- |
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
@@ -70,8 +70,8 @@ in-flight row as shipped.
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
@@ -148,8 +148,8 @@ Stable entries should be folded into the mainline family rows above.
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
@@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package.
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
@@ -391,7 +391,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
candidate_path=(
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
"<br>python/sglang/srt/layers/layernorm.py"
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"<br>python/sglang/srt/layers/communicator.py"
),
active_keywords=(
"allreduce_fusion",
@@ -419,7 +419,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
pattern="AITER allreduce fusion",
candidate_path=(
"python/sglang/srt/distributed/communication_op.py"
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"<br>python/sglang/srt/layers/communicator.py"
"<br>python/sglang/srt/layers/layernorm.py"
),
active_keywords=(
@@ -721,7 +721,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
FusionPatternSpec(
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
candidate_path=(
"python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"python/sglang/srt/layers/communicator.py"
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
"forward_mla.py"
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
@@ -5,10 +5,6 @@ description: Guide for writing SGLang CI/UT tests. Covers CustomTestCase, CI reg
# Writing SGLang CI / UT Tests
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fail-fast, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md). Whether a case is worth adding at all is decided by [`unit-test-admission`](../../rules/unit-test-admission.md) — read it before writing the case, not after.
## Core Rules
+2 -2
View File
@@ -5,9 +5,9 @@
/docs @wisclmy0611 @zijiexia @Richardczl98 @JustinTong0323 @sogalin
/python/pyproject.toml @merrymercy @Fridge003 @ispobock
/python/sglang/kernels @DarkSharpness @HaiShaw @BBuf @celve @HydraQYH @yuan-luo
/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu @niehen6174
/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu
/python/sglang/kernels/ops/attention/fla @yizhang2077 @hebiao064 @yuan-luo
/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu @kevin-mii @niehen6174
/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu @kevin-mii
/python/sglang/multimodal_gen/runtime/cache @DefTruth
/python/sglang/multimodal_gen/runtime/layers/quantization/ @HaiShaw @BowenBao @OrangeRedeng @TallMessiWu
/python/sglang/multimodal_gen/test/server/ascend @ping1jing2 @ssshinigami @Makcum888e @e-martirosian
+2
View File
@@ -114,6 +114,7 @@ unified-radix-cache:
- changed-files:
- any-glob-to-any-file:
- 'python/sglang/srt/mem_cache/unified_cache/**/*'
- 'python/sglang/srt/mem_cache/cpp_radix_tree/**/*'
- 'python/sglang/srt/mem_cache/**/*radix*'
- 'python/sglang/srt/mem_cache/chunk_cache.py'
- 'python/sglang/srt/mem_cache/evict_policy.py'
@@ -124,6 +125,7 @@ hicache:
- changed-files:
- any-glob-to-any-file:
- '**/*hicache*'
- '**/*hiradix*'
- '**/HiCache/**/*'
- 'benchmark/hicache/**/*'
- 'test/registered/hicache/**/*'
+3 -108
View File
@@ -19,7 +19,7 @@ on:
type: string
required: true
npu_device_type:
description: 'NPU device type, e.g. a3, 950.'
description: 'NPU device type, e.g. 910b, a3.'
type: string
default: 'a3'
run_timeout_minutes:
@@ -39,7 +39,7 @@ on:
type: string
default: ''
skip_pr_test_health_check:
description: 'Set to true to skip the PR test health check (fast-fail gate).'
description: 'Set to true to skip the PR test health check (fail-fast gate).'
type: string
default: 'false'
is_nightly_pipeline_job:
@@ -90,112 +90,7 @@ jobs:
run: |
git config --system --add safe.directory ${GITHUB_WORKSPACE}
- name: Check PR test health
uses: actions/github-script@v8
env:
SKIP_PR_TEST_HEALTH_CHECK: ${{ env.SKIP_PR_TEST_HEALTH_CHECK }}
with:
github-token: ${{ inputs.github-token || github.token }}
script: |
core.notice(`[health-check] START — event=${context.eventName}, runId=${context.runId}`);
// Skip when explicitly requested via env var (e.g. release branch cut)
if (process.env.SKIP_PR_TEST_HEALTH_CHECK === 'true') {
core.notice('[health-check] SKIP: SKIP_PR_TEST_HEALTH_CHECK=true');
return;
}
// Skip for scheduled runs — they should collect all failures, not fast-fail
if (context.eventName === 'schedule') {
core.notice('[health-check] SKIP: scheduled run');
return;
}
// Check lint status from the separate Lint workflow (lint.yml).
// listJobsForWorkflowRun only sees jobs within the SAME run, so we use
// checks.listForRef which queries by commit SHA across ALL workflows.
const ref = context.payload.pull_request?.head?.sha || context.sha;
core.info(`[health-check] Checking lint for ref=${ref}`);
const { data } = await github.rest.checks.listForRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: ref,
check_name: 'lint',
});
const lintRun = data.check_runs.find(
cr => cr.app?.slug === 'github-actions'
);
core.info(`[health-check] Lint check: status=${lintRun?.status}, conclusion=${lintRun?.conclusion}`);
if (lintRun?.status === 'completed' && lintRun?.conclusion === 'failure') {
core.setFailed('Fast-fail: lint check failed');
return;
}
// Skip the jobs-failed check when the PR carries the bypass-fastfail label.
// Lint check above still runs.
let labels = [];
if (context.payload.pull_request?.labels) {
labels = context.payload.pull_request.labels.map(l => l.name);
} else {
const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
owner: context.repo.owner,
repo: context.repo.repo,
commit_sha: ref,
});
if (prs.length > 0) {
labels = prs[0].labels.map(l => l.name);
}
}
core.info(`[health-check] PR labels: [${labels.join(', ')}]`);
if (labels.includes('bypass-fastfail')) {
core.notice('[health-check] SKIP jobs-failed check: bypass-fastfail label present');
return;
}
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.runId,
per_page: 100,
});
core.info(`[health-check] Total jobs in run: ${jobs.length}`);
const failedJobs = jobs.filter(j => j.status === 'completed' && j.conclusion === 'failure');
core.info(`[health-check] Failed jobs (before filtering): ${failedJobs.map(j => `${j.name}(${j.conclusion})`).join(', ') || 'none'}`);
// Find jobs that failed from a real error, not from fast-fail cascade
const rootCauseFailures = jobs.filter(j => {
if (j.status !== 'completed' || j.conclusion !== 'failure') return false;
// h20 runners are flaky (dirty GPU state from prior runs); their failures
// should not cascade fast-fail to other stages. j.name shape from
// listJobsForWorkflowRun: "<job-key>" + optional " / <reusable-job>"
// + optional " (<matrix>)". Split off the base job key before exact
// match so we cover both inline + reusable forms without confusing
// 'h20' with the 'h200' prefix.
const baseName = j.name.split(/[ /]/)[0];
if (baseName === 'base-c-test-8-gpu-h20') {
core.info(`[health-check] Filtered out h20 job: ${j.name}`);
return false;
}
// multimodal-gen NPU tests should not cascade fast-fail to perf/accuracy stages.
if (baseName === 'multimodal-gen-test-1-npu-a3' || baseName === 'multimodal-gen-test-2-npu-a3') {
core.info(`[health-check] Filtered out multimodal-gen NPU job: ${j.name}`);
return false;
}
// If the failing step is the health check, it's a cascade — skip it
const failedStep = (j.steps || []).find(s => s.conclusion === 'failure');
if (failedStep && (failedStep.name.includes('check-pr-test-health') || failedStep.name.includes('Check PR test health'))) {
core.info(`[health-check] Filtered out cascade failure: ${j.name} (failed step: ${failedStep.name})`);
return false;
}
return true;
});
core.info(`[health-check] Root cause failures (after filtering): ${rootCauseFailures.map(j => j.name).join(', ') || 'none'}`);
if (rootCauseFailures.length > 0) {
core.setFailed(`Fast-fail: skipping — root cause job(s): ${rootCauseFailures.map(j => j.name).join(', ')}`);
} else {
core.notice('[health-check] PASS: no root cause failures detected');
}
- uses: ./.github/actions/check-pr-test-health
- name: Install dependencies
env:
@@ -21,7 +21,7 @@ on:
required: true
type: string
description: image for pods
default: "swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3"
default: "swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3"
install_sglang_deps:
required: false
type: boolean
@@ -31,7 +31,7 @@ on:
required: false
type: string
default: 'a3'
description: device type for dependency installation (a3 or 950)
description: device type for dependency installation (a3 or 910b)
transformers_version:
required: false
type: string
@@ -43,7 +43,7 @@ on:
default: '{}'
description: 'JSON run metadata {branch_label, workflow_name, create_time}, recorded once at workflow start'
skip_pr_test_health_check:
description: 'Set to true to skip the PR test health check (fast-fail gate).'
description: 'Set to true to skip the PR test health check (fail-fast gate).'
type: string
default: 'false'
is_nightly_pipeline_job:
@@ -101,112 +101,7 @@ jobs:
# checks out the same commit even if the branch advances mid-run.
ref: ${{ inputs.ref || github.sha }}
- name: Check PR test health
uses: actions/github-script@v8
env:
SKIP_PR_TEST_HEALTH_CHECK: ${{ env.SKIP_PR_TEST_HEALTH_CHECK }}
with:
github-token: ${{ inputs.github-token || github.token }}
script: |
core.notice(`[health-check] START — event=${context.eventName}, runId=${context.runId}`);
// Skip when explicitly requested via env var (e.g. release branch cut)
if (process.env.SKIP_PR_TEST_HEALTH_CHECK === 'true') {
core.notice('[health-check] SKIP: SKIP_PR_TEST_HEALTH_CHECK=true');
return;
}
// Skip for scheduled runs — they should collect all failures, not fast-fail
if (context.eventName === 'schedule') {
core.notice('[health-check] SKIP: scheduled run');
return;
}
// Check lint status from the separate Lint workflow (lint.yml).
// listJobsForWorkflowRun only sees jobs within the SAME run, so we use
// checks.listForRef which queries by commit SHA across ALL workflows.
const ref = context.payload.pull_request?.head?.sha || context.sha;
core.info(`[health-check] Checking lint for ref=${ref}`);
const { data } = await github.rest.checks.listForRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: ref,
check_name: 'lint',
});
const lintRun = data.check_runs.find(
cr => cr.app?.slug === 'github-actions'
);
core.info(`[health-check] Lint check: status=${lintRun?.status}, conclusion=${lintRun?.conclusion}`);
if (lintRun?.status === 'completed' && lintRun?.conclusion === 'failure') {
core.setFailed('Fast-fail: lint check failed');
return;
}
// Skip the jobs-failed check when the PR carries the bypass-fastfail label.
// Lint check above still runs.
let labels = [];
if (context.payload.pull_request?.labels) {
labels = context.payload.pull_request.labels.map(l => l.name);
} else {
const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
owner: context.repo.owner,
repo: context.repo.repo,
commit_sha: ref,
});
if (prs.length > 0) {
labels = prs[0].labels.map(l => l.name);
}
}
core.info(`[health-check] PR labels: [${labels.join(', ')}]`);
if (labels.includes('bypass-fastfail')) {
core.notice('[health-check] SKIP jobs-failed check: bypass-fastfail label present');
return;
}
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.runId,
per_page: 100,
});
core.info(`[health-check] Total jobs in run: ${jobs.length}`);
const failedJobs = jobs.filter(j => j.status === 'completed' && j.conclusion === 'failure');
core.info(`[health-check] Failed jobs (before filtering): ${failedJobs.map(j => `${j.name}(${j.conclusion})`).join(', ') || 'none'}`);
// Find jobs that failed from a real error, not from fast-fail cascade
const rootCauseFailures = jobs.filter(j => {
if (j.status !== 'completed' || j.conclusion !== 'failure') return false;
// h20 runners are flaky (dirty GPU state from prior runs); their failures
// should not cascade fast-fail to other stages. j.name shape from
// listJobsForWorkflowRun: "<job-key>" + optional " / <reusable-job>"
// + optional " (<matrix>)". Split off the base job key before exact
// match so we cover both inline + reusable forms without confusing
// 'h20' with the 'h200' prefix.
const baseName = j.name.split(/[ /]/)[0];
if (baseName === 'base-c-test-8-gpu-h20') {
core.info(`[health-check] Filtered out h20 job: ${j.name}`);
return false;
}
// multimodal-gen NPU tests should not cascade fast-fail to perf/accuracy stages.
if (baseName === 'multimodal-gen-test-1-npu-a3' || baseName === 'multimodal-gen-test-2-npu-a3') {
core.info(`[health-check] Filtered out multimodal-gen NPU job: ${j.name}`);
return false;
}
// If the failing step is the health check, it's a cascade — skip it
const failedStep = (j.steps || []).find(s => s.conclusion === 'failure');
if (failedStep && (failedStep.name.includes('check-pr-test-health') || failedStep.name.includes('Check PR test health'))) {
core.info(`[health-check] Filtered out cascade failure: ${j.name} (failed step: ${failedStep.name})`);
return false;
}
return true;
});
core.info(`[health-check] Root cause failures (after filtering): ${rootCauseFailures.map(j => j.name).join(', ') || 'none'}`);
if (rootCauseFailures.length > 0) {
core.setFailed(`Fast-fail: skipping — root cause job(s): ${rootCauseFailures.map(j => j.name).join(', ')}`);
} else {
core.notice('[health-check] PASS: no root cause failures detected');
}
- uses: ./.github/actions/check-pr-test-health
- name: Check npu info
run: |
+1 -7
View File
@@ -27,8 +27,6 @@ on:
value: ${{ jobs.run.outputs.jit_kernel }}
multimodal_gen:
value: ${{ jobs.run.outputs.multimodal_gen }}
renderer:
value: ${{ jobs.run.outputs.renderer }}
rust_workspace:
value: ${{ jobs.run.outputs.rust_workspace }}
partitions:
@@ -52,7 +50,6 @@ jobs:
sgl_kernel: ${{ steps.filter.outputs.sgl_kernel }}
jit_kernel: ${{ steps.filter.outputs.jit_kernel || steps.run-mode.outputs.run_all_tests }}
multimodal_gen: ${{ steps.filter.outputs.multimodal_gen || steps.run-mode.outputs.run_all_tests }}
renderer: ${{ steps.filter.outputs.renderer || steps.run-mode.outputs.run_all_tests }}
rust_workspace: ${{ steps.filter.outputs.rust_workspace || steps.run-mode.outputs.run_all_tests }}
partitions: ${{ steps.partitions.outputs.partitions }}
partition_model_sha: ${{ steps.partition-model-sha.outputs.sha }}
@@ -99,11 +96,8 @@ jobs:
- "scripts/ci/cuda/*"
- "scripts/ci/utils/*"
- "test/**/!(*.md)"
- "rust/*"
- "rust/!(sglang-renderer)/**"
- "rust/**"
- "proto/sglang/runtime/v1/sglang.proto"
renderer:
- "rust/sglang-renderer/**"
simulator:
- ".github/workflows/pr-test-extra.yml"
- ".github/workflows/_pr-test-simulator-cpu.yml"
+2 -2
View File
@@ -13,7 +13,7 @@ on:
type: string
required: true
check_changes:
description: 'toJson(needs.check-changes.outputs). Read via fromJson(...).main_package / renderer / rust_workspace / continue_on_error / partition_model_sha.'
description: 'toJson(needs.check-changes.outputs). Read via fromJson(...).main_package / rust_workspace / continue_on_error / partition_model_sha.'
type: string
required: true
caller_inputs:
@@ -49,7 +49,7 @@ jobs:
# emits a skipped entry for wait-for-jobs to match. The schedule /
# parallel-dispatch / !failure() half stays on the caller, where the
# `needs` context it guards actually lives.
if: fromJson(inputs.check_changes).main_package == 'true' || fromJson(inputs.check_changes).renderer == 'true'
if: fromJson(inputs.check_changes).main_package == 'true'
runs-on: ubuntu-latest
timeout-minutes: 240
env:
@@ -2,7 +2,7 @@
# Delegates to pr-test-npu.yml with coverage_mode=true, which switches:
# - Runners to the coverage pool (shared /root/.cache with setup-covstub)
# - use_coverage_runner=true on all test jobs
# - Skips pr-gate, multimodal-gen, recommend, analyze
# - Skips pr-gate, base-a (910b), multimodal-gen, recommend, analyze
# - Runs setup-covstub to assemble coverage data and publish it to the
# shared disk at a fixed path (/root/.cache/tests/precise-test/coverage)
# pr-test-npu-finish is the final gate in the called workflow.
@@ -94,7 +94,7 @@ jobs:
matrix: ${{ fromJson(needs.compute-diffusion-partitions.outputs.matrix-1npu) }}
timeout-minutes: 150
container:
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.1.0-a3-ubuntu22.04-py3.12
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -166,7 +166,7 @@ jobs:
matrix: ${{ fromJson(needs.compute-diffusion-partitions.outputs.matrix-2npu) }}
timeout-minutes: 150
container:
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.1.0-a3-ubuntu22.04-py3.12
image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11
steps:
- name: Checkout code
uses: actions/checkout@v4
+29 -2
View File
@@ -18,11 +18,16 @@ on:
required: false
type: string
default: 'all'
image_a2:
description: 'The a2 running docker image of the test task.'
required: false
type: string
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-910b'
image_a3:
description: 'The a3 running docker image of the test task. Must be an image with sglang preinstalled (e.g. lmsysorg/sglang:*a3*).'
required: false
type: string
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3'
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3'
concurrency:
group: full-test-npu-${{ inputs.ref || github.ref }}
@@ -34,6 +39,7 @@ jobs:
outputs:
ref: ${{ steps.set-vars.outputs.ref }}
job_filter: ${{ steps.set-vars.outputs.job_filter }}
image_a2: ${{ steps.set-vars.outputs.image_a2 }}
image_a3: ${{ steps.set-vars.outputs.image_a3 }}
steps:
# When triggered by PR, no inputs parameters are used. The latest community code is tested by default.
@@ -52,8 +58,14 @@ jobs:
echo "job_filter=${{ inputs.job_filter }}" >> $GITHUB_OUTPUT
fi
if [ -z "${{ inputs.image_a2 }}" ]; then
echo "image_a2=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-910b" >> $GITHUB_OUTPUT
else
echo "image_a2=${{ inputs.image_a2 }}" >> $GITHUB_OUTPUT
fi
if [ -z "${{ inputs.image_a3 }}" ]; then
echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3" >> $GITHUB_OUTPUT
echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3" >> $GITHUB_OUTPUT
else
echo "image_a3=${{ inputs.image_a3 }}" >> $GITHUB_OUTPUT
fi
@@ -69,6 +81,20 @@ jobs:
image_a3: ${{ needs.set-image-config.outputs.image_a3 }}
secrets: inherit
full-4-npu-a2:
name: full-4-npu-a2
if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
needs: [set-image-config]
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: linux-aarch64-a2-4
test_type: 'perf'
test_suite: full-4-npu-a2
is_nightly_pipeline_job: true
skip_pr_test_health_check: 'true'
image: ${{ needs.set-image-config.outputs.image_a2 }}
device_type_for_deps: '910b'
full-1-npu-a3:
name: full-1-npu-a3
if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
@@ -291,6 +317,7 @@ jobs:
if: github.repository == 'sgl-project/sglang' && always()
needs:
- nighly-test-npu
- full-4-npu-a2
- full-1-npu-a3
- full-2-npu-a3
- full-4-npu-a3
+18 -23
View File
@@ -1,16 +1,18 @@
name: Nightly Test (AMD MI355X Disagg)
name: Nightly Test (AMD MI355X 2N 1P1D Disagg)
# Self-hosted runners on the amd-sglang login node salloc two or four MI355X
# nodes and run the Docker harness (prefill + decode over MORI, a standalone
# load balancer, then an sglang.bench_serving concurrency sweep) via
# scripts/ci/slurm/launch_mi355x.sh.
on:
schedule:
- cron: '0 7 * * *'
# Temporarily paused while the DI CI nodes are unavailable. Uncomment the
# schedule below when the nodes return.
# schedule:
# - cron: '0 7 * * *'
workflow_dispatch:
inputs:
image:
description: 'Optional. ROCm SGLang Docker image to benchmark. Leave empty for the latest MI35x ROCm 10 image.'
description: 'Optional. ROCm SGLang Docker image to benchmark. Leave empty for the recipe default.'
required: false
default: ''
configs:
@@ -42,38 +44,29 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4
- name: Resolve latest MI35x ROCm 10 nightly image
- name: Resolve latest MI35x nightly image
id: image
# Track the newest published image so the nightly actually catches
# regressions instead of re-testing one pinned build. We resolve on the
# GitHub-hosted runner (clean Docker Hub access, no rate limits) and pass
# the tag to the self-hosted runner. Scope is deliberately tight: only
# the clean main MI35x ROCm 10 line
# `lmsysorg/sglang-rocm:v*-rocm10-mi35x-YYYYMMDD` (excludes -test-,
# miles-, mori-, vllm- variants). If resolution fails, use the pinned
# ROCm 10 image from the last validated full workflow run.
# the clean main MI35x ROCm 7.2 line
# `lmsysorg/sglang-rocm:v*-rocm720-mi35x-YYYYMMDD` (excludes -test-,
# miles-, mori-, vllm-, rocm7_14 variants). On any failure we emit an
# empty string and the launcher falls back to the recipe's pinned
# `runtime.image`.
run: |
set -o pipefail
FALLBACK_IMAGE="lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260922"
RESOLVED=$(curl -fsSL "https://hub.docker.com/v2/repositories/lmsysorg/sglang-rocm/tags?page_size=100&ordering=last_updated" \
| python3 -c "import json,re,sys;p=re.compile(r'^v[0-9][0-9A-Za-z.]*-rocm10-mi35x-([0-9]{8})\$');c=sorted((m.group(1),t['name']) for t in json.load(sys.stdin).get('results',[]) for m in [p.match(t.get('name',''))] if m);print(c[-1][1] if c else '')" || true)
| python3 -c "import json,re,sys;p=re.compile(r'^v[0-9][0-9A-Za-z.]*-rocm720-mi35x-([0-9]{8})\$');c=sorted((m.group(1),t['name']) for t in json.load(sys.stdin).get('results',[]) for m in [p.match(t.get('name',''))] if m);print(c[-1][1] if c else '')" || true)
if [ -n "$RESOLVED" ]; then
echo "Resolved latest MI35x image: lmsysorg/sglang-rocm:$RESOLVED"
echo "image=lmsysorg/sglang-rocm:$RESOLVED" >> $GITHUB_OUTPUT
else
echo "Could not resolve a latest image; using fallback: $FALLBACK_IMAGE"
echo "image=$FALLBACK_IMAGE" >> $GITHUB_OUTPUT
echo "Could not resolve a latest image; launcher will use recipe default."
echo "image=" >> $GITHUB_OUTPUT
fi
- name: Record which image this run uses
env:
DISPATCH_IMAGE: ${{ inputs.image }}
RESOLVED_IMAGE: ${{ steps.image.outputs.image }}
run: |
IMAGE="${DISPATCH_IMAGE:-$RESOLVED_IMAGE}"
echo "### Image" >> "$GITHUB_STEP_SUMMARY"
echo "\`$IMAGE\`" >> "$GITHUB_STEP_SUMMARY"
- name: Generate benchmark matrix
id: generate
env:
@@ -140,6 +133,7 @@ jobs:
CONFIG_FILE: ${{ matrix.config.config_file }}
RESULT_FILENAME: mi355x-${{ matrix.config.name }}
MATRIX_CONFIG_NAME: ${{ matrix.config.name }}
SLURM_EXCLUDE: mia1-p01-g20,mia1-p02-g09
# NOTE: RUNNER_NAME is a built-in default env var on every runner, so the
# launch + cleanup steps read it directly. Do NOT set it here from
# ${{ runner.name }} -- the `runner` context is not available in job-level
@@ -267,7 +261,8 @@ jobs:
- setup
- nightly-mi355x-2n
if: |
!cancelled() && github.repository == 'sgl-project/sglang'
false
&& !cancelled() && github.repository == 'sgl-project/sglang'
&& needs.setup.result == 'success'
&& needs.setup.outputs.matrix-4n != '{"include":[]}'
runs-on: linux-sglang-mi35x-di
+25 -12
View File
@@ -93,8 +93,8 @@ on:
- nightly-8-gpu-glm51
# 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance)
- nightly-8-gpu-mi35x-glm52-fp8
# 8-GPU GLM-5.3-Flash (MI35x)
- nightly-8-gpu-mi35x-glm53-flash
# 8-GPU GLM-5-MXFP4 (MI35x only)
- nightly-8-gpu-mi35x-glm5-mxfp4
# 4-GPU MiniMax-M2.5 (MI35x)
- nightly-4-gpu-mi35x-minimax-m25
# 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance)
@@ -2044,16 +2044,16 @@ jobs:
exit ${TEST_EXIT_CODE:-0}
# ==============================================================================
# 8-GPU GLM-5.3-Flash (MI35x)
# 8-GPU GLM-5-MXFP4 (MI35x only)
# ==============================================================================
nightly-8-gpu-mi35x-glm53-flash:
name: ${{ format('nightly-8-gpu-mi35x-glm53-flash ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }}
nightly-8-gpu-mi35x-glm5-mxfp4:
name: ${{ format('nightly-8-gpu-mi35x-glm5-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }}
strategy:
fail-fast: false
matrix:
rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }}
if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm53-flash,'))
if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm5-mxfp4,'))
runs-on: linux-mi35x-gpu-8
steps:
- name: Checkout code
@@ -2070,21 +2070,34 @@ jobs:
bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }}
env:
GITHUB_WORKSPACE: ${{ github.workspace }}
ENABLE_CACHE_HOST: "1"
- name: Install dependencies
run: |
bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps
bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate
- name: Accuracy Test MI35x ROCm (8-GPU GLM-5.3-Flash DSA+KDA)
- name: Accuracy Test MI35x ROCm (8-GPU GLM-5-MXFP4)
timeout-minutes: 180
run: |
> github_summary.md # Clear summary file
> github_summary.md
bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \
-e SGLANG_USE_AITER=1 \
-e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \
-e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \
python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm53-flash --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$?
python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm5-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$?
echo "$(<github_summary.md )" >> $GITHUB_STEP_SUMMARY || true
exit ${TEST_EXIT_CODE:-0}
- name: Performance Test MI35x ROCm (8-GPU GLM-5-MXFP4)
timeout-minutes: 300
continue-on-error: true
run: |
> github_summary.md
bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \
-e SGLANG_USE_AITER=1 \
-e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \
-e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \
python3 registered/amd/perf/mi35x/test_glm5_mxfp4_perf_mi35x.py || TEST_EXIT_CODE=$?
echo "$(<github_summary.md )" >> $GITHUB_STEP_SUMMARY || true
exit ${TEST_EXIT_CODE:-0}
@@ -2355,8 +2368,8 @@ jobs:
- nightly-8-gpu-glm51
# 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance)
- nightly-8-gpu-mi35x-glm52-fp8
# 8-GPU GLM-5.3-Flash (MI35x)
- nightly-8-gpu-mi35x-glm53-flash
# 8-GPU GLM-5-MXFP4 (MI35x only)
- nightly-8-gpu-mi35x-glm5-mxfp4
# 4-GPU MiniMax-M2.5 (MI35x)
- nightly-4-gpu-mi35x-minimax-m25
# 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance)
+150 -5
View File
@@ -19,16 +19,26 @@ on:
required: false
type: string
default: 'all'
image_a2:
description: 'The a2 running docker image of the test task.'
required: false
type: string
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-910b'
image_a3:
description: 'The a3 running docker image of the test task.'
required: false
type: string
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3'
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3'
image_a3_cann910:
description: 'The a3 running docker image of the test task with cann 9.1.0.'
required: false
type: string
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-nightly'
image_a5:
description: 'The a5 running docker image of the test task.'
required: false
type: string
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-950'
default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-nightly'
skip_install_flag:
description: 'Indicates whether to skip the installation of sglang, defaulting to false.'
required: false
@@ -46,11 +56,21 @@ on:
required: false
type: string
default: 'all'
image_a2:
description: 'The a2 running docker image of the test task.'
required: false
type: string
default: ''
image_a3:
description: 'The a3 running docker image of the test task.'
required: false
type: string
default: ''
image_a3_cann910:
description: 'The a3 running docker image of the test task with cann 9.1.0.'
required: false
type: string
default: ''
image_a5:
description: 'The a5 running docker image of the test task.'
required: false
@@ -70,7 +90,9 @@ jobs:
outputs:
ref: ${{ steps.set-vars.outputs.ref }}
job_filter: ${{ steps.set-vars.outputs.job_filter }}
image_a2: ${{ steps.set-vars.outputs.image_a2 }}
image_a3: ${{ steps.set-vars.outputs.image_a3 }}
image_a3_cann910: ${{ steps.set-vars.outputs.image_a3_cann910 }}
image_a5: ${{ steps.set-vars.outputs.image_a5 }}
skip_install_flag: ${{ steps.set-vars.outputs.skip_install_flag }}
run_start_metadata: ${{ steps.set-vars.outputs.run_start_metadata }}
@@ -93,13 +115,23 @@ jobs:
else
echo "job_filter=${{ inputs.job_filter }}" >> $GITHUB_OUTPUT
fi
if [ -z "${{ inputs.image_a2 }}" ]; then
echo "image_a2=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-910b" >> $GITHUB_OUTPUT
else
echo "image_a2=${{ inputs.image_a2 }}" >> $GITHUB_OUTPUT
fi
if [ -z "${{ inputs.image_a3 }}" ]; then
echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3" >> $GITHUB_OUTPUT
echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3" >> $GITHUB_OUTPUT
else
echo "image_a3=${{ inputs.image_a3 }}" >> $GITHUB_OUTPUT
fi
if [ -z "${{ inputs.image_a3_cann910 }}" ]; then
echo "image_a3_cann910=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-nightly" >> $GITHUB_OUTPUT
else
echo "image_a3_cann910=${{ inputs.image_a3_cann910 }}" >> $GITHUB_OUTPUT
fi
if [ -z "${{ inputs.image_a5 }}" ]; then
echo "image_a5=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-950" >> $GITHUB_OUTPUT
echo "image_a5=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-nightly" >> $GITHUB_OUTPUT
else
echo "image_a5=${{ inputs.image_a5 }}" >> $GITHUB_OUTPUT
fi
@@ -297,6 +329,41 @@ jobs:
install_sglang_deps: false
device_type_for_deps: 'a5'
run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }}
# Suites suffixed with `-cann910` run against the CANN 9.1.0 image.
nightly-perf-16-npu-a3-cann910:
name: nightly-perf-16-npu-a3-cann910
if: ${{ !cancelled() }}
needs: [set-image-config]
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: linux-aarch64-a3-800t-16
test_type: 'perf'
test_suite: nightly-perf-16-npu-a3-cann910
is_nightly_pipeline_job: true
skip_pr_test_health_check: 'true'
image: ${{ needs.set-image-config.outputs.image_a3_cann910 }}
install_sglang_deps: false
device_type_for_deps: 'a3'
run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }}
test_timeout_minutes: '180'
nightly-acc-16-npu-a3-cann910:
name: nightly-acc-16-npu-a3-cann910
if: ${{ !cancelled() }}
needs: [set-image-config]
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: linux-aarch64-a3-16-
test_type: 'accuracy'
test_suite: nightly-acc-16-npu-a3-cann910
is_nightly_pipeline_job: true
skip_pr_test_health_check: 'true'
image: ${{ needs.set-image-config.outputs.image_a3_cann910 }}
install_sglang_deps: false
device_type_for_deps: 'a3'
run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }}
nightly-poc-multi-node-tests:
name: multi-node-poc
if: ${{ !cancelled() }}
@@ -377,6 +444,72 @@ jobs:
transformers_version: ''
run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }}
nightly-poc-multi-node-tests-cann910:
name: multi-node-poc-cann910
if: ${{ !cancelled() }}
# Multi-node jobs run strictly one at a time to avoid contending for the
# same A3 multi-node resources: 9.0.0 multi -> 9.0.0 mix -> cann910 multi -> cann910 mix.
needs: [set-image-config, nightly-perf-16-npu-a3-cann910, nightly-acc-16-npu-a3-cann910, nightly-poc-multi-node-tests, nightly-poc-multi-node-mix-tests]
strategy:
fail-fast: false
max-parallel: 1
matrix:
test_config:
# deepseek_v4_flash performance tests
- name: deepseek_v4_flash_w8a8_1p1d_16p_in8k_out1k_50ms
prefill_size: 1
decode_size: 1
router_size: 1
test_case: test/registered/npu/performance/deepseek_v4_flash/test_npu_deepseek_v4_flash_w8a8_1p1d_16p_in8k_out1k_50ms.py
test_type: 'perf'
prefill_decode_deployment: 'separation'
uses: ./.github/workflows/nightly-test-npu-e2e-multi-node.yml
with:
runner: linux-amd64-cpu-4
test_type: ${{ matrix.test_config.test_type || 'perf' }}
test_config_name: ${{ matrix.test_config.name }}
prefill_size: ${{ matrix.test_config.prefill_size }}
decode_size: ${{ matrix.test_config.decode_size }}
router_size: ${{ matrix.test_config.router_size }}
test_case: ${{ matrix.test_config.test_case }}
image: ${{ needs.set-image-config.outputs.image_a3_cann910 }}
install_sglang_from_source: false
prefill_decode_deployment: ${{ matrix.test_config.prefill_decode_deployment }}
transformers_version: ''
run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }}
nightly-poc-multi-node-mix-tests-cann910:
name: multi-node-mix-poc-cann910
if: false
needs: [set-image-config, nightly-poc-multi-node-tests-cann910]
strategy:
fail-fast: false
max-parallel: 1
matrix:
test_config:
# glm_5_2 accuracy tests
- name: glm_5_2_w8a8_16p_gpqa
node_size: 2
test_case: test/registered/npu/accuracy/glm5_2/test_npu_glm_5_2_w8a8_16p_gpqa.py
test_type: 'accuracy'
# kimi_k3 accuracy tests
- name: kimi_k3_w4a8_32p_gpqa
node_size: 4
test_case: test/registered/npu/accuracy/kimi_k3/test_npu_kimi_k3_w4a8_32p_gpqa.py
test_type: 'accuracy'
uses: ./.github/workflows/nightly-test-npu-e2e-multi-node.yml
with:
runner: linux-amd64-cpu-4
test_type: ${{ matrix.test_config.test_type || 'perf' }}
test_config_name: ${{ matrix.test_config.name }}
node_size: ${{ matrix.test_config.node_size }}
test_case: ${{ matrix.test_config.test_case }}
image: ${{ needs.set-image-config.outputs.image_a3_cann910 }}
install_sglang_from_source: false
prefill_decode_deployment: 'mix'
transformers_version: ''
run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }}
check-all-jobs:
if: ${{ !cancelled() }}
needs:
@@ -390,15 +523,21 @@ jobs:
- nightly-perf-16-npu-a3
- nightly-acc-2-npu-a3
- nightly-acc-16-npu-a3
- nightly-perf-16-npu-a3-cann910
- nightly-acc-16-npu-a3-cann910
- nightly-acc-4-npu-a5
- nightly-poc-multi-node-tests
- nightly-poc-multi-node-mix-tests
- nightly-poc-multi-node-tests-cann910
- nightly-poc-multi-node-mix-tests-cann910
runs-on: ubuntu-latest
steps:
- name: Generate results table
run: |
multi_result="${{ needs.nightly-poc-multi-node-tests.result }}"
mix_result="${{ needs.nightly-poc-multi-node-mix-tests.result }}"
multi_cann910_result="${{ needs.nightly-poc-multi-node-tests-cann910.result }}"
mix_cann910_result="${{ needs.nightly-poc-multi-node-mix-tests-cann910.result }}"
# single-node suites are reported per suite; overall status is success if none failed
single_result="success"
@@ -413,6 +552,8 @@ jobs:
"${{ needs.nightly-perf-16-npu-a3.result }}" \
"${{ needs.nightly-acc-2-npu-a3.result }}" \
"${{ needs.nightly-acc-16-npu-a3.result }}" \
"${{ needs.nightly-perf-16-npu-a3-cann910.result }}" \
"${{ needs.nightly-acc-16-npu-a3-cann910.result }}" \
"${{ needs.nightly-acc-4-npu-a5.result }}"; do
if [ "${r}" != "success" ] && [ "${r}" != "skipped" ]; then
single_result="failure"
@@ -444,6 +585,8 @@ jobs:
"nightly-perf-16-npu-a3:${{ needs.nightly-perf-16-npu-a3.result }}" \
"nightly-acc-2-npu-a3:${{ needs.nightly-acc-2-npu-a3.result }}" \
"nightly-acc-16-npu-a3:${{ needs.nightly-acc-16-npu-a3.result }}" \
"nightly-perf-16-npu-a3-cann910:${{ needs.nightly-perf-16-npu-a3-cann910.result }}" \
"nightly-acc-16-npu-a3-cann910:${{ needs.nightly-acc-16-npu-a3-cann910.result }}" \
"nightly-acc-4-npu-a5:${{ needs.nightly-acc-4-npu-a5.result }}"; do
suite="${entry%%:*}"
r="${entry##*:}"
@@ -451,10 +594,12 @@ jobs:
done
echo "| multi-node-poc | $(group_icon ${multi_result}) ${multi_result} |" >> $GITHUB_STEP_SUMMARY
echo "| multi-node-mix-poc | $(group_icon ${mix_result}) ${mix_result} |" >> $GITHUB_STEP_SUMMARY
echo "| multi-node-poc-cann910 | $(group_icon ${multi_cann910_result}) ${multi_cann910_result} |" >> $GITHUB_STEP_SUMMARY
echo "| multi-node-mix-poc-cann910 | $(group_icon ${mix_cann910_result}) ${mix_cann910_result} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
FAIL=0
for r in "${single_result}" "${multi_result}" "${mix_result}"; do
for r in "${single_result}" "${multi_result}" "${mix_result}" "${multi_cann910_result}" "${mix_cann910_result}"; do
if [ "${r}" != "success" ] && [ "${r}" != "skipped" ]; then FAIL=1; fi
done
exit $FAIL
+8 -36
View File
@@ -9,10 +9,6 @@ on:
description: "Whether the PR must also have the run-ci-extra label (in addition to run-ci)"
type: boolean
default: false
require-label:
description: "Additional label the PR must carry besides run-ci (empty = disabled)"
type: string
default: ''
cool-down-minutes:
description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting"
type: number
@@ -22,15 +18,10 @@ jobs:
pr-gate:
# 1. for commits on main: no gating needed
# 2. for workflow_dispatch: this can only be triggered by users with write access
# 3. for pull_request and pull_request_target: apply the checks below.
# This job only reads PR metadata through the API and never checks out
# or executes PR code, so it is safe for pull_request_target callers.
runs-on: ubuntu-latest
env:
IS_PR_EVENT: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }}
steps:
- name: Fetch latest PR info
if: env.IS_PR_EVENT == 'true'
if: github.event_name == 'pull_request'
id: pr
uses: actions/github-script@v8
with:
@@ -46,7 +37,7 @@ jobs:
core.setOutput("user", pr.data.user.login);
- name: Log PR info
if: env.IS_PR_EVENT == 'true'
if: github.event_name == 'pull_request'
run: |
echo "===== PR Info ====="
echo "PR Event: ${{ github.event_name }}"
@@ -55,18 +46,17 @@ jobs:
echo "PR User: ${{ steps.pr.outputs.user }}"
echo "Require run-ci: ${{ inputs.require-run-ci }}"
echo "Require run-ci-extra: ${{ inputs.require-run-ci-extra }}"
echo "Require label: ${{ inputs.require-label }}"
echo "Cool down minutes: ${{ inputs.cool-down-minutes }}"
echo "==================="
- name: Block draft PR
if: env.IS_PR_EVENT == 'true' && fromJson(steps.pr.outputs.draft)
if: github.event_name == 'pull_request' && fromJson(steps.pr.outputs.draft)
run: |
echo "PR is draft. Blocking CI."
exit 1
- name: Require run-ci label (optional)
if: env.IS_PR_EVENT == 'true' && inputs.require-run-ci == true
if: github.event_name == 'pull_request' && inputs.require-run-ci == true
run: |
labels='${{ steps.pr.outputs.labels }}'
if [[ "${{ contains(fromJson(steps.pr.outputs.labels), 'run-ci') }}" == "false" ]]; then
@@ -80,28 +70,15 @@ jobs:
# then reruns) could never un-skip. A runtime step using gh-api-fetched
# labels lets reruns pick up the new label set.
- name: Require run-ci-extra label (optional)
if: env.IS_PR_EVENT == 'true' && inputs.require-run-ci-extra == true
if: github.event_name == 'pull_request' && inputs.require-run-ci-extra == true
run: |
if [[ "${{ contains(fromJson(steps.pr.outputs.labels), 'run-ci-extra') }}" == "false" ]]; then
echo "Missing required label 'run-ci-extra'. Add the label (e.g. via /tag-and-rerun-ci extra) to opt this PR into the extra test workflow."
exit 1
fi
# Generic opt-in label gate (same live-fetch semantics as run-ci-extra).
# Lets a workflow require its own label on top of run-ci without adding
# a dedicated input per label.
- name: Require additional label (optional)
if: env.IS_PR_EVENT == 'true' && inputs.require-label != ''
env:
REQUIRED_LABEL: ${{ inputs.require-label }}
run: |
if [[ "${{ contains(fromJson(steps.pr.outputs.labels), inputs.require-label) }}" == "false" ]]; then
echo "Missing required label '${REQUIRED_LABEL}' (in addition to 'run-ci'). Add the label to opt this PR into this workflow."
exit 1
fi
- name: Enforce rate limit for low-permission actors (optional)
if: env.IS_PR_EVENT == 'true' && inputs.cool-down-minutes > 0
if: github.event_name == 'pull_request' && inputs.cool-down-minutes > 0
uses: actions/github-script@v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -110,11 +87,6 @@ jobs:
const owner = context.repo.owner;
const repo = context.repo.repo;
const eventName = context.eventName;
// pr-test.yml runs on pull_request, so pull_request_target callers must
// query that event to share the same rate-limit history.
const rateLimitEventName = eventName === "pull_request_target"
? "pull_request"
: eventName;
const curRun = await github.rest.actions.getWorkflowRun({
owner, repo, run_id: context.runId
});
@@ -189,14 +161,14 @@ jobs:
const cutoff = new Date(Date.now() - effectiveCooldownMinutes * 60 * 1000);
core.info(
`Checking for workflow runs since ${cutoff.toISOString()} (last ${effectiveCooldownMinutes} minutes) for event '${rateLimitEventName}'.`
`Checking for workflow runs since ${cutoff.toISOString()} (last ${effectiveCooldownMinutes} minutes) for event '${eventName}'.`
);
const { data } = await github.rest.actions.listWorkflowRuns({
owner,
repo,
workflow_id: 'pr-test.yml',
event: rateLimitEventName,
event: eventName,
per_page: 100,
});
+22 -64
View File
@@ -106,71 +106,29 @@ jobs:
runs-on: ubuntu-latest
outputs:
CANN_image_a3: ${{ steps.set-vars.outputs.CANN_image_a3 }}
CANN_image_910b: ${{ steps.set-vars.outputs.CANN_image_910b }}
steps:
# When triggered by PR, no inputs parameters are used. The latest community code is tested by default.
- name: Set image config
id: set-vars
run: |
echo "CANN_image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.1.0-a3-ubuntu22.04-py3.12" >> $GITHUB_OUTPUT
echo "CANN_image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT
echo "CANN_image_910b=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-910b-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT
# Stage-A gate: CPU-only unit tests. The NPU attention-backend UTs stub out
# torch_npu / sgl_kernel_npu before importing, so a hosted runner suffices.
base-a-test-npu:
needs: [check-changes, pr-gate]
base-a-test-1-npu-a2:
needs: [check-changes, pr-gate, set-image-config]
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.main_package == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 45
env:
SGLANG_IS_IN_CI: true
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
# Pin to the event SHA so this job tests the same commit as the rest of the run.
ref: ${{ inputs.ref || github.sha }}
# Same gate the NPU stages run first: fail before the NPU jobs are scheduled.
- uses: ./.github/actions/check-pr-test-health
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: 'python/pyproject_npu.toml'
- name: Install dependencies
timeout-minutes: 30
env:
# uv pip targets a venv by default; setup-python has no venv.
UV_SYSTEM_PYTHON: "1"
run: |
# Resolve the same dependency set as the NPU stages, minus Ascend-only wheels.
rm -f python/pyproject.toml
mv python/pyproject_npu.toml python/pyproject.toml
# Versions mirror npu_ci_install_dependency.sh's defaults; keep them in sync.
# triton: sglang.srt.utils.common imports it unconditionally and the CPU
# wheel does not pull it in (the NPU image uses triton-ascend instead).
# unsafe-best-match: --index-url is the lowest-priority index, so the
# default strategy would resolve PyPI's CUDA torch, not the +cpu build.
uv pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 triton \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://pypi.org/simple/ \
--index-strategy unsafe-best-match
uv pip install -e "python[dev_npu]" --index-strategy unsafe-best-match
- name: Run NPU unit tests
timeout-minutes: 10
run: |
cd test
python3 run_suite.py --hw npu --suite base-a-test-npu
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
self_name: base-a-test-1-npu-a2
runner_config: linux-aarch64-a2-1
image: ${{ needs.set-image-config.outputs.CANN_image_910b }}
npu_device_type: 910b
run_timeout_minutes: '15'
secrets: inherit
base-b-test-1-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
@@ -184,7 +142,7 @@ jobs:
secrets: inherit
base-b-test-2-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
@@ -198,7 +156,7 @@ jobs:
secrets: inherit
base-b-test-4-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
@@ -213,7 +171,7 @@ jobs:
secrets: inherit
base-b-test-8-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
@@ -227,7 +185,7 @@ jobs:
secrets: inherit
base-b-test-16-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
@@ -241,7 +199,7 @@ jobs:
secrets: inherit
multimodal-gen-test-1-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }}
runs-on: linux-aarch64-a3-800t-2
strategy:
@@ -305,7 +263,7 @@ jobs:
retention-days: 7
multimodal-gen-test-4-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
needs: [check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2]
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }}
runs-on: linux-aarch64-a3-800t-4
strategy:
@@ -525,7 +483,7 @@ jobs:
name: Analyze failure report
needs:
[
base-a-test-npu,
base-a-test-1-npu-a2,
base-b-test-1-npu-a3,
base-b-test-2-npu-a3,
base-b-test-4-npu-a3,
@@ -688,7 +646,7 @@ jobs:
[
check-changes,
base-a-test-npu,
base-a-test-1-npu-a2,
base-b-test-1-npu-a3,
base-b-test-2-npu-a3,
base-b-test-4-npu-a3,
+3 -27
View File
@@ -28,8 +28,8 @@ jobs:
check-changes:
runs-on: ubuntu-latest
outputs:
changes_exist: ${{ steps.filter.outputs.main_package == 'true' || steps.filter.outputs.multimodal_gen == 'true' || steps.xpu-tests.outputs.changed == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
main_package: ${{ steps.filter.outputs.main_package == 'true' || steps.xpu-tests.outputs.changed == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
changes_exist: ${{ steps.filter.outputs.main_package == 'true' || steps.filter.outputs.multimodal_gen == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
main_package: ${{ steps.filter.outputs.main_package == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
multimodal_gen: ${{ steps.filter.outputs.multimodal_gen == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
steps:
- name: Checkout code
@@ -54,38 +54,21 @@ jobs:
uses: dorny/paths-filter@v3
if: steps.run-mode.outputs.run_all_tests != 'true'
with:
list-files: json
filters: |
# Baseline list below; any other changed test/registered/ file with a
# per-commit register_xpu_ci is also picked up by the next step.
registered_tests:
- "test/registered/**/*.py"
main_package:
# Extend test/registered/ entries when adding a non-nightly register_xpu_ci.
- "python/sglang/!(multimodal_gen|kernels|cli|test)/**/!(*.md)"
- "python/sglang/test/*.py"
- "python/sglang/test/!(ascend|observability|mock_model|manual|external_models|kernels)/**/!(*.md)"
- "python/pyproject_xpu.toml"
- "python/sglang/kernels/ops/speculative/@(reconstruct_tree|ngram_corpus).py"
- "python/sglang/kernels/jit/csrc/ngram_corpus/**"
- "test/registered/xpu/**/!(*.md)"
- "test/registered/e2e/xpu/**/!(*.md)"
- "test/registered/disaggregation/test_disaggregation_xpu.py"
- "test/registered/attention/test_chunk_gated_delta_rule.py"
- "test/registered/attention/test_deterministic.py"
- "test/registered/kernels/ops/kvcache/test_hisparse.py"
- "test/registered/kv_canary/test_self_unit_future_tensor.py"
- "test/registered/kv_canary/test_self_unit_plan_input.py"
- "test/registered/kv_canary/test_self_unit_req_to_expected_token_ids_manager.py"
- "test/registered/kv_canary/test_self_unit_runner_swa_divergence.py"
- "test/registered/layers/mamba/test_mamba_slot_fused.py"
- "test/registered/lora/test_moe_lora_info.py"
- "test/registered/lora/test_virtual_experts_kernels.py"
- "test/registered/unit/layers/test_mamba_state_scatter_triton.py"
- "test/registered/unit/managers/test_hisparse_unit.py"
- "test/registered/unit/mem_cache/test_mamba_unittest.py"
- "test/registered/unit/sampling/test_sampling_params.py"
- "test/registered/unit/spec/test_adaptive_spec_params.py"
- "test/registered/unit/spec/test_ngram_corpus.py"
- "test/run_suite.py"
- "test/pytest.ini"
- ".github/workflows/pr-test-xpu.yml"
@@ -111,13 +94,6 @@ jobs:
- "docker/xpu.Dockerfile"
- "scripts/ci/xpu/**"
- name: Detect changed per-commit XPU tests
id: xpu-tests
if: steps.filter.outputs.registered_tests == 'true'
env:
CHANGED_FILES: ${{ steps.filter.outputs.registered_tests_files }}
run: echo "changed=$(python3 scripts/ci/xpu/detect_xpu_test_changes.py)" >> "$GITHUB_OUTPUT"
# ==================== PR Gate ==================== #
pr-gate:
needs: check-changes
+1 -2
View File
@@ -161,8 +161,7 @@ jobs:
needs.check-changes.outputs.main_package == 'true' ||
needs.check-changes.outputs.sgl_kernel == 'true' ||
needs.check-changes.outputs.jit_kernel == 'true' ||
needs.check-changes.outputs.multimodal_gen == 'true' ||
needs.check-changes.outputs.renderer == 'true'
needs.check-changes.outputs.multimodal_gen == 'true'
)
uses: ./.github/workflows/pr-gate.yml
secrets: inherit
@@ -1,4 +1,4 @@
name: Release Docker Images Nightly Miles ROCm7.2.4 (AMD)
name: Release Docker Images Nightly Miles ROCm7.2 (AMD)
on:
workflow_dispatch:
@@ -20,7 +20,7 @@ jobs:
fail-fast: false
matrix:
variant:
- rocm724-mi35x
- rocm720-mi35x
steps:
- name: Checkout miles repository
uses: actions/checkout@v4
@@ -1,5 +1,4 @@
name: Release Docker Images Nightly (NPU)
on:
pull_request:
branches:
@@ -9,41 +8,25 @@ on:
- 'docker/npu.Dockerfile'
workflow_dispatch:
schedule:
- cron: "0 14 * * *"
- cron: "0 14 * * *" # Execute at 10:00 p.m. Beijing Time every day
concurrency:
group: ${{ github.workflow }}-${{ github.sha }}
cancel-in-progress: true
jobs:
build:
name: build ${{ matrix.platform }}-${{ matrix.device_type }}
runs-on: ${{ matrix.runner }}
runs-on: ubuntu-22.04-arm
strategy:
fail-fast: false
matrix:
include:
- device_type: a3
cann_version: "9.1.0"
platform: amd64
runner: ubuntu-latest
- device_type: a3
cann_version: "9.1.0"
platform: arm64
runner: ubuntu-22.04-arm
- device_type: 950
cann_version: "9.1.0"
platform: amd64
runner: ubuntu-latest
- device_type: 950
cann_version: "9.1.0"
platform: arm64
runner: ubuntu-22.04-arm
cann_version: ["9.0.0"]
device_type: ["910b", "a3"]
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Free up disk space
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
with:
tool-cache: true
docker-images: false
@@ -55,64 +38,48 @@ jobs:
id: meta
uses: docker/metadata-action@v5
with:
images: lmsysorg/sglang
images: |
lmsysorg/sglang
# push with schedule event
# push with workflow_dispatch event
tags: |
type=ref,event=pr
type=ref,event=branch
type=schedule,pattern=main
flavor: |
latest=false
suffix=-cann${{ matrix.cann_version }}-${{ matrix.device_type }}-${{ matrix.platform }}
- name: Log into Docker Hub
suffix=-cann${{ matrix.cann_version }}-${{ matrix.device_type }},onlatest=true
# Login against a Docker registry except on PR
# https://github.com/docker/login-action
- name: Log into docker hub
uses: docker/login-action@v3
if: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# Enable Docker multi-architecture build environment
# Emulate non-native architectures
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
# Required for building and pushing multi-arch Docker images
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# Build and push Docker image with Buildx (don't push on PR)
# https://github.com/docker/build-push-action
- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v6
with:
context: docker
file: docker/npu.Dockerfile
platforms: linux/${{ matrix.platform }}
platforms: linux/arm64,linux/amd64
labels: ${{ steps.meta.outputs.labels }}
tags: ${{ steps.meta.outputs.tags }}
outputs: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' && 'type=registry,oci-mediatypes=false' || '' }}
push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
provenance: false
sbom: false
build-args: |
SGLANG_KERNEL_NPU_TAG=2026.9.0.post5
CANN_VERSION=${{ matrix.cann_version }}
DEVICE_TYPE=${{ matrix.device_type }}
merge:
name: merge multi-arch manifest
needs: build
if: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
runs-on: ubuntu-latest
strategy:
matrix:
cann_version: ["9.1.0"]
device_type: ["a3", "950"]
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log into Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Compute base image
run: |
echo "IMAGE=lmsysorg/sglang:${GITHUB_REF_NAME}-cann${{ matrix.cann_version }}-${{ matrix.device_type }}" >> "$GITHUB_ENV"
- name: Create multi-arch manifest
run: |
docker buildx imagetools create \
-t "${IMAGE}" \
"${IMAGE}-amd64" \
"${IMAGE}-arm64"
+2 -2
View File
@@ -14,8 +14,8 @@ jobs:
runs-on: ubuntu-22.04-arm
strategy:
matrix:
cann_version: ["9.1.0"]
device_type: ["a3", "950"]
cann_version: ["9.0.0"]
device_type: ["910b", "a3"]
steps:
- name: Checkout repository
uses: actions/checkout@v4
@@ -1,137 +0,0 @@
name: Publish Renderer Docker Image
on:
push:
branches: [main]
tags: ["v[0-9]+.*"]
paths:
- "rust/sglang-renderer/**"
- "rust/Cargo.toml"
- "rust/Cargo.lock"
- "rust/rust-toolchain.toml"
- "docker/renderer.Dockerfile"
- "LICENSE"
- ".github/workflows/release-docker-renderer.yml"
permissions:
contents: read
jobs:
prepare:
if: github.repository == 'sgl-project/sglang'
runs-on: ubuntu-latest
outputs:
publish: ${{ steps.image.outputs.publish }}
repo: ${{ steps.image.outputs.repo }}
tag: ${{ steps.image.outputs.tag }}
steps:
- name: Select image destination
id: image
run: |
if [[ "$GITHUB_REF_TYPE" == "tag" ]]; then
if [[ ! "$GITHUB_REF_NAME" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(\.post[0-9]+)?$ ]]; then
echo "Skipping non-final release tag $GITHUB_REF_NAME"
echo "publish=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "repo=lmsysorg/sglang" >> "$GITHUB_OUTPUT"
echo "tag=${GITHUB_REF_NAME}-renderer" >> "$GITHUB_OUTPUT"
else
echo "repo=lmsysorg/sglang-staging" >> "$GITHUB_OUTPUT"
echo "tag=renderer-${GITHUB_SHA:0:8}" >> "$GITHUB_OUTPUT"
fi
echo "publish=true" >> "$GITHUB_OUTPUT"
build:
needs: prepare
if: needs.prepare.outputs.publish == 'true'
environment: ${{ github.ref_type == 'tag' && 'prod' || null }}
strategy:
fail-fast: false
matrix:
include:
- platform: linux/amd64
runner: ubuntu-24.04
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
runs-on: ${{ matrix.runner }}
env:
IMAGE_REPO: ${{ needs.prepare.outputs.repo }}
IMAGE_TAG: ${{ needs.prepare.outputs.tag }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Run renderer tests
run: |
docker buildx build . -f docker/renderer.Dockerfile \
--platform ${{ matrix.platform }} --target test --progress=plain
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push by digest
run: |
BUILD_COMMIT=$(git rev-parse HEAD)
BUILD_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
docker buildx build . -f docker/renderer.Dockerfile \
--platform ${{ matrix.platform }} --target runtime --progress=plain \
--build-arg "SGLANG_BUILD_COMMIT=${BUILD_COMMIT}" \
--build-arg "SGLANG_BUILD_URL=${BUILD_URL}" \
--build-arg "SGLANG_IMAGE_TAG=${IMAGE_REPO}:${IMAGE_TAG}" \
--output "type=image,name=${IMAGE_REPO},push-by-digest=true,name-canonical=true,push=true" \
--metadata-file /tmp/renderer-metadata.json
DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/renderer-metadata.json'))['containerimage.digest'])")
docker run --rm "${IMAGE_REPO}@${DIGEST}" --help
mkdir -p /tmp/renderer-digests
echo "$DIGEST" > "/tmp/renderer-digests/${{ matrix.arch }}"
- name: Upload digest
uses: actions/upload-artifact@v4
with:
name: renderer-digest-${{ matrix.arch }}
path: /tmp/renderer-digests/${{ matrix.arch }}
retention-days: 7
publish:
needs: [prepare, build]
if: needs.prepare.outputs.publish == 'true'
environment: ${{ github.ref_type == 'tag' && 'prod' || null }}
runs-on: ubuntu-24.04
env:
IMAGE_REPO: ${{ needs.prepare.outputs.repo }}
IMAGE_TAG: ${{ needs.prepare.outputs.tag }}
steps:
- name: Download digests
uses: actions/download-artifact@v4
with:
path: /tmp/renderer-digests
pattern: renderer-digest-*
merge-multiple: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Publish multi-architecture tag
run: |
AMD64_DIGEST=$(cat /tmp/renderer-digests/amd64)
ARM64_DIGEST=$(cat /tmp/renderer-digests/arm64)
docker buildx imagetools create \
-t "${IMAGE_REPO}:${IMAGE_TAG}" \
"${IMAGE_REPO}@${AMD64_DIGEST}" \
"${IMAGE_REPO}@${ARM64_DIGEST}"
docker buildx imagetools inspect "${IMAGE_REPO}:${IMAGE_TAG}"
+8 -98
View File
@@ -86,43 +86,8 @@ permissions:
issues: read
jobs:
prepare-partitions:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
outputs:
matrix: ${{ steps.partition.outputs.matrix }}
steps:
- name: Checkout partition script
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}
persist-credentials: false
- name: Checkout test registrations
uses: actions/checkout@v4
with:
ref: ${{ inputs.pr_head_sha || github.sha }}
path: partition-source
persist-credentials: false
- name: Partition tests by estimated duration
id: partition
env:
TEST_COMMAND: ${{ inputs.test_command }}
TEST_ROOT: partition-source
MODE: ${{ inputs.mode }}
RUNS_ON: ${{ inputs.runs_on }}
run: python3 scripts/ci/partition_rerun_tests.py
rerun-test-cuda:
name: rerun-test-cuda (partition ${{ matrix.partition }})
needs: prepare-partitions
if: inputs.mode == 'cuda'
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare-partitions.outputs.matrix) }}
runs-on: ${{ inputs.runs_on }}
timeout-minutes: 120
permissions:
@@ -185,8 +150,7 @@ jobs:
- name: Run test
timeout-minutes: 60
env:
TEST_COMMAND: ${{ matrix.test_command }}
REQUESTED_TEST_COMMAND: ${{ inputs.test_command }}
TEST_COMMAND: ${{ inputs.test_command }}
SGLANG_PRECISION_HF_TOKEN: ${{ inputs.refresh_precision_baseline && inputs.mode == 'cuda' && inputs.runs_on == '8-gpu-h200' && inputs.test_command == 'registered/debug_utils/test_nightly_precision_regression.py' && inputs.pr_head_sha == '' && secrets.HF_TOKEN_PRECISION_STORE || '' }}
run: |
if [[ "${{ inputs.refresh_precision_baseline }}" == "true" ]]; then
@@ -194,7 +158,7 @@ jobs:
if [[ "${{ inputs.mode }}" != "cuda" \
|| "${{ inputs.runs_on }}" != "8-gpu-h200" \
|| -n "${{ inputs.pr_head_sha }}" \
|| "$REQUESTED_TEST_COMMAND" != "$expected" ]]; then
|| "$TEST_COMMAND" != "$expected" ]]; then
echo "::error::Precision baseline refresh only accepts $expected on 8-gpu-h200"
exit 1
fi
@@ -208,7 +172,6 @@ jobs:
cmds+=("$cmd")
done <<< "$TEST_COMMAND"
total=${#cmds[@]}
failed=()
suite_start=$SECONDS
for idx in "${!cmds[@]}"; do
i=$((idx + 1))
@@ -226,9 +189,7 @@ jobs:
if grep -qE 'register_cuda_ci\(.*stage[[:space:]]*=[[:space:]]*"base-a"' "test/$test_file" 2>/dev/null; then
async_assert=false
fi
if ! (cd test/ && SGLANG_ENABLE_ASYNC_ASSERT=$async_assert python3 $cmd -f); then
failed+=("$cmd")
fi
(cd test/ && SGLANG_ENABLE_ASYNC_ASSERT=$async_assert python3 $cmd -f) || exit 1
elapsed=$(( SECONDS - file_start ))
echo "."
echo "End ($i/$total): elapsed=${elapsed}s"
@@ -236,29 +197,13 @@ jobs:
echo ""
done
total_elapsed=$(( SECONDS - suite_start ))
if [ ${#failed[@]} -gt 0 ]; then
echo "${#failed[@]}/$total test(s) failed in ${total_elapsed}s:"
printf ' %s\n' "${failed[@]}"
{
echo "### Failed tests (partition ${{ matrix.partition }})"
printf -- '- `%s`\n' "${failed[@]}"
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
echo "All $total test(s) passed in ${total_elapsed}s"
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
with:
artifact-suffix: ${{ matrix.partition }}
rerun-test-multimodal-gen:
name: rerun-test-multimodal-gen (partition ${{ matrix.partition }})
needs: prepare-partitions
if: inputs.mode == 'multimodal_gen'
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare-partitions.outputs.matrix) }}
runs-on: ${{ inputs.runs_on }}
timeout-minutes: 120
permissions:
@@ -299,16 +244,13 @@ jobs:
- name: Run test
timeout-minutes: 60
env:
TEST_COMMAND: ${{ matrix.test_command }}
run: |
cmds=()
while IFS= read -r cmd; do
[ -z "$cmd" ] && continue
cmds+=("$cmd")
done <<< "$TEST_COMMAND"
done <<< "${{ inputs.test_command }}"
total=${#cmds[@]}
failed=()
suite_start=$SECONDS
for idx in "${!cmds[@]}"; do
i=$((idx + 1))
@@ -318,9 +260,7 @@ jobs:
echo "Begin ($i/$total): python3 -m pytest $cmd -x"
echo "."
file_start=$SECONDS
if ! python3 -m pytest $cmd -x; then
failed+=("$cmd")
fi
python3 -m pytest $cmd -x || exit 1
elapsed=$(( SECONDS - file_start ))
echo "."
echo "End ($i/$total): elapsed=${elapsed}s"
@@ -328,29 +268,13 @@ jobs:
echo ""
done
total_elapsed=$(( SECONDS - suite_start ))
if [ ${#failed[@]} -gt 0 ]; then
echo "${#failed[@]}/$total test(s) failed in ${total_elapsed}s:"
printf ' %s\n' "${failed[@]}"
{
echo "### Failed tests (partition ${{ matrix.partition }})"
printf -- '- `%s`\n' "${failed[@]}"
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
echo "All $total test(s) passed in ${total_elapsed}s"
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
with:
artifact-suffix: ${{ matrix.partition }}
rerun-test-cpu:
name: rerun-test-cpu (partition ${{ matrix.partition }})
needs: prepare-partitions
if: inputs.mode == 'cpu'
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare-partitions.outputs.matrix) }}
runs-on: ubuntu-latest
timeout-minutes: 120
permissions:
@@ -419,17 +343,14 @@ jobs:
- name: Run test
timeout-minutes: 60
env:
TEST_COMMAND: ${{ matrix.test_command }}
run: |
cd test/
cmds=()
while IFS= read -r cmd; do
[ -z "$cmd" ] && continue
cmds+=("$cmd")
done <<< "$TEST_COMMAND"
done <<< "${{ inputs.test_command }}"
total=${#cmds[@]}
failed=()
suite_start=$SECONDS
for idx in "${!cmds[@]}"; do
i=$((idx + 1))
@@ -439,9 +360,7 @@ jobs:
echo "Begin ($i/$total): python3 $cmd"
echo "."
file_start=$SECONDS
if ! python3 $cmd -f; then
failed+=("$cmd")
fi
python3 $cmd -f || exit 1
elapsed=$(( SECONDS - file_start ))
echo "."
echo "End ($i/$total): elapsed=${elapsed}s"
@@ -449,19 +368,10 @@ jobs:
echo ""
done
total_elapsed=$(( SECONDS - suite_start ))
if [ ${#failed[@]} -gt 0 ]; then
echo "${#failed[@]}/$total test(s) failed in ${total_elapsed}s:"
printf ' %s\n' "${failed[@]}"
{
echo "### Failed tests (partition ${{ matrix.partition }})"
printf -- '- `%s`\n' "${failed[@]}"
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
echo "All $total test(s) passed in ${total_elapsed}s"
write-back-result:
needs: [prepare-partitions, rerun-test-cuda, rerun-test-multimodal-gen, rerun-test-cpu]
needs: [rerun-test-cuda, rerun-test-multimodal-gen, rerun-test-cpu]
if: always() && inputs.reply_comment_id != '' && inputs.reply_marker != ''
runs-on: ubuntu-latest
permissions:
+2
View File
@@ -166,6 +166,8 @@ cython_debug/
*.swp
# SGL
benchmark/mmlu/data
benchmark/mmlu/data.tar
benchmark/llava_bench/images
benchmark/llava_bench/mme_pack
*.jsonl
+1 -1
View File
@@ -35,7 +35,7 @@ repos:
hooks:
- id: ruff
args:
- --select=F401,F821,F823,UP037
- --select=F401,F821,UP037
- --fix
files: ^(benchmark/|docs/|examples/|python/sglang/|sgl-model-gateway/py_*|test/)
exclude: |
-1
View File
@@ -75,7 +75,6 @@ Its core features include:
## Getting Started
- [Install SGLang](https://docs.sglang.io/get_started/install.html)
- [Quick Start](https://docs.sglang.io/basic_usage/send_request.html)
- [Cookbook](https://docs.sglang.io/cookbook) — the deployment command we recommend for each supported model
- [Backend Tutorial](https://docs.sglang.io/basic_usage/openai_api_completions.html)
- [Frontend Tutorial](https://docs.sglang.io/references/frontend/frontend_tutorial.html)
- [Contribution Guide](https://docs.sglang.io/developer_guide/contribution_guide.html)
+3 -7
View File
@@ -271,9 +271,7 @@ Then we can benchmark the accuracy and latency by accessing the first node's exp
```bash
# bench accuracy
sgl-eval run gsm8k \
--base-url http://10.0.0.1:30000/v1 \
--num-examples 1319 --num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --num-questions 1319 --host 10.0.0.1 --port 30000
# bench latency
python3 -m sglang.bench_one_batch_server --model None --base-url http://10.0.0.1:30000 --batch-size 1 --input-len 128 --output-len 128
@@ -324,15 +322,13 @@ Then on the **master node**, supposing the ShareGPT data is located at `/path/to
```bash
# bench accuracy
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 --num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --num-questions 1319
# bench serving
python3 -m sglang.bench_serving --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --random-input 128 --random-output 128 --num-prompts 1000 --request-rate 128 --random-range-ratio 1.0
```
> **Note: using `--num-threads 200` can accelerate accuracy benchmarking**.
> **Note: using `--parallel 200` can accelerate accuracy benchmarking**.
### Example: Serving with 32 L40S with int8 Quantization
-27
View File
@@ -1,27 +0,0 @@
# DiffusionGemma serving measurements
`bench_diffusion_gemma.py` exercises a running server through its completions API using pretokenized chat prompts. For completed cases it records trial timings, returned text, prompt hashes, and usage counts. It rejects empty responses and unexpected token counts.
Start the server with the checkpoint and supported serving defaults:
```sh
python -m sglang.launch_server --model-path "$MODEL" \
--dllm-algorithm Gemma4Renoise
```
Point the client at that server and the same tokenizer:
```sh
python benchmark/dllm/bench_diffusion_gemma.py \
--url http://127.0.0.1:30000 --tokenizer "$MODEL" \
--input-lengths 128 2048 --concurrencies 1 4 8 --output-length 256 \
--run-id serving-check --result result.json
```
## Interpretation
The client disables EOS stopping to request a fixed output length. Canvas positions after EOS count toward the reported token rate, which is not useful-text throughput. `--denoising-steps` records metadata and does not configure or verify the server's convergence settings. Its default is unset. Set it only when the actual server work has been verified. If a run fails, keep both server and client logs, since the result file contains only completed cases.
For comparisons, freeze the protocol before collecting results. Match checkpoint identity, physical hardware, precision, prompts, resource limits, generation settings, and stopping policy. Use the same run ID to generate identical prompts. Give each runtime equivalent access to its normal compilation and graph features, preserve every trial and failed request, and disclose dependency and implementation differences. Validate normal text, images, mixed requests, and multi-block generation separately.
Performance for the merged revision remains unmeasured. Earlier measurements and test counts do not validate this revision. A fixed-work experiment does not establish adaptive-generation performance, model accuracy, or a serving performance lead.
-166
View File
@@ -1,166 +0,0 @@
"""Compare fixed-work DiffusionGemma serving through the completions API.
Both servers must use the same checkpoint, canvas length, denoising limit and
stopping settings. Set confidence_threshold=0.0 and stability_threshold=1 on
both servers so early convergence cannot shorten work. This keeps the normal
history size instead of adding history work to enforce the step count. Run them
sequentially on the same physical GPUs, using the same run ID and arguments.
Each request has a unique prompt prefix to avoid prefix-cache reuse.
This measures complete canvas computation with EOS stopping disabled. Canvas
tokens per second include tokens after EOS and are not useful-text throughput.
Validate output quality separately with the model's default generation settings.
The denoising-steps argument records the server configuration; it does not change it.
"""
import argparse
import asyncio
import hashlib
import json
import statistics
import time
from pathlib import Path
import aiohttp
from transformers import AutoTokenizer
async def benchmark(args):
tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
body = tokenizer.encode(
"The city library has books about science, history, music and art. " * 256,
add_special_tokens=False,
)
template = tokenizer.apply_chat_template(
[
{
"role": "user",
"content": (
"Request __REQUEST_ID__.\n__BENCHMARK_BODY__\n"
"Write a detailed explanation of why public libraries are useful."
),
}
],
tokenize=False,
add_generation_prompt=True,
enable_thinking=False,
)
prefix_template, suffix_text = template.split("__BENCHMARK_BODY__")
suffix = tokenizer.encode(suffix_text, add_special_tokens=False)
results = []
request_number = 0
timeout = aiohttp.ClientTimeout(total=600)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(args.url + "/v1/models") as response:
response.raise_for_status()
model = (await response.json())["data"][0]["id"]
async def request(prompt):
start = time.perf_counter()
async with session.post(
args.url + "/v1/completions",
json={
"model": model,
"prompt": prompt,
"max_tokens": args.output_length,
"ignore_eos": True,
"stream": False,
},
) as response:
payload = await response.json()
if response.status != 200:
raise RuntimeError(f"HTTP {response.status}: {payload}")
elapsed = time.perf_counter() - start
if not payload["choices"][0]["text"].strip():
raise RuntimeError(
"Empty completion; check the chat template and model"
)
usage = payload["usage"]
if usage["completion_tokens"] != args.output_length:
raise RuntimeError(f"Unexpected output length: {usage}")
if usage["prompt_tokens"] != len(prompt):
raise RuntimeError(f"Unexpected prompt length: {usage}")
return {
"latency_s": elapsed,
"usage": usage,
"text": payload["choices"][0]["text"],
"prompt_sha256": hashlib.sha256(
json.dumps(prompt).encode()
).hexdigest(),
}
for input_length in args.input_lengths:
for concurrency in args.concurrencies:
trials = []
for trial in range(args.warmups + args.trials):
prompts = []
for _ in range(concurrency):
# Vary the beginning of every prompt to avoid measuring
# cached prompts. Keep run_id identical across runtimes.
prefix = tokenizer.encode(
prefix_template.replace(
"__REQUEST_ID__", f"{args.run_id}-{request_number}"
),
add_special_tokens=False,
)
request_number += 1
n = input_length - len(prefix) - len(suffix)
if n < 0:
raise ValueError("Input length is too short")
prompts.append(prefix + body[:n] + suffix)
start = time.perf_counter()
requests = await asyncio.gather(*(request(p) for p in prompts))
trials.append(
{
"warmup": trial < args.warmups,
"wall_s": time.perf_counter() - start,
"requests": requests,
}
)
measured = trials[args.warmups :]
wall_times = [t["wall_s"] for t in measured]
latencies = [r["latency_s"] for t in measured for r in t["requests"]]
result = {
"input_length": input_length,
"output_length": args.output_length,
"concurrency": concurrency,
"median_wall_s": statistics.median(wall_times),
"median_request_s": statistics.median(latencies),
"canvas_tokens_per_s": concurrency
* args.output_length
/ statistics.median(wall_times),
"trials": trials,
}
results.append(result)
print(
json.dumps({k: v for k, v in result.items() if k != "trials"}),
flush=True,
)
args.result.write_text(
json.dumps(
{
"settings": vars(args) | {"result": str(args.result)},
"results": results,
},
indent=2,
)
)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--url", required=True)
parser.add_argument("--tokenizer", required=True)
parser.add_argument("--input-lengths", type=int, nargs="+", default=[128, 512])
parser.add_argument("--output-length", type=int, default=256)
parser.add_argument(
"--denoising-steps",
type=int,
help="Verified fixed denoising step count, recorded as metadata only",
)
parser.add_argument("--concurrencies", type=int, nargs="+", default=[1, 4, 8])
parser.add_argument("--warmups", type=int, default=3)
parser.add_argument("--trials", type=int, default=15)
parser.add_argument("--run-id", required=True)
parser.add_argument("--result", type=Path, required=True)
asyncio.run(benchmark(parser.parse_args()))
+20
View File
@@ -0,0 +1,20 @@
## Run benchmark
### Using GSM8K Platinum
GSM8K Platinum is a revised version of the GSM8K test set with corrected labels and removed ambiguous questions. It can be more stable than the original GSM8K dataset. It's a drop-in replacement that can be used by adding the `--platinum` flag:
```
python3 bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
```
For more information, see: https://huggingface.co/datasets/madrylab/gsm8k-platinum
### Benchmark sglang
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
```
```
python3 bench_sglang.py --num-questions 200
```
+207
View File
@@ -0,0 +1,207 @@
import argparse
import ast
import json
import os
import re
import time
import numpy as np
from datasets import load_dataset
from sglang.lang.api import set_default_backend
from sglang.test.test_utils import (
add_common_sglang_args_and_parse,
dump_bench_raw_result,
select_sglang_backend,
)
from sglang.utils import download_and_cache_file, dump_state_text, read_jsonl
INVALID = -9999999
def get_one_example(lines, i, include_answer):
ret = "Question: " + lines[i]["question"] + "\nAnswer:"
if include_answer:
ret += " " + lines[i]["answer"]
return ret
def get_few_shot_examples(lines, k):
ret = ""
for i in range(k):
ret += get_one_example(lines, i, True) + "\n\n"
return ret
def get_answer_value(answer_str):
answer_str = answer_str.replace(",", "")
numbers = re.findall(r"\d+", answer_str)
if len(numbers) < 1:
return INVALID
try:
return ast.literal_eval(numbers[-1])
except SyntaxError:
return INVALID
def main(args):
# Select backend
set_default_backend(select_sglang_backend(args))
# Load tokenizer if enable_thinking is set
tokenizer = None
if args.enable_thinking:
from transformers import AutoTokenizer
assert args.tokenizer_path is not None, (
"--tokenizer-path is required when --enable-thinking is set"
)
tokenizer = AutoTokenizer.from_pretrained(
args.tokenizer_path, trust_remote_code=True
)
# Read data
if args.platinum:
print("Loading GSM8K Platinum dataset from HuggingFace...")
dataset = load_dataset("madrylab/gsm8k-platinum", "main", split="test")
lines = [
{"question": item["question"], "answer": item["answer"]} for item in dataset
]
else:
data_path = args.data_path
url = "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl"
if not os.path.isfile(data_path):
data_path = download_and_cache_file(url)
lines = list(read_jsonl(data_path))
# Construct prompts
num_questions = args.num_questions
num_shots = args.num_shots
few_shot_examples = get_few_shot_examples(lines, num_shots)
questions = []
labels = []
for i in range(len(lines[:num_questions])):
raw_question = few_shot_examples + get_one_example(lines, i, False)
if tokenizer is not None:
messages = [{"role": "user", "content": raw_question}]
raw_question = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
**args.chat_template_kwargs,
)
questions.append(raw_question)
labels.append(get_answer_value(lines[i]["answer"]))
assert all(l != INVALID for l in labels)
arguments = [{"question": q} for q in questions]
#####################################
######### SGL Program Begin #########
#####################################
import sglang as sgl
@sgl.function
def few_shot_gsm8k(s, question):
s += question
s += sgl.gen(
"answer",
max_tokens=args.max_new_tokens,
stop=["Question", "Assistant:", "<|separator|>"],
)
#####################################
########## SGL Program End ##########
#####################################
# Run requests
tic = time.perf_counter()
states = few_shot_gsm8k.run_batch(
arguments,
temperature=args.temperature,
top_p=args.top_p,
num_threads=args.parallel,
progress_bar=True,
)
latency = time.perf_counter() - tic
preds = []
for i in range(len(states)):
preds.append(get_answer_value(states[i]["answer"]))
# Compute accuracy
acc = np.mean(np.array(preds) == np.array(labels))
invalid = np.mean(np.array(preds) == INVALID)
# Compute speed
num_output_tokens = sum(
s.get_meta_info("answer")["completion_tokens"] for s in states
)
output_throughput = num_output_tokens / latency
# Print results
print(f"Accuracy: {acc:.3f}")
print(f"Invalid: {invalid:.3f}")
print(f"Latency: {latency:.3f} s")
print(f"Output throughput: {output_throughput:.3f} token/s")
# Dump results
dump_state_text(f"tmp_output_{args.backend}.txt", states)
dump_bench_raw_result(
path=args.raw_result_file,
states=states,
preds=preds,
labels=labels,
)
with open(args.result_file, "a") as fout:
value = {
"task": "gsm8k-platinum" if args.platinum else "gsm8k",
"backend": args.backend,
"num_gpus": 1,
"latency": round(latency, 3),
"accuracy": round(acc, 3),
"num_requests": args.num_questions,
"other": {
"num_questions": args.num_questions,
"parallel": args.parallel,
},
}
fout.write(json.dumps(value) + "\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--num-shots", type=int, default=5)
parser.add_argument("--data-path", type=str, default="test.jsonl")
parser.add_argument("--num-questions", type=int, default=200)
parser.add_argument("--max-new-tokens", type=int, default=512)
parser.add_argument("--temperature", type=float, default=0.0)
parser.add_argument("--top-p", type=float, default=1.0)
parser.add_argument(
"--enable-thinking",
action="store_true",
help="Enable thinking mode by wrapping prompts with chat template",
)
parser.add_argument(
"--chat-template-kwargs",
type=json.loads,
default='{"enable_thinking": true}',
help="JSON dict passed through to tokenizer.apply_chat_template. "
"The thinking-toggle kwarg name is model-specific, e.g. "
"'{\"enable_thinking\": true}' (Qwen) or '{\"thinking\": true}' (Kimi).",
)
parser.add_argument(
"--tokenizer-path",
type=str,
default=None,
help="Path to tokenizer (required when --enable-thinking is set)",
)
parser.add_argument(
"--platinum",
action="store_true",
help="Use GSM8K Platinum dataset (drop-in replacement with corrected labels)",
)
args = add_common_sglang_args_and_parse(parser)
main(args)
-198
View File
@@ -1,198 +0,0 @@
"""CPU comparison of baseline and optimized offset-based scheduler padding.
Run from the repository root:
PYTHONPATH=python python benchmark/io/bench_mm_token_padding.py --baseline-ref BASE_SHA
Inputs are synthetic. Both paths include the scheduler's final ownership copy.
This measures padding CPU time, not model forward time or end-to-end TTFT.
"""
import argparse
import ast
import gc
import json
import platform
import statistics
import subprocess
import time
from array import array
from pathlib import Path
import torch
from sglang.srt.managers import mm_utils
from sglang.srt.managers.mm_utils import (
MultiModalityDataPaddingPatternMultimodalTokens,
duplicate_pad_mm_input_ids,
)
from sglang.srt.managers.schedule_batch import (
Modality,
MultimodalDataItem,
MultimodalInputs,
)
def load_baseline(ref):
"""Load the exact baseline helper classes without replacing installed SGLang."""
root = Path(__file__).resolve().parents[2]
commit = subprocess.check_output(
["git", "rev-parse", "--verify", f"{ref}^{{commit}}"], cwd=root, text=True
).strip()
path = "python/sglang/srt/managers/mm_utils.py"
source = subprocess.check_output(
["git", "show", f"{commit}:{path}"], cwd=root, text=True
)
names = {
"MultiModalityDataPaddingPattern",
"MultiModalityDataPaddingPatternMultimodalTokens",
}
classes = [
node
for node in ast.parse(source).body
if isinstance(node, ast.ClassDef) and node.name in names
]
if {node.name for node in classes} != names:
raise ValueError("Baseline does not contain the expected padding helpers")
namespace = dict(vars(mm_utils))
# Reuse dependency imports, but compile the original method bodies verbatim.
exec(
compile(ast.Module(body=classes, type_ignores=[]), f"{commit}:{path}", "exec"),
namespace,
)
return commit, namespace["MultiModalityDataPaddingPatternMultimodalTokens"]()
def make_case(num_tokens, num_items, span_tokens):
if num_items and num_tokens < 3 * num_items:
raise ValueError(
"Each item needs three tokens for its span and surrounding text"
)
input_ids = array("q", range(257, 257 + num_tokens))
items = []
stride = num_tokens // max(num_items, 1)
for index in range(num_items):
start = index * stride + 1
end = start + min(span_tokens, stride - 2) - 1
items.append(
MultimodalDataItem(
modality=Modality.IMAGE,
pad_value=2**40 + index,
offsets=[(start, end)],
)
)
return input_ids, MultimodalInputs(mm_items=items, im_token_id=1)
def benchmark_case(num_tokens, num_items, args, baseline):
pattern = MultiModalityDataPaddingPatternMultimodalTokens()
input_ids, mm_inputs = make_case(num_tokens, num_items, args.span_tokens)
original = input_ids.tobytes()
def baseline_padding():
return array("q", baseline.pad_input_tokens(input_ids, mm_inputs))
def optimized_padding():
return duplicate_pad_mm_input_ids(
input_ids, mm_inputs, pattern.pad_input_tokens
)
expected = baseline_padding()
actual = optimized_padding()
assert actual == expected, "Token values differ"
assert input_ids.tobytes() == original, "Original input was mutated"
assert actual is not input_ids, "Scheduler result aliases the original input"
functions = {"baseline": baseline_padding, "optimized": optimized_padding}
timings = {name: [] for name in functions}
for _ in range(args.warmups):
for function in functions.values():
function()
gc_was_enabled = gc.isenabled()
gc.disable()
try:
for trial in range(args.trials):
order = (
("baseline", "optimized") if trial % 2 else ("optimized", "baseline")
)
for name in order:
start = time.perf_counter_ns()
result = functions[name]()
elapsed = time.perf_counter_ns() - start
timings[name].append(elapsed / 1e6)
del result
finally:
if gc_was_enabled:
gc.enable()
assert input_ids.tobytes() == original
medians = {name: statistics.median(samples) for name, samples in timings.items()}
return {
"tokens": num_tokens,
"items": num_items,
"baseline_ms": medians["baseline"],
"optimized_ms": medians["optimized"],
"speedup": medians["baseline"] / medians["optimized"],
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--baseline-ref", required=True, help="Local pre-change git ref"
)
parser.add_argument(
"--tokens", nargs="+", type=int, default=[128, 4096, 16384, 131072]
)
parser.add_argument("--items", nargs="+", type=int, default=[0, 1, 10])
parser.add_argument("--span-tokens", type=int, default=32)
parser.add_argument("--trials", type=int, default=41)
parser.add_argument("--warmups", type=int, default=5)
parser.add_argument("--threads", type=int, default=1)
parser.add_argument("--json", action="store_true")
args = parser.parse_args()
if (
min(args.tokens + args.items) < 0
or min(args.span_tokens, args.trials, args.threads) < 1
or args.warmups < 0
):
parser.error("Lengths/items/warmups must be nonnegative; other counts positive")
if any(n < 3 * items for n in args.tokens for items in args.items):
parser.error("Each token length must accommodate at least 3 tokens per item")
torch.set_num_threads(args.threads)
baseline_commit, baseline = load_baseline(args.baseline_ref)
rows = [
benchmark_case(tokens, items, args, baseline)
for tokens in args.tokens
for items in args.items
]
if args.json:
print(
json.dumps(
{
"baseline_commit": baseline_commit,
"python": platform.python_version(),
"torch": torch.__version__,
"architecture": platform.machine(),
"threads": args.threads,
"trials": args.trials,
"warmups": args.warmups,
"span_tokens": args.span_tokens,
"results": rows,
},
indent=2,
)
)
return
print("Offset-based CPU padding; median milliseconds; synthetic inputs")
print(
f"{'Tokens':>8} {'Items':>6} {'Baseline (ms)':>14} {'Optimized (ms)':>14} {'Speedup':>9}"
)
for row in rows:
print(
f"{row['tokens']:>8} {row['items']:>6} "
f"{row['baseline_ms']:>14.4f} {row['optimized_ms']:>14.4f} "
f"{row['speedup']:>8.2f}x"
)
if __name__ == "__main__":
main()
@@ -1,284 +0,0 @@
"""Benchmark multi-node MSCCL++ AllGather against NCCL."""
import argparse
import os
import torch
import torch.distributed as dist
from sglang.srt.distributed import init_distributed_environment
from sglang.srt.distributed.parallel_state import (
cleanup_dist_env_and_memory,
get_tensor_model_parallel_group,
initialize_model_parallel,
set_mscclpp,
)
from sglang.test.test_utils import publish_build_topology
def _max_across_ranks(value: float, group) -> float:
tensor = torch.tensor([value], dtype=torch.float64)
dist.all_reduce(tensor, op=dist.ReduceOp.MAX, group=group)
return tensor.item()
def _time_eager(func, group, warmup: int = 2, repeat: int = 10) -> float:
for _ in range(warmup):
func()
torch.cuda.synchronize()
dist.barrier(group=group)
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for _ in range(repeat):
func()
end.record()
end.synchronize()
return _max_across_ranks(start.elapsed_time(end) * 1000 / repeat, group)
def _time_cuda_graph(
func,
group,
warmup: int = 100,
graph_loop: int = 10,
repeat: int = 100,
) -> float:
current_stream = torch.cuda.current_stream()
capture_stream = torch.cuda.Stream()
capture_stream.wait_stream(current_stream)
with torch.cuda.stream(capture_stream):
func(capture_stream)
capture_stream.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph, stream=capture_stream):
for _ in range(graph_loop):
func(capture_stream)
with torch.cuda.stream(capture_stream):
for _ in range(warmup):
graph.replay()
capture_stream.synchronize()
dist.barrier(group=group)
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record(capture_stream)
with torch.cuda.stream(capture_stream):
for _ in range(repeat):
graph.replay()
end.record(capture_stream)
end.synchronize()
torch.cuda.current_stream().wait_stream(capture_stream)
return _max_across_ranks(
start.elapsed_time(end) * 1000 / (repeat * graph_loop),
group,
)
def _validate_cuda_graph_updates(
func,
reference_func,
input_tensor: torch.Tensor,
actual: torch.Tensor,
expected: torch.Tensor,
group,
) -> None:
current_stream = torch.cuda.current_stream()
capture_stream = torch.cuda.Stream()
capture_stream.wait_stream(current_stream)
with torch.cuda.stream(capture_stream):
func(capture_stream)
capture_stream.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph, stream=capture_stream):
func(capture_stream)
input_tensor.neg_()
reference_func()
current_stream.synchronize()
capture_stream.wait_stream(current_stream)
with torch.cuda.stream(capture_stream):
graph.replay()
capture_stream.synchronize()
if not torch.equal(actual, expected):
raise AssertionError("MSCCL++ CUDA graph replay used stale AllGather data")
dist.barrier(group=group)
def _message_sizes(min_bytes: int, max_bytes: int):
size = 1 << (min_bytes - 1).bit_length()
while size <= max_bytes:
yield size
size <<= 1
def human_readable_size(size, decimal_places=1):
for unit in ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]:
if size < 1024.0 or unit == "PiB":
break
size /= 1024.0
return f"{size:.{decimal_places}f} {unit}"
try:
from tabulate import tabulate
except ImportError:
print("tabulate not installed, skipping table printing")
tabulate = None
def print_markdown_table(data):
if tabulate is not None:
print(tabulate(data, headers="keys", tablefmt="github"))
return
headers = data[0].keys()
header_row = "| " + " | ".join(headers) + " |"
separator = "| " + " | ".join(["---"] * len(headers)) + " |"
rows = []
for item in data:
row = "| " + " | ".join(str(item[key]) for key in headers) + " |"
rows.append(row)
markdown_table = "\n".join([header_row, separator] + rows)
print(markdown_table)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--min-bytes", type=int, default=1 << 10)
parser.add_argument("--max-bytes", type=int, default=8 << 20)
args = parser.parse_args()
if not (1 << 10) <= args.min_bytes <= args.max_bytes <= (8 << 20):
parser.error("Require 1 KiB <= min-bytes <= max-bytes <= 8 MiB")
local_rank = int(os.environ.get("LOCAL_RANK", 0))
torch.cuda.set_device(local_rank)
if not dist.is_initialized():
dist.init_process_group(
backend="nccl",
device_id=torch.device("cuda", local_rank),
)
world_size = dist.get_world_size()
rank = dist.get_rank()
device = torch.cuda.current_device()
set_mscclpp(True)
init_distributed_environment(
world_size=world_size,
rank=rank,
local_rank=local_rank,
)
publish_build_topology(world_rank=rank, tp_size=world_size)
initialize_model_parallel()
tp_group = get_tensor_model_parallel_group()
gpu_group = tp_group.device_group
cpu_group = tp_group.cpu_group
pynccl_comm = tp_group.pynccl_comm
mscclpp_comm = tp_group.pymscclpp_comm
assert pynccl_comm is not None
assert mscclpp_comm is not None
if args.min_bytes % world_size != 0 or args.max_bytes % world_size != 0:
parser.error("Total message sizes must be divisible by WORLD_SIZE")
if not mscclpp_comm._best_configs["allgather"]:
raise RuntimeError("No tuned MSCCL++ AllGather configuration is available")
result = []
for total_bytes in _message_sizes(args.min_bytes, args.max_bytes):
input_bytes = total_bytes // world_size
if input_bytes % torch.bfloat16.itemsize != 0:
continue
elements = input_bytes // torch.bfloat16.itemsize
input_tensor = (
torch.arange(
elements,
dtype=torch.long,
device=device,
)
% 193
+ rank
).to(torch.bfloat16)
nccl_output = torch.empty(
elements * world_size,
dtype=input_tensor.dtype,
device=device,
)
mscclpp_output = torch.empty_like(nccl_output)
def torch_nccl():
dist.all_gather_into_tensor(
nccl_output,
input_tensor,
group=gpu_group,
)
def py_nccl():
pynccl_comm.all_gather(nccl_output, input_tensor)
def mscclpp():
mscclpp_comm.all_gather(mscclpp_output, input_tensor)
def mscclpp_on_stream(stream):
mscclpp_comm.all_gather(
mscclpp_output,
input_tensor,
stream=stream,
)
def py_nccl_on_stream(_stream):
with pynccl_comm.change_state(enable=True):
pynccl_comm.all_gather(nccl_output, input_tensor)
torch_nccl()
mscclpp()
py_nccl()
torch.cuda.synchronize()
if not torch.equal(mscclpp_output, nccl_output):
raise AssertionError(
f"MSCCL++ AllGather mismatch at {input_bytes} input bytes"
)
_validate_cuda_graph_updates(
mscclpp_on_stream,
torch_nccl,
input_tensor,
mscclpp_output,
nccl_output,
cpu_group,
)
nccl_eager_us = _time_eager(torch_nccl, cpu_group)
mscclpp_eager_us = _time_eager(mscclpp, cpu_group)
mscclpp_graph_us = _time_cuda_graph(mscclpp_on_stream, cpu_group)
pynccl_graph_us = _time_cuda_graph(py_nccl_on_stream, cpu_group)
result.append(
{
"msg_size": human_readable_size(total_bytes),
"torch eager time": nccl_eager_us,
"msccl eager time": mscclpp_eager_us,
"msccl graph time": mscclpp_graph_us,
"pynccl graph time": pynccl_graph_us,
}
)
if rank == 0:
print(
f"sz={total_bytes // input_tensor.element_size()}, "
f"dtype={input_tensor.dtype}: correctness check PASS!"
)
torch.cuda.empty_cache()
if rank == 0:
print_markdown_table(result)
mscclpp_comm.destroy()
dist.barrier()
cleanup_dist_env_and_memory()
if __name__ == "__main__":
main()
@@ -12,9 +12,9 @@ torchrun --nproc_per_node gpu \
--master_port $MASTER_PORT benchmark/kernels/all_reduce/benchmark_mscclpp.py
"""
import argparse
import os
from contextlib import nullcontext
from typing import List
import torch
import torch.distributed as dist
@@ -23,16 +23,13 @@ from torch.distributed import ProcessGroup
from sglang.srt.distributed import init_distributed_environment
from sglang.srt.distributed.device_communicators.pymscclpp import PyMscclppCommunicator
from sglang.srt.distributed.device_communicators.pynccl import PyNcclCommunicator
from sglang.srt.distributed.device_communicators.pynccl_allocator import (
use_symmetric_memory,
)
from sglang.srt.distributed.parallel_state import (
cleanup_dist_env_and_memory,
get_tensor_model_parallel_group,
graph_capture,
initialize_model_parallel,
set_mscclpp,
set_mscclpp_all_reduce,
)
from sglang.srt.runtime_context import get_parallel
from sglang.test.test_utils import publish_build_topology
@@ -54,21 +51,14 @@ def pynccl_allreduce(
return msccl_input
def _bench_graph_time(
func,
inp_randn,
tp_group,
warmup_loop=2,
graph_loop=10,
test_loop=10,
):
with use_symmetric_memory(tp_group):
graph_input = inp_randn.clone()
def _bench_graph_time(func, inp_randn, warmup_loop=2, graph_loop=10, test_loop=10):
graph_input = inp_randn.clone()
graph_input_snapshot = inp_randn.clone()
with graph_capture() as graph_capture_context:
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph, stream=graph_capture_context.stream):
for _ in range(graph_loop):
graph_input.copy_(inp_randn)
graph_input.copy_(graph_input_snapshot)
graph_out = func(graph_input)
graph.replay()
@@ -81,7 +71,7 @@ def _bench_graph_time(
start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
latencies = []
latencies: List[float] = []
for _ in range(test_loop):
torch.cuda.synchronize()
dist.barrier()
@@ -95,9 +85,8 @@ def _bench_graph_time(
return func_output, func_cost_us
def _bench_eager_time(func, inp_randn, tp_group, warmup_loop=2, test_loop=10):
with use_symmetric_memory(tp_group):
eager_input = inp_randn.clone()
def _bench_eager_time(func, inp_randn, warmup_loop=2, test_loop=10):
eager_input = inp_randn.clone()
eager_output = func(eager_input)
func_output = eager_output.clone()
@@ -167,77 +156,57 @@ def print_markdown_table(data):
if __name__ == "__main__":
import logging
parser = argparse.ArgumentParser()
parser.add_argument(
"--enable-symm-mem",
action="store_true",
help="Allocate collective inputs from NCCL symmetric memory.",
)
args = parser.parse_args()
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
force=True,
)
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
device = torch.cuda.current_device()
if not dist.is_initialized():
dist.init_process_group(
backend="nccl", device_id=torch.device("cuda", local_rank)
)
dist.init_process_group(backend="nccl")
world, world_size = dist.group.WORLD, dist.get_world_size()
rank = dist.get_rank()
set_mscclpp(True)
torch.cuda.set_device(rank % 8)
device = torch.cuda.current_device()
set_mscclpp_all_reduce(True)
init_distributed_environment(
world_size=world_size,
rank=rank,
local_rank=local_rank,
local_rank=rank % 8,
)
publish_build_topology(
world_rank=rank,
tp_size=world_size,
enable_symm_mem=args.enable_symm_mem,
)
initialize_model_parallel(enable_symm_mem=args.enable_symm_mem)
tp_group = get_parallel().tp_group
group = tp_group.device_group
pynccl_comm = tp_group.pynccl_comm
pymscclpp_comm = tp_group.pymscclpp_comm
publish_build_topology(world_rank=rank, tp_size=world_size)
initialize_model_parallel()
group = get_tensor_model_parallel_group().device_group
cpu_group = get_tensor_model_parallel_group().cpu_group
pynccl_comm = get_tensor_model_parallel_group().pynccl_comm
pymscclpp_comm = get_tensor_model_parallel_group().pymscclpp_comm
dist.barrier()
profile = False
dtype = torch.bfloat16
ctx = get_torch_prof_ctx(profile)
result = []
max_message_bytes = 1 << 20
with ctx:
for i in range(10, 20):
sz = 2**i
if sz * dtype.itemsize > max_message_bytes:
if sz * dtype.itemsize > 2**20:
break
inp_randn = torch.randint(0, 2, (sz,), dtype=dtype, device=device)
inp_randn = torch.randint(1, 16, (sz,), dtype=dtype, device=device)
memory = torch.empty_like(inp_randn)
memory_out = torch.empty_like(memory)
torch_eager_output, torch_eager_time = _bench_eager_time(
lambda inp: torch_allreduce(inp, group), inp_randn, tp_group
lambda inp: torch_allreduce(inp, group), inp_randn
)
msccl_eager_output, msccl_eager_time = _bench_eager_time(
lambda inp: msccl_allreduce(inp, pymscclpp_comm),
inp_randn,
tp_group,
lambda inp: msccl_allreduce(inp, pymscclpp_comm), inp_randn
)
msccl_graph_output, msccl_graph_time = _bench_graph_time(
lambda inp: msccl_allreduce(inp, pymscclpp_comm),
inp_randn,
tp_group,
lambda inp: msccl_allreduce(inp, pymscclpp_comm), inp_randn
)
# since pynccl is inplace op, this return result is not correct if graph loop > 1
_, pynccl_graph_time = _bench_graph_time(
lambda inp: pynccl_allreduce(inp, pynccl_comm),
inp_randn,
tp_group,
lambda inp: pynccl_allreduce(inp, pynccl_comm), inp_randn
)
torch.testing.assert_close(torch_eager_output, msccl_graph_output)
torch.testing.assert_close(torch_eager_output, msccl_eager_output)
@@ -39,11 +39,11 @@ from sglang.srt.distributed.device_communicators.torch_symm_mem import (
TorchSymmMemCommunicator,
)
from sglang.srt.distributed.parallel_state import (
get_tensor_model_parallel_group,
graph_capture,
initialize_model_parallel,
set_torch_symm_mem_all_reduce,
)
from sglang.srt.runtime_context import get_parallel
from sglang.test.test_utils import publish_build_topology
from sglang.utils import is_in_ci
@@ -191,10 +191,10 @@ if __name__ == "__main__":
)
publish_build_topology(world_rank=rank, tp_size=world_size)
initialize_model_parallel()
group = get_parallel().tp_group.device_group
cpu_group = get_parallel().tp_group.cpu_group
pynccl_comm = get_parallel().tp_group.pynccl_comm
torch_symm_mem_comm = get_parallel().tp_group.torch_symm_mem_comm
group = get_tensor_model_parallel_group().device_group
cpu_group = get_tensor_model_parallel_group().cpu_group
pynccl_comm = get_tensor_model_parallel_group().pynccl_comm
torch_symm_mem_comm = get_tensor_model_parallel_group().torch_symm_mem_comm
dist.barrier()
profile = False
dtype = torch.bfloat16
@@ -1,154 +0,0 @@
# ViT attention with FA4 on SM103: row max from tcgen05.ld.red
# (SGLANG_FA4_TMEM_LOAD_RED_MAX=1) vs FMNMX (=0). Shapes and call follow
# VisionFlash4Attention (varlen, non-causal, no LSE). Each round CUDA-graph times both
# back to back, in alternating order; the per-round ratio is the headline number.
# Outputs must be bit-identical.
#
# Run on B300 / GB300 (elsewhere the flag is ignored):
# python benchmark/kernels/attention/bench_flash_attention_tmem_red_max.py
from __future__ import annotations
import argparse
import os
from typing import Iterable
import torch
from sglang.kernels.ops.attention.flash_attn.cute.interface import (
flash_attn_varlen_func,
)
RED_MAX_ENV = "SGLANG_FA4_TMEM_LOAD_RED_MAX"
# images, tokens per image, heads, head dim
SHAPES = [
(8, 4096, 16, 80), # Qwen2.5-VL
(8, 4096, 16, 72), # Qwen3-VL
(8, 4096, 16, 64), # InternViT, GLM / DeepSeek-V4 ViT
(64, 1024, 16, 80),
(64, 1024, 16, 64),
(1, 32768, 16, 80), # one large image or a video
(1, 32768, 16, 72),
(1, 32768, 16, 64),
]
def _bits(t: torch.Tensor) -> torch.Tensor:
int_dtype = {1: torch.uint8, 2: torch.int16, 4: torch.int32}[t.element_size()]
return t.contiguous().view(int_dtype)
def time_graph(fn, iters: int, calls_per_graph: int = 5) -> float:
"""Mean seconds per call, replaying a CUDA graph of `calls_per_graph` calls."""
for _ in range(3):
fn()
torch.cuda.synchronize()
graph = torch.cuda.CUDAGraph()
stream = torch.cuda.Stream()
stream.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(stream):
with torch.cuda.graph(graph, stream=stream):
for _ in range(calls_per_graph):
fn()
torch.cuda.current_stream().wait_stream(stream)
graph.replay()
torch.cuda.synchronize()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for _ in range(iters):
graph.replay()
end.record()
torch.cuda.synchronize()
return start.elapsed_time(end) / 1e3 / (iters * calls_per_graph)
def main(argv: Iterable[str] | None = None) -> int:
parser = argparse.ArgumentParser()
parser.add_argument(
"--rounds",
type=int,
default=10,
help="Rounds per shape; each round times both variants back to back.",
)
parser.add_argument(
"--iters", type=int, default=50, help="Graph replays per round."
)
parser.add_argument("--seed", type=int, default=0)
args = parser.parse_args(list(argv) if argv is not None else None)
if not torch.cuda.is_available():
raise RuntimeError("CUDA is required")
major, minor = torch.cuda.get_device_capability()
if major != 10:
raise RuntimeError(f"This benchmark is for SM10x. Got {major}.{minor}.")
if minor < 3:
print(
f"Note: SM{major}{minor} has no tcgen05.ld.red; {RED_MAX_ENV} is ignored "
"and both variants run the same kernel."
)
torch.manual_seed(args.seed)
device = "cuda"
mismatches = []
print(
f"{'shape':28s} {'FMNMX us':>9s} {'ld.red us':>9s} {'ld.red TF':>9s} "
f"{'best':>7s} {'paired':>7s} {'wins':>5s} bitwise"
)
for images, tokens, heads, head_dim in SHAPES:
torch.cuda.empty_cache()
q, k, v = [
torch.randn(
images * tokens, heads, head_dim, device=device, dtype=torch.bfloat16
)
for _ in range(3)
]
cu_seqlens = torch.arange(images + 1, dtype=torch.int32, device=device) * tokens
kwargs = dict(
q=q,
k=k,
v=v,
cu_seqlens_q=cu_seqlens,
cu_seqlens_k=cu_seqlens,
max_seqlen_q=tokens,
max_seqlen_k=tokens,
)
outputs, times = {}, {False: [], True: []}
for red_max in (False, True):
os.environ[RED_MAX_ENV] = "1" if red_max else "0"
outputs[red_max] = flash_attn_varlen_func(**kwargs)[0]
# Swap the order every round so clock and thermal drift hit both equally.
for r in range(args.rounds):
for red_max in (False, True) if r % 2 == 0 else (True, False):
os.environ[RED_MAX_ENV] = "1" if red_max else "0"
times[red_max].append(
time_graph(lambda: flash_attn_varlen_func(**kwargs), args.iters)
)
same = torch.equal(_bits(outputs[True]), _bits(outputs[False]))
name = f"{images} x {tokens} tok, h{heads} d{head_dim}"
if not same:
mismatches.append(name)
t0, t1 = min(times[False]), min(times[True])
# Ratio of the two variants within each round, then the median over rounds.
paired = sorted(a / c for a, c in zip(times[False], times[True]))
wins = f"{sum(p > 1 for p in paired)}/{len(paired)}"
flops = 4.0 * images * heads * head_dim * tokens * tokens
print(
f"{name:28s} {t0 * 1e6:9.1f} {t1 * 1e6:9.1f} {flops / t1 / 1e12:9.1f} "
f"{t0 / t1:6.3f}x {paired[len(paired) // 2]:6.3f}x {wins:>5s} "
f"{'identical' if same else 'DIFFERENT'}",
flush=True,
)
del q, k, v, outputs
os.environ.pop(RED_MAX_ENV, None)
if mismatches:
print(f"\nOutputs differ between the variants for: {', '.join(mismatches)}")
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -7,7 +7,7 @@ import torch.utils.benchmark as benchmark
from flashinfer import BatchDecodeWithPagedKVCacheWrapper
from sglang.kernels.ops.attention.decode_attention import decode_attention_fwd
from sglang.srt.layers.attention.flashinfer_backend import should_use_tensor_core
from sglang.kernels.ops.attention.flashinfer_backend import should_use_tensor_core
def benchmark_forward(
@@ -11,7 +11,7 @@ from sglang.benchmark.deepseek_utils import (
per_block_cast_to_fp8,
per_token_cast_to_fp8,
)
from sglang.kernels.ops.gemm.fp8_kernel import (
from sglang.kernels.ops.quantization.fp8_kernel import (
w8a8_block_fp8_matmul_deepgemm as w8a8_block_fp8_matmul,
)
@@ -9,8 +9,10 @@ from sglang.benchmark.deepseek_utils import (
get_weight_shapes,
per_block_cast_to_fp8,
)
from sglang.kernels.ops.gemm.fp8_kernel import w8a8_block_fp8_matmul_deepgemm
from sglang.kernels.ops.quantization.fp8_kernel import sglang_per_token_group_quant_fp8
from sglang.kernels.ops.quantization.fp8_kernel import (
sglang_per_token_group_quant_fp8,
w8a8_block_fp8_matmul_deepgemm,
)
from sglang.srt.layers.quantization.fp8_utils import requant_weight_ue8m0
BLOCK_SIZE = 128
@@ -1,146 +0,0 @@
"""Compare default, H200, and H20-3e block-FP8 configs on an H20-3e.
Example:
python benchmark/kernels/deepseek/benchmark_h20_block_fp8.py \
--output h20_block_fp8.jsonl
This measures individual GEMMs, not end-to-end model throughput. Run on an
otherwise idle GPU. Inputs are synthetic; no model weights are needed.
"""
import argparse
import json
import pathlib
import statistics
from unittest.mock import patch
import torch
import triton
from sglang.kernels.ops.gemm import fp8_kernel
SHAPES = [
(1792, 5120),
(25600, 6144),
(4096, 1280),
(5120, 1024),
(5120, 288),
(576, 5120),
]
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=pathlib.Path, required=True)
parser.add_argument(
"--batch-sizes",
type=int,
nargs="+",
default=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64],
)
parser.add_argument("--trials", type=int, default=5)
parser.add_argument("--rep-ms", type=int, default=50)
parser.add_argument("--seed", type=int, default=98765)
args = parser.parse_args()
assert torch.cuda.get_device_name() == "NVIDIA H20-3e"
assert args.trials > 0 and args.rep_ms > 0 and all(m > 0 for m in args.batch_sizes)
torch.backends.cuda.matmul.allow_tf32 = False
torch.manual_seed(args.seed)
config_dir = pathlib.Path(fp8_kernel.__file__).parent / "configs"
with args.output.open("x") as output:
def emit(row):
text = json.dumps(row)
output.write(text + "\n")
output.flush()
print(text, flush=True)
emit(
{
"kind": "environment",
"device": torch.cuda.get_device_name(),
"torch": torch.__version__,
"triton": triton.__version__,
"seed": args.seed,
"batch_sizes": args.batch_sizes,
}
)
for n, k in SHAPES:
b = (torch.randn(n, k, device="cuda") * 0.2).to(torch.float8_e4m3fn)
bs = torch.rand(n // 32, k // 32, device="cuda") + 0.1
b_dequant = b.float() * bs.repeat_interleave(32, 0).repeat_interleave(32, 1)
tables = {"default": None}
for device in ("NVIDIA_H200", "NVIDIA_H20-3e"):
path = (
config_dir
/ f"N={n},K={k},device_name={device},dtype=fp8_w8a8,block_shape=[32, 32].json"
)
if path.exists():
tables[device] = {
int(m): cfg for m, cfg in json.loads(path.read_text()).items()
}
assert "NVIDIA_H20-3e" in tables, (
"Install the H20-3e configuration files before benchmarking"
)
for m in args.batch_sizes:
a = (torch.randn(m, k, device="cuda") * 0.2).to(torch.float8_e4m3fn)
scales = (torch.rand(k // 32, m, device="cuda") + 0.1).T
reference = (
(a.float() * scales.repeat_interleave(32, 1)) @ b_dequant.T
).to(torch.bfloat16)
results = {}
# Compile and validate every variant before timing any variant.
for label, table in tables.items():
with patch.object(
fp8_kernel, "get_w8a8_block_fp8_configs", return_value=table
):
actual = fp8_kernel.w8a8_block_fp8_matmul_triton(
a, b, scales, bs, [32, 32], torch.bfloat16
)
relative_l2 = (
torch.linalg.vector_norm(actual.float() - reference.float())
/ torch.linalg.vector_norm(reference.float())
).item()
assert torch.isfinite(actual).all() and relative_l2 < 0.01
results[label] = {
"config": None
if table is None
else table[min(table, key=lambda key: abs(key - m))],
"relative_l2": relative_l2,
"trials_us": [],
}
for trial in range(args.trials):
labels = list(tables)
# Rotate variant order to reduce systematic timing-order bias.
labels = (
labels[trial % len(labels) :] + labels[: trial % len(labels)]
)
for label in labels:
with patch.object(
fp8_kernel,
"get_w8a8_block_fp8_configs",
return_value=tables[label],
):
def invoke():
return fp8_kernel.w8a8_block_fp8_matmul_triton(
a, b, scales, bs, [32, 32], torch.bfloat16
)
elapsed = (
triton.testing.do_bench_cudagraph(
invoke, rep=args.rep_ms
)
* 1000
)
results[label]["trials_us"].append(elapsed)
for result in results.values():
result["median_us"] = statistics.median(result["trials_us"])
emit(
{"kind": "measurement", "m": m, "n": n, "k": k, "results": results}
)
emit({"kind": "complete"})
if __name__ == "__main__":
main()
+1 -1
View File
@@ -78,7 +78,7 @@ python benchmark/kernels/quantization/tuning_block_wise_kernel.py --N 5120 --K 1
## Output
Generates JSON config files saved to `python/sglang/kernels/ops/gemm/configs/`:
Generates JSON config files saved to `python/sglang/srt/layers/quantization/configs/`:
```
N={N},K={K},device_name={DEVICE},dtype=fp8_w8a8,block_shape=[128,128].json
```
@@ -28,11 +28,11 @@ from tqdm import tqdm
mp.set_start_method("spawn", force=True)
from sglang.benchmark.deepseek_utils import get_weight_shapes
from sglang.kernels.ops.gemm.fp8_kernel import (
from sglang.kernels.ops.quantization.fp8_kernel import (
_w8a8_block_fp8_matmul,
_w8a8_block_fp8_matmul_unrolledx4,
)
from sglang.kernels.ops.gemm.int8_kernel import _w8a8_block_int8_matmul
from sglang.kernels.ops.quantization.int8_kernel import _w8a8_block_int8_matmul
from sglang.srt.utils import (
get_device,
get_device_core_count,
@@ -491,7 +491,7 @@ if __name__ == "__main__":
parser.add_argument(
"--save-path",
type=str,
default="python/sglang/kernels/ops/gemm/configs",
default="python/sglang/kernels/ops/quantization/configs",
)
args = parser.parse_args()
@@ -1,156 +0,0 @@
"""Tune the gfx950 native MXFP8 dense route for one (N, K) and write its rows into
python/sglang/kernels/ops/quantization/mxfp8_gemv_gfx95_configs.json.
Two tables: the scaled-MFMA GEMV config per M bucket (1 ... 32) and the dot_scaled tile per
M bucket (64 ... 16384). Each candidate is timed by HIP-graph replay rotating over weight
copies that together exceed the last-level cache, so the weight streams from HBM as it does
in decode. The GEMV pick minimizes the fp8-input plus the bf16-input time: fused producers
hand it fp8, other callers bf16.
python benchmark/kernels/quantization/tuning_mxfp8_native_gfx95.py --N 1792 --K 5120
"""
import argparse
import itertools
import json
import torch
from triton.runtime.errors import OutOfResources
from sglang.kernels.jit.utils import empty_sentinel
from sglang.kernels.ops.quantization import mxfp8_native_amd_gfx95 as native
from sglang.kernels.ops.quantization.mxfp8_amd_gfx95 import fp8_grid_quantize
GRAPH_CALLS = 20
ROTATION_BYTES = 512 << 20
def graph_time_us(calls, replays: int) -> float:
"""Mean time of one call, replaying a graph of GRAPH_CALLS calls cycling through calls."""
for call in calls:
call()
torch.cuda.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
for i in range(GRAPH_CALLS):
calls[i % len(calls)]()
graph.replay()
torch.cuda.synchronize()
start, end = (
torch.cuda.Event(enable_timing=True),
torch.cuda.Event(enable_timing=True),
)
start.record()
for _ in range(replays):
graph.replay()
end.record()
torch.cuda.synchronize()
return start.elapsed_time(end) * 1000 / (GRAPH_CALLS * replays)
def make_weights(n: int, k: int):
copies = max(2, min(16, -(-ROTATION_BYTES // (n * k))))
weights = []
for _ in range(copies):
weight = (torch.randn(n, k, device="cuda") * 0.05).to(torch.float8_e4m3fn)
scale = torch.exp2(
torch.randint(-3, 3, (n // 32, k // 32), device="cuda").float()
)
weights.append(native.prepare_mxfp8_native_weight(weight, scale, [32, 32]))
return weights
def tune_gemv(n: int, k: int, weights) -> dict:
rows = {}
for bucket in native._M_BUCKETS:
x = torch.randn(bucket, k, device="cuda", dtype=torch.bfloat16)
xq, xs = fp8_grid_quantize(x)
xq = xq.view(torch.uint8)
no_scale = empty_sentinel(x.device, torch.uint8)
out = torch.empty(bucket, n, dtype=torch.bfloat16, device="cuda")
times = {}
for waves, steps, tile_rows, tokens in itertools.product(
(4, 8, 16), (1, 2, 4), (16, 32), (16, 32)
):
config = native._GemvConfig(waves, steps, tile_rows, tokens)
if not config.valid_for(bucket, n):
continue
fp8_kernel = native._jit_mxfp8_gemv_module(config, False)
bf16_kernel = native._jit_mxfp8_gemv_module(config, True)
fp8_us = graph_time_us(
[lambda w=w: fp8_kernel.run(w[0], w[1], xq, xs, out) for w in weights],
replays=10,
)
bf16_us = graph_time_us(
[
lambda w=w: bf16_kernel.run(w[0], w[1], x, no_scale, out)
for w in weights
],
replays=10,
)
times[f"w{waves}s{steps}r{tile_rows}t{tokens}k"] = fp8_us + bf16_us
best = min(times, key=times.get)
rows[f"gfx950:{n}:{k}:{bucket}"] = best
print(f"gemv M={bucket:5d}: {best} {times[best]:.2f} us (fp8 + bf16)")
return rows
def tune_large_m(n: int, k: int, weights) -> dict:
k_pad = weights[0][0].shape[1] * 128
rows = {}
for bucket in native._LARGE_M_BUCKETS:
xq, xs = fp8_grid_quantize(
torch.randn(bucket, k, device="cuda", dtype=torch.bfloat16)
)
times = {}
for bm, bn, bk, warps, split_k in itertools.product(
(32, 64, 128, 256), (64, 128, 256), (128, 256), (4, 8), (1, 2, 4, 5, 8)
):
if k_pad % bk or (k_pad // bk) % split_k or bm > max(64, 2 * bucket):
continue
if split_k > 1 and bucket > 1024:
continue
tile = (bm, bn, bk, warps)
try:
times[f"{bm},{bn},{bk},{warps},{split_k}"] = graph_time_us(
[
lambda w=w: native._mxfp8_shuffled_gemm(
xq, xs, w[0], w[1], tile, split_k
)
for w in weights
],
replays=5,
)
except OutOfResources:
continue # the tile's LDS footprint exceeds the CU's
best = min(times, key=times.get)
rows[f"gfx950:{n}:{k}:{bucket}"] = best
print(f"large_m M={bucket:5d}: {best} {times[best]:.2f} us")
return rows
def main():
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--N", type=int, required=True)
parser.add_argument("--K", type=int, required=True)
args = parser.parse_args()
assert native.native_route_supports(args.N, args.K), (args.N, args.K)
assert torch.cuda.get_device_properties(0).gcnArchName.startswith("gfx950")
torch.manual_seed(0)
weights = make_weights(args.N, args.K)
gemv_rows = tune_gemv(args.N, args.K, weights)
large_m_rows = tune_large_m(args.N, args.K, weights)
with open(native._CONFIG_FILE) as f:
table = json.load(f)
for section, rows in (("configs", gemv_rows), ("large_m", large_m_rows)):
table[section] = dict(sorted({**table[section], **rows}.items()))
with open(native._CONFIG_FILE, "w") as f:
json.dump(table, f, indent=1)
f.write("\n")
print(f"wrote {len(gemv_rows) + len(large_m_rows)} rows to {native._CONFIG_FILE}")
if __name__ == "__main__":
main()
+20
View File
@@ -0,0 +1,20 @@
## Download data
```
bash download_data.sh
```
## Run benchmark
### Benchmark sglang
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
```
```
python3 bench_sglang.py --nsub 10
```
```
# OpenAI models
python3 bench_sglang.py --backend gpt-3.5-turbo --parallel 8
```
+151
View File
@@ -0,0 +1,151 @@
"""
Usage:
python3 bench_hf.py --model-path meta-llama/Llama-2-7b-hf --data-dir data --ntrain 5
"""
import argparse
import json
import os
import time
import numpy as np
import pandas as pd
import torch
from tqdm import tqdm
from transformers import AutoModelForCausalLM, AutoTokenizer
choices = ["A", "B", "C", "D"]
def format_subject(subject):
l = subject.split("_")
s = ""
for entry in l:
s += " " + entry
return s
def format_example(df, idx, include_answer=True):
prompt = df.iloc[idx, 0]
k = df.shape[1] - 2
for j in range(k):
prompt += "\n{}. {}".format(choices[j], df.iloc[idx, j + 1])
prompt += "\nAnswer:"
if include_answer:
prompt += " {}\n\n".format(df.iloc[idx, k + 1])
return prompt
def gen_prompt(train_df, subject, k=-1):
prompt = "The following are multiple choice questions (with answers) about{}.\n\n".format(
format_subject(subject)
)
if k == -1:
k = train_df.shape[0]
for i in range(k):
prompt += format_example(train_df, i)
return prompt
@torch.no_grad()
def main(args):
print(f"Loading model: {args.model_path}")
tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
args.model_path,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto",
).eval()
subjects = sorted(
[
f.split("_test.csv")[0]
for f in os.listdir(os.path.join(args.data_dir, "test"))
if "_test.csv" in f
]
)
all_cors = []
num_requests = 0
total_latency = 0
for subject in tqdm(subjects[: args.nsub]):
dev_df = pd.read_csv(
os.path.join(args.data_dir, "dev", subject + "_dev.csv"), header=None
)[: args.ntrain]
test_df = pd.read_csv(
os.path.join(args.data_dir, "test", subject + "_test.csv"), header=None
)
k = args.ntrain
few_shot_examples = gen_prompt(dev_df, subject, k)
while len(tokenizer.encode(few_shot_examples)) > 1536:
k -= 1
if k < 0:
break
few_shot_examples = gen_prompt(dev_df, subject, k)
preds = []
labels = []
tic = time.perf_counter()
for i in range(test_df.shape[0]):
prompt_end = format_example(test_df, i, include_answer=False)
prompt = few_shot_examples + prompt_end
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(
input_ids,
max_new_tokens=1,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
output_str = tokenizer.decode(
output_ids[0][input_ids.shape[-1] :], skip_special_tokens=True
)
preds.append(output_str.strip()[0] if len(output_str.strip()) > 0 else "")
labels.append(test_df.iloc[i, test_df.shape[1] - 1])
latency = time.perf_counter() - tic
total_latency += latency
cors = [pred == label for pred, label in zip(preds, labels)]
all_cors.append(cors)
num_requests += len(test_df)
print(
f"Subject: {subject}, Accuracy: {np.mean(cors):.3f}, Latency: {latency:.3f}s"
)
weighted_acc = np.mean(np.concatenate(all_cors))
print(f"Total Latency: {total_latency:.3f}s")
print(f"Average Accuracy: {weighted_acc:.3f}")
if args.output:
with open(args.output, "a") as fout:
value = {
"task": "mmlu",
"backend": "hf",
"model": args.model_path,
"latency": round(total_latency, 3),
"accuracy": round(weighted_acc, 3),
"num_requests": num_requests,
"other": {
"nsub": args.nsub,
"ntrain": args.ntrain,
},
}
fout.write(json.dumps(value) + "\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--model-path", type=str, required=True)
parser.add_argument("--ntrain", type=int, default=5)
parser.add_argument("--data-dir", type=str, default="data")
parser.add_argument("--nsub", type=int, default=60)
parser.add_argument("--output", type=str, help="Output file path")
args = parser.parse_args()
main(args)
+210
View File
@@ -0,0 +1,210 @@
import argparse
import json
import os
import subprocess
import tarfile
import time
import numpy as np
import pandas as pd
import tiktoken
from sglang.test.test_utils import (
add_common_sglang_args_and_parse,
dump_bench_raw_result,
select_sglang_backend,
)
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
choices = ["A", "B", "C", "D"]
tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo")
def format_subject(subject):
l = subject.split("_")
s = ""
for entry in l:
s += " " + entry
return s
def format_example(df, idx, include_answer=True):
prompt = df.iloc[idx, 0]
k = df.shape[1] - 2
for j in range(k):
prompt += "\n{}. {}".format(choices[j], df.iloc[idx, j + 1])
prompt += "\nAnswer:"
if include_answer:
prompt += " {}\n\n".format(df.iloc[idx, k + 1])
return prompt
def gen_prompt(train_df, subject, k=-1):
prompt = "The following are multiple choice questions (with answers) about{}.\n\n".format(
format_subject(subject)
)
if k == -1:
k = train_df.shape[0]
for i in range(k):
prompt += format_example(train_df, i)
return prompt
def download_data(data_dir):
"""Download and extract MMLU data if it doesn't exist."""
if os.path.isdir(os.path.join(data_dir, "test")):
return
print(f"Data not found at {data_dir}. Downloading...")
os.makedirs(data_dir, exist_ok=True)
tar_path = os.path.join(data_dir, "data.tar")
subprocess.check_call(
["wget", "-O", tar_path, "https://people.eecs.berkeley.edu/~hendrycks/data.tar"]
)
with tarfile.open(tar_path) as tar:
tar.extractall(path=data_dir, filter="data")
# The tarball extracts into a "data/" subdirectory; move contents up if needed
nested = os.path.join(data_dir, "data")
if os.path.isdir(nested):
for item in os.listdir(nested):
os.rename(os.path.join(nested, item), os.path.join(data_dir, item))
os.rmdir(nested)
os.remove(tar_path)
print("Download complete.")
def main(args):
subjects = sorted(
[
f.split("_test.csv")[0]
for f in os.listdir(os.path.join(args.data_dir, "test"))
if "_test.csv" in f
]
)
# Build prompts
arguments = []
labels = []
num_questions = []
for subject in subjects[: args.nsub]:
dev_df = pd.read_csv(
os.path.join(args.data_dir, "dev", subject + "_dev.csv"), header=None
)[: args.ntrain]
test_df = pd.read_csv(
os.path.join(args.data_dir, "test", subject + "_test.csv"), header=None
)
num_questions.append(test_df.shape[0])
k = args.ntrain
few_shot_examples = gen_prompt(dev_df, subject, k)
while len(tokenizer.encode(few_shot_examples)) > 1536:
k -= 1
few_shot_examples = gen_prompt(dev_df, subject, k)
for i in range(test_df.shape[0]):
prompt_end = format_example(test_df, i, include_answer=False)
arguments.append(
{
"examples": few_shot_examples,
"question": prompt_end,
}
)
label = test_df.iloc[i, test_df.shape[1] - 1]
labels.append(label)
#####################################
######### SGL Program Begin #########
#####################################
import sglang as sgl
if args.backend.startswith("gpt-"):
@sgl.function
def few_shot_mmlu(s, examples, question):
s += sgl.user(examples + question)
s += sgl.assistant(sgl.gen("answer"))
else:
@sgl.function
def few_shot_mmlu(s, examples, question):
s += examples + question + sgl.gen("answer")
#####################################
########## SGL Program End ##########
#####################################
# Select backend
backend = select_sglang_backend(args)
# Run
tic = time.perf_counter()
states = few_shot_mmlu.run_batch(
arguments,
temperature=0,
max_new_tokens=1,
backend=backend,
num_threads=args.parallel,
progress_bar=True,
)
preds = [
s["answer"].strip()[0] if len(s["answer"].strip()) > 0 else "" for s in states
]
latency = time.perf_counter() - tic
# Compute accuracy
cors = [pred == label for pred, label in zip(preds, labels)]
pt = 0
for subject, num_qs in zip(subjects[: args.nsub], num_questions):
print(
f"subject: {subject}, #q:{num_qs}, acc: {np.mean(cors[pt : pt + num_qs]):.3f}"
)
pt += num_qs
assert pt == len(cors)
weighted_acc = np.mean(cors)
dump_bench_raw_result(
path=args.raw_result_file,
states=states,
preds=preds,
labels=labels,
)
# Print results
print("Total latency: {:.3f}".format(latency))
print("Average accuracy: {:.3f}".format(weighted_acc))
# Write results
with open(args.result_file, "a") as fout:
value = {
"task": "mmlu",
"backend": args.backend,
"num_gpus": 1,
"latency": round(latency, 3),
"accuracy": round(weighted_acc, 3),
"num_requests": len(arguments),
"other": {
"nsub": args.nsub,
"parallel": args.parallel,
},
}
fout.write(json.dumps(value) + "\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--ntrain", "-k", type=int, default=5)
parser.add_argument(
"--data_dir", "-d", type=str, default=os.path.join(SCRIPT_DIR, "data")
)
parser.add_argument("--save_dir", "-s", type=str, default="results")
parser.add_argument("--nsub", type=int, default=60)
args = add_common_sglang_args_and_parse(parser)
download_data(args.data_dir)
main(args)
+2
View File
@@ -0,0 +1,2 @@
wget https://people.eecs.berkeley.edu/~hendrycks/data.tar
tar xf data.tar
@@ -1,271 +0,0 @@
"""Engine benchmark for setwise (multi-position) scoring.
Benchmarks `Engine.async_score(..., score_extraction_token_id=...)` — the setwise
path that pools the classification head at each candidate's anchor token and
returns an ``[N, num_labels]`` matrix in a single forward pass.
Unlike the general ``bench_score.py`` (label-token / MIS scoring), this is a
focused, self-contained harness: it sweeps the **set size** (number of
candidates packed into one item) and reports latency percentiles and throughput
(requests/s and candidates/s) so you can see how a single-forward setwise pass
scales with N.
It talks to the Engine directly (no HTTP) and requires the setwise deployment
constraints (``--disable-radix-cache``, ``--chunked-prefill-size -1``), which it
sets automatically.
Usage:
# Default sweep over set sizes on the target model
python bench_setwise_score.py --model-path /path/to/seqcls_model
# Custom sweep + concurrency + per-candidate length
python bench_setwise_score.py --model-path /path/to/model \
--set-sizes 8 16 32 64 --concurrency 16 --num-requests 100 \
--item-tokens 180 --query-tokens 64
# Include pooled hidden states in the response (per-anchor pre-head vectors)
python bench_setwise_score.py --model-path /path/to/model --return-pooled-hidden-states
"""
import argparse
import asyncio
import csv
import random
import statistics
import time
from typing import List, Optional
from transformers import AutoTokenizer
def parse_args():
parser = argparse.ArgumentParser(
description="Benchmark setwise scoring via Engine.async_score",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--model-path",
type=str,
required=True,
help="Path to a SequenceClassification model (e.g. Qwen3ForSequenceClassification).",
)
parser.add_argument(
"--score-extraction-token",
type=str,
default="<|object_ref_start|>",
help="Anchor token pooled at each candidate (default: <|object_ref_start|>).",
)
parser.add_argument(
"--set-sizes",
type=int,
nargs="+",
default=[8, 16, 32, 64],
help="Candidates per set (each is one anchor in one item). Swept in order.",
)
parser.add_argument(
"--item-tokens",
type=int,
default=180,
help="Content tokens per candidate before its anchor (default: 180).",
)
parser.add_argument(
"--query-tokens",
type=int,
default=0,
help="Shared query-prefix tokens (default: 0).",
)
parser.add_argument(
"--num-requests",
type=int,
default=50,
help="Timed requests per set size (default: 50).",
)
parser.add_argument(
"--concurrency",
type=int,
default=8,
help="Max in-flight requests (default: 8).",
)
parser.add_argument(
"--warmup",
type=int,
default=3,
help="Warmup requests per set size, not timed (default: 3).",
)
parser.add_argument("--apply-softmax", action="store_true")
parser.add_argument("--return-pooled-hidden-states", action="store_true")
# Engine configuration
parser.add_argument("--dtype", type=str, default="float16")
parser.add_argument("--mem-fraction-static", type=float, default=0.5)
parser.add_argument("--max-prefill-tokens", type=int, default=60000)
parser.add_argument("--attention-backend", type=str, default="flashinfer")
parser.add_argument("--seed", type=int, default=0)
parser.add_argument(
"--output", "-o", type=str, default=None, help="Optional CSV output path."
)
return parser.parse_args()
def _random_content(num_tokens: int, anchor_id: int, lo: int = 10, hi: int = 30000):
"""Random content token ids that never collide with the anchor id."""
ids = [random.randint(lo, hi) for _ in range(num_tokens)]
return [t + 1 if t == anchor_id else t for t in ids]
def _build_setwise_request(args, anchor_id: int, n_candidates: int):
"""One setwise request: shared query prefix + ONE item with N anchors."""
query_ids = (
_random_content(args.query_tokens, anchor_id) if args.query_tokens else []
)
item_ids: List[int] = []
for _ in range(n_candidates):
item_ids.extend(_random_content(args.item_tokens, anchor_id))
item_ids.append(anchor_id)
return query_ids, [item_ids]
def _percentile(sorted_vals: List[float], pct: float) -> float:
if not sorted_vals:
return 0.0
k = max(0, min(len(sorted_vals) - 1, round((pct / 100.0) * (len(sorted_vals) - 1))))
return sorted_vals[k]
async def _run_set_size(engine, args, anchor_id: int, n: int) -> Optional[dict]:
# Pre-build requests so generation cost is outside the timed section.
requests = [
_build_setwise_request(args, anchor_id, n)
for _ in range(args.num_requests + args.warmup)
]
prompt_tokens = len(requests[0][0]) + len(requests[0][1][0])
async def _one(query_ids, items):
t0 = time.perf_counter()
res = await engine.async_score(
query=query_ids,
items=items,
score_extraction_token_id=anchor_id,
apply_softmax=args.apply_softmax,
return_pooled_hidden_states=args.return_pooled_hidden_states,
)
# Setwise scores are nested (one [Ni x num_labels] matrix per item) and
# this harness sends exactly one item, so the per-candidate row count is
# len(res.scores[0]), not len(res.scores) (which is always 1 = num items).
return time.perf_counter() - t0, len(res.scores[0]) if res.scores else 0
# Warmup (also validates the row count == N).
for q, it in requests[: args.warmup]:
_, rows = await _one(q, it)
if rows != n:
print(f"[warn] set_size={n}: expected {n} score rows, got {rows}")
sem = asyncio.Semaphore(args.concurrency)
async def _guarded(q, it):
async with sem:
return await _one(q, it)
timed = requests[args.warmup :]
wall_start = time.perf_counter()
results = await asyncio.gather(*(_guarded(q, it) for q, it in timed))
wall = time.perf_counter() - wall_start
latencies_ms = sorted(1000.0 * lat for lat, _ in results)
n_req = len(results)
return {
"set_size": n,
"prompt_tokens": prompt_tokens,
"requests": n_req,
"concurrency": args.concurrency,
"avg_ms": statistics.mean(latencies_ms),
"p50_ms": _percentile(latencies_ms, 50),
"p95_ms": _percentile(latencies_ms, 95),
"p99_ms": _percentile(latencies_ms, 99),
"req_per_s": n_req / wall if wall > 0 else 0.0,
"cand_per_s": (n_req * n) / wall if wall > 0 else 0.0,
}
def _print_table(rows: List[dict]):
headers = [
"set_size",
"prompt_tokens",
"requests",
"avg_ms",
"p50_ms",
"p95_ms",
"p99_ms",
"req_per_s",
"cand_per_s",
]
print("\n" + " | ".join(f"{h:>13}" for h in headers))
print("-" * (16 * len(headers)))
for r in rows:
cells = []
for h in headers:
v = r[h]
cells.append(f"{v:>13.2f}" if isinstance(v, float) else f"{v:>13}")
print(" | ".join(cells))
async def main():
args = parse_args()
random.seed(args.seed)
tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True)
anchor_id = tokenizer.convert_tokens_to_ids(args.score_extraction_token)
unk_id = tokenizer.unk_token_id
if anchor_id is None or (unk_id is not None and anchor_id == unk_id):
raise ValueError(
f"score_extraction_token {args.score_extraction_token!r} did not resolve "
f"to a dedicated token id."
)
from sglang.srt.entrypoints.engine import Engine
# Setwise deployment constraints: positions are full-prompt coordinates, so
# radix reuse / chunked prefill must be off (the engine also validates this).
engine = Engine(
model_path=args.model_path,
dtype=args.dtype,
mem_fraction_static=args.mem_fraction_static,
max_prefill_tokens=args.max_prefill_tokens,
attention_backend=args.attention_backend,
disable_radix_cache=True,
chunked_prefill_size=-1,
)
print(
f"Setwise scoring benchmark | model={args.model_path}\n"
f"anchor={args.score_extraction_token!r} (id={anchor_id}) | "
f"item_tokens={args.item_tokens} query_tokens={args.query_tokens} | "
f"concurrency={args.concurrency} num_requests={args.num_requests}"
)
rows = []
try:
for n in args.set_sizes:
row = await _run_set_size(engine, args, anchor_id, n)
if row is not None:
rows.append(row)
print(
f"set_size={n:>4}: avg={row['avg_ms']:.1f}ms "
f"p95={row['p95_ms']:.1f}ms "
f"{row['req_per_s']:.1f} req/s {row['cand_per_s']:.0f} cand/s"
)
finally:
engine.shutdown()
_print_table(rows)
if args.output and rows:
with open(args.output, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
writer.writeheader()
writer.writerows(rows)
print(f"\nWrote {len(rows)} rows to {args.output}")
if __name__ == "__main__":
asyncio.run(main())
-181
View File
@@ -1,181 +0,0 @@
"""Finite cold-cache decode benchmark using synthetic token IDs.
Run against an otherwise idle SGLang server. This clears its prefix cache.
Example:
python benchmark/serving/benchmark_h20_decode.py \
--model-path /path/to/DeepSeek-V4.1-Flash --output before.jsonl
Repeat after installing the H20-3e configurations with identical server flags.
"""
import argparse
import concurrent.futures
import datetime
import hashlib
import json
import pathlib
import statistics
import time
import urllib.request
from tokenizers import Tokenizer
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--label", default="measurement")
p.add_argument("--url", default="http://127.0.0.1:8000")
p.add_argument("--model-path", type=pathlib.Path, required=True)
p.add_argument("--output", type=pathlib.Path, required=True)
args = p.parse_args()
out = args.output
assert not out.exists()
tokenizer = Tokenizer.from_file(str(args.model_path / "tokenizer.json"))
filler = tokenizer.encode(
(
"The research server processes independent records. Each record contains an identifier, a timestamp, and a measured value. We compare latency using the same data and the same output budget.\n"
)
* 16000,
add_special_tokens=False,
).ids
def emit(row):
row = {
"time": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"label": args.label,
**row,
}
with out.open("a") as f:
f.write(json.dumps(row) + "\n")
print(json.dumps(row), flush=True)
def request(path, body=None):
return urllib.request.urlopen(
urllib.request.Request(
args.url.rstrip("/") + path,
data=None if body is None else json.dumps(body).encode(),
headers={"Content-Type": "application/json"},
),
timeout=None,
)
def flush():
with request("/flush_cache?timeout=30", {}) as r:
r.read()
def native(case, isl, osl, index):
ids = (
tokenizer.encode(
f"Independent record {index}: {case}.\n", add_special_tokens=False
).ids
+ filler
)[:isl]
start = time.perf_counter()
first = last = None
final = None
events = 0
body = {
"input_ids": ids,
"sampling_params": {
"temperature": 0.0,
"max_new_tokens": osl,
"ignore_eos": True,
},
"stream": True,
}
with request("/generate", body) as response:
for line in response:
if not line.startswith(b"data:"):
continue
value = line[5:].strip()
if value == b"[DONE]":
break
obj = json.loads(value)
now = time.perf_counter()
if obj.get("error"):
raise RuntimeError(str(obj["error"]))
if obj.get("meta_info", {}).get("completion_tokens", 0) > 0:
first = first or now
last = now
final = obj
events += 1
elapsed = time.perf_counter() - start
assert final is not None, (case, index)
meta = final["meta_info"]
assert meta["completion_tokens"] == osl and meta["prompt_tokens"] == isl, meta
assert meta.get("cached_tokens") == 0, meta
return {
"kind": "request",
"case": case,
"index": index,
"input_hash": hashlib.sha256(json.dumps(ids).encode()).hexdigest(),
"prompt_tokens": isl,
"completion_tokens": osl,
"cached_tokens": meta["cached_tokens"],
"ttft_s": first - start,
"elapsed_s": elapsed,
"decode_tokens_per_s": (osl - 1) / (last - first) if osl > 1 else None,
"tpot_ms": 1000 * (last - first) / (osl - 1) if osl > 1 else None,
"events": events,
}
cases = [
("8k_1000_c1", 8192, 1000, 1),
("1k_128_c1", 1024, 128, 1),
("1k_128_c8", 1024, 128, 8),
("1k_128_c32", 1024, 128, 32),
("8k_128_c8", 8192, 128, 8),
("32k_128_c1", 32768, 128, 1),
]
emit(
{
"kind": "start",
"cases": cases,
"repetitions": 3,
}
)
try:
for case, isl, osl, concurrency in cases:
flush()
emit({"kind": "warmup_start", "case": case})
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
list(
pool.map(
lambda i: native("warm_" + case, isl, 16, i), range(concurrency)
)
)
for repeat in range(3):
flush()
start = time.perf_counter()
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
rows = list(
pool.map(
lambda i: native(case, isl, osl, i + repeat * 100),
range(concurrency),
)
)
wall = time.perf_counter() - start
for row in rows:
emit({**row, "repeat": repeat})
emit(
{
"kind": "batch",
"case": case,
"repeat": repeat,
"concurrency": concurrency,
"wall_s": wall,
"output_tokens_per_s": concurrency * osl / wall,
"median_ttft_s": statistics.median(x["ttft_s"] for x in rows),
"median_decode_tokens_per_s": statistics.median(
x["decode_tokens_per_s"] for x in rows
),
"median_tpot_ms": statistics.median(x["tpot_ms"] for x in rows),
}
)
emit({"kind": "complete"})
except Exception as exc:
emit({"kind": "failed", "error": repr(exc)})
raise
+3 -4
View File
@@ -15,7 +15,7 @@ ARG GITHUB_ARTIFACTORY=github.com
ARG INSTALL_FLASHINFER_JIT_CACHE=0
ARG FLASHINFER_VERSION=0.6.18
ARG MOONCAKE_VERSION=0.3.13
ARG MSCCLPP_VERSION=sglang-v0.10.1
ARG MSCCLPP_VERSION=sglang-v0.9.1
ENV DEBIAN_FRONTEND=noninteractive \
CUDA_HOME=/usr/local/cuda \
@@ -767,13 +767,12 @@ COPY --from=framework_final /root/.cache/huggingface /root/.cache/huggingface
COPY --from=framework_final /root/.cache/sglang /root/.cache/sglang
# Copy GDRCopy runtime libraries (but not the build artifacts)
COPY --from=framework_final /usr/lib/*-linux-gnu/libgdrapi.so* /usr/lib/
COPY --from=framework_final /usr/lib/libgdrapi.so* /usr/lib/
COPY --from=framework_final /usr/bin/gdrcopy_* /usr/bin/
COPY --from=framework_final /usr/src/gdrdrv-2.5.1 /usr/src/gdrdrv-2.5.1
# Fix DeepEP IBGDA symlink in runtime
RUN ln -sf /usr/lib/$(uname -m)-linux-gnu/libmlx5.so.1 /usr/lib/$(uname -m)-linux-gnu/libmlx5.so \
&& ldconfig
RUN ln -sf /usr/lib/$(uname -m)-linux-gnu/libmlx5.so.1 /usr/lib/$(uname -m)-linux-gnu/libmlx5.so
WORKDIR /sgl-workspace/sglang
+100 -27
View File
@@ -1,28 +1,100 @@
# CUDA 13.4 only args
# Torch nighly args
ARG CUDA_VERSION=13.4.0
ARG UBUNTU_BASE_IMAGE=ubuntu:24.04
ARG CUDA_PKG_VERSION=13-4
ARG CUDA_PREVIEW_REPO=https://packages.nvidia.com/noble
ARG CUDA_PREVIEW_SUITE=prerelease/cuda/13.4.0
ARG TORCH_NIGHTLY_INDEX=https://download.pytorch.org/whl/nightly/cu134
ARG TORCH_NIGHTLY_VERSION=2.15.0.dev20260818+cu134
ARG TORCHVISION_NIGHTLY_VERSION=0.30.0.dev20260819+cu134
ARG TORCHAUDIO_NIGHTLY_VERSION=2.11.0.dev20260818+cu134
# Wheel build args
ARG MANYLINUX_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.4
ARG DEEPGEMM_REF=v0.2.0
ARG DEEPGEMM_REF=dev
ARG DEEPEP_SOURCE_REF=sgl-deepep
ARG DEEPEP_PACKAGING_REF=sgl-deepep-packaging
ARG SGL_DEEP_EP_VERSION=0.1.2
ARG SGL_DEEP_EP_VERSION=0.1.0
ARG CUDA_VERSION=13.4.1
FROM nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu24.04 AS base
########################################################
# CUDA base: Ubuntu + CUDA 13.4 developer preview toolkit
########################################################
FROM ${UBUNTU_BASE_IMAGE} AS cuda_base
ARG CUDA_PKG_VERSION
ARG CUDA_PREVIEW_REPO
ARG CUDA_PREVIEW_SUITE
RUN printf 'Acquire::Retries "5";\nAcquire::http::Timeout "30";\nAcquire::https::Timeout "30";\n' \
> /etc/apt/apt.conf.d/80-net-hardening
RUN export DEBIAN_FRONTEND=noninteractive \
&& apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates wget gnupg \
&& wget -q -O /tmp/nvidia-preview-keyring.deb "${CUDA_PREVIEW_REPO}/nvidia-preview-keyring.deb" \
&& dpkg -i /tmp/nvidia-preview-keyring.deb \
&& rm -f /tmp/nvidia-preview-keyring.deb \
&& printf '%s\n' \
'X-Repolib-Name: NVIDIA Packages (frozen)' \
'Types: deb' \
"URIs: ${CUDA_PREVIEW_REPO}" \
"Suites: ${CUDA_PREVIEW_SUITE}" \
'Components: main' \
'Signed-By: /usr/share/keyrings/nvidia-packages-preview.gpg' \
'Enabled: yes' \
> /etc/apt/sources.list.d/nvidia-packages-preview.sources \
&& apt-get update \
&& apt-get install -y --no-install-recommends "cuda-toolkit-${CUDA_PKG_VERSION}" \
&& cuda_dir="$(ls -d /usr/local/cuda-1* 2>/dev/null | head -1)" \
&& test -n "${cuda_dir}" \
&& ln -sfn "${cuda_dir}" /usr/local/cuda \
&& /usr/local/cuda/bin/nvcc --version \
&& rm -rf /var/lib/apt/lists/*
# The 13.4 preview repo ships no NCCL, but later stages need libnccl2 /
# libnccl-dev: pynccl_allocator JIT-compiles with -lnccl under symmetric memory.
# The repo must be removed again in this same layer, not left behind an apt pin:
# the devtools repo that nsight-systems-cli adds later publishes byte-identical
# release metadata (o=NVIDIA, l=NVIDIA CUDA, c=) on this same host, so any pin
# that stops the CUDA repo's older cuda-* packages from shadowing the 13.4
# toolkit also blocks nsight-systems-cli. Removal is safe: the later
# `apt-get install libnccl2 libnccl-dev` resolves from dpkg status.
RUN export DEBIAN_FRONTEND=noninteractive \
&& case "$(dpkg --print-architecture)" in \
arm64) repo_arch=sbsa ;; \
amd64) repo_arch=x86_64 ;; \
*) echo "unsupported arch: $(dpkg --print-architecture)" >&2; exit 1 ;; \
esac \
&& wget -q -O /tmp/cuda-keyring.deb \
"https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/${repo_arch}/cuda-keyring_1.1-1_all.deb" \
&& dpkg -i /tmp/cuda-keyring.deb \
&& rm -f /tmp/cuda-keyring.deb \
&& apt-get update \
&& apt-get install -y --no-install-recommends libnccl2 libnccl-dev \
&& test -e "/usr/lib/$(uname -m)-linux-gnu/libnccl.so" \
&& rm -f /etc/apt/sources.list.d/cuda*.list /etc/apt/sources.list.d/cuda*.sources \
&& dpkg -r cuda-keyring \
&& rm -rf /var/lib/apt/lists/*
ENV PATH=/usr/local/cuda/bin:${PATH} \
LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
NVIDIA_VISIBLE_DEVICES=all \
NVIDIA_DRIVER_CAPABILITIES=compute,utility
########################################################
# Base stage
########################################################
FROM cuda_base AS base
ARG TARGETARCH
ARG BUILD_TYPE=all
ARG BRANCH_TYPE=remote
ARG SGL_KERNEL_VERSION=0.4.7
ARG SGL_KERNEL_VERSION=0.4.6.post1
ARG SGL_VERSION
ARG SGL_DEEP_GEMM_VERSION=0.2.0
ARG SGL_DEEP_GEMM_VERSION=0.1.5.post2
ARG USE_LATEST_SGLANG=0
ARG GDRCOPY_VERSION=2.5.1
ARG SGL_NCCL_VERSION=2.30.7
ARG NCCL_VERSION=2.30.7
ARG PIP_DEFAULT_INDEX
ARG UBUNTU_MIRROR
ARG GITHUB_ARTIFACTORY=github.com
@@ -279,7 +351,7 @@ RUN --mount=type=cache,id=cu134-sgl-kernel-ccache,target=/ccache \
FROM cu134_wheel_base AS cu134_wheel_deepgemm
ARG DEEPGEMM_REF
ARG SGL_DEEP_GEMM_VERSION=0.2.0
ARG SGL_DEEP_GEMM_VERSION=0.1.5.post2
ARG TVM_FFI_VERSION=0.1.11
# DeepJIT's exception handling requires elfutils/libdwfl.h and libelf headers.
@@ -298,6 +370,8 @@ COPY scripts/rename_sgl_deep_gemm_whl.sh /rename_sgl_deep_gemm_whl.sh
RUN --mount=type=cache,id=cu134-wheels-pip,target=/root/.cache/pip \
set -eux; \
cd /deepgemm; \
sed -i "s/'-std=c++17'/'-std=c++20'/g" \
build_sgl_deep_gemm.sh setup.py sgl_deep_gemm/__init__.py; \
printf '%s' "${SGL_DEEP_GEMM_VERSION}" > sgl_deep_gemm/VERSION; \
bash build_sgl_deep_gemm.sh; \
bash /rename_sgl_deep_gemm_whl.sh dist cu134 "$(uname -m)"; \
@@ -340,6 +414,7 @@ RUN git clone --recursive --depth 1 --branch "${DEEPEP_SOURCE_REF}" \
RUN set -eux; \
printf '%s' "${SGL_DEEP_EP_VERSION}" > /deepep-packaging/sgl_deep_ep/VERSION; \
sed -i 's/"-std=c++17"/"-std=c++20"/g' /deepep-source/setup.py; \
${PYTHON_ROOT_PATH}/bin/python - <<'PY'
import pathlib
p = pathlib.Path("/deepep-packaging/sgl_deep_ep/build_sgl_deep_ep.sh")
@@ -398,9 +473,7 @@ ARG CUDA_VERSION
ARG BUILD_TYPE
ARG SGL_KERNEL_VERSION
ARG GITHUB_ARTIFACTORY
# Renamed from NCCL_VERSION: the nvidia/cuda base image sets ENV NCCL_VERSION
# (Debian apt format, e.g. 2.31.2-1), which shadows a same-named ARG inside RUN.
ARG SGL_NCCL_VERSION
ARG NCCL_VERSION
ARG TORCH_NIGHTLY_INDEX
ARG TORCH_NIGHTLY_VERSION
ARG TORCHVISION_NIGHTLY_VERSION
@@ -423,14 +496,14 @@ RUN --mount=type=cache,target=/root/.cache/pip \
python3 -m pip install --upgrade pip setuptools wheel html5lib six \
&& case "$CUDA_VERSION" in \
13.0.3) CUINDEX=130 ;; \
13.4.1) CUINDEX=134 ;; \
13.4.0) CUINDEX=134 ;; \
*) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \
esac \
&& if [ "$CUDA_VERSION" = "13.0.3" ]; then \
# --no-deps prevents pip from pulling torch from default PyPI
python3 -m pip install sglang-kernel==${SGL_KERNEL_VERSION} --force-reinstall --no-deps \
; \
elif [ "$CUDA_VERSION" = "13.4.1" ]; then \
elif [ "$CUDA_VERSION" = "13.4.0" ]; then \
# Use locally built sgl-* wheels.
python3 -m pip install --force-reinstall --no-deps /tmp/cu134_wheels/*.whl \
; \
@@ -454,7 +527,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cargo/registry \
case "$CUDA_VERSION" in \
13.0.3) CUINDEX=130 ;; \
13.4.1) CUINDEX=134 ;; \
13.4.0) CUINDEX=134 ;; \
*) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \
esac \
&& cd /tmp/sglang_deps/python \
@@ -463,10 +536,10 @@ RUN --mount=type=cache,target=/root/.cache/pip \
&& echo '__version__ = "0.0.0"' > sglang/version.py \
&& touch README.md \
&& touch LICENSE \
&& if [ "$CUDA_VERSION" = "13.4.1" ]; then \
&& if [ "$CUDA_VERSION" = "13.4.0" ]; then \
sed -i "s|^ \"flashinfer_python\[cu13\]==.*| \"flashinfer_python[cu13]==${FLASHINFER_VERSION}\",|" pyproject.toml; \
fi \
&& if [ "$CUDA_VERSION" = "13.4.1" ]; then \
&& if [ "$CUDA_VERSION" = "13.4.0" ]; then \
TORCH_INDEX="${TORCH_NIGHTLY_INDEX}"; \
python3 -m pip install --index-url "${TORCH_NIGHTLY_INDEX}" \
"torch==${TORCH_NIGHTLY_VERSION}" \
@@ -484,8 +557,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \
fi \
&& python3 -m pip install --extra-index-url "${TORCH_INDEX}" ".[${BUILD_TYPE}]" \
&& python3 -m pip install --force-reinstall --no-deps \
"nvidia-nccl-cu13==${SGL_NCCL_VERSION}" \
&& if [ "$CUDA_VERSION" = "13.4.1" ]; then \
"nvidia-nccl-cu13==${NCCL_VERSION}" \
&& if [ "$CUDA_VERSION" = "13.4.0" ]; then \
python3 -m pip install --force-reinstall --no-deps /tmp/cu134_wheels/*.whl; \
fi \
&& cd /sgl-workspace \
@@ -498,7 +571,7 @@ RUN python3 -m pip install --ignore-installed --no-deps distro
# constraints.txt pins the cu134 nightly torch, which does not exist on PyPI;
# later stages cannot satisfy it without the nightly index in scope.
RUN if [ "$CUDA_VERSION" = "13.4.1" ]; then \
RUN if [ "$CUDA_VERSION" = "13.4.0" ]; then \
python3 -m pip config set global.extra-index-url "${TORCH_NIGHTLY_INDEX}"; \
fi
@@ -539,7 +612,7 @@ ARG FLASHINFER_VERSION
RUN --mount=type=cache,target=/root/.cache/pip \
case "$CUDA_VERSION" in \
13.0.3) CUINDEX=130 ;; \
13.4.1) CUINDEX=134 ;; \
13.4.0) CUINDEX=134 ;; \
*) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \
esac \
&& mkdir -p /flashinfer_jit_output \
@@ -745,7 +818,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# one, so the constraints-pinned cu134 nightly must be restated here; otherwise
# runai-model-streamer's "torch<3.0.0,>=2.0.0" backtracks the whole resolve.
RUN --mount=type=cache,target=/root/.cache/pip \
if [ "$CUDA_VERSION" = "13.4.1" ]; then TORCH_SPEC="torch==${TORCH_NIGHTLY_VERSION}"; else TORCH_SPEC=""; fi \
if [ "$CUDA_VERSION" = "13.4.0" ]; then TORCH_SPEC="torch==${TORCH_NIGHTLY_VERSION}"; else TORCH_SPEC=""; fi \
&& python3 -m pip install -c /sgl-workspace/constraints.txt \
${TORCH_SPEC} \
datamodel_code_generator \
@@ -844,7 +917,7 @@ RUN if [ "$BRANCH_TYPE" = "local" ]; then \
# Clean up __pycache__/tests/pyc in same RUN to avoid writing ~28k files to layer
RUN --mount=type=cache,target=/root/.cache/pip \
cd /sgl-workspace/sglang \
&& if [ "$CUDA_VERSION" = "13.4.1" ]; then \
&& if [ "$CUDA_VERSION" = "13.4.0" ]; then \
sed -i "s|^ \"flashinfer_python\[cu13\]==.*| \"flashinfer_python[cu13]==${FLASHINFER_VERSION}\",|" python/pyproject.toml; \
fi \
&& python3 -m pip install --no-deps -e "python[${BUILD_TYPE}]" \
@@ -885,7 +958,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# quack-kernels hard-pins nvidia-cutlass-dsl==4.6.2, override.
RUN --mount=type=cache,target=/root/.cache/pip \
if [ "$CUDA_VERSION" = "13.4.1" ]; then \
if [ "$CUDA_VERSION" = "13.4.0" ]; then \
python3 -m pip uninstall -y \
nvidia-cutlass-dsl \
nvidia-cutlass-dsl-libs-base \
@@ -932,7 +1005,7 @@ LABEL org.opencontainers.image.source="https://github.com/sgl-project/sglang" \
# - Ready-to-run inference server environment
#
# Note: Uses devel base for complete NVCC toolchain required by DeepGEMM JIT
FROM nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu24.04 AS runtime
FROM cuda_base AS runtime
ARG CUDA_VERSION
ARG TARGETARCH
+29 -87
View File
@@ -1,46 +1,40 @@
ARG CANN_VERSION=9.1.0
ARG DEVICE_TYPE=950
ARG CANN_VERSION=9.0.0
ARG DEVICE_TYPE=a3
ARG OS=ubuntu22.04
ARG PYTHON_VERSION=py3.12
ARG arch
ARG PYTHON_VERSION=py3.11
FROM quay.io/ascend/cann:$CANN_VERSION-$DEVICE_TYPE-$OS-$PYTHON_VERSION
# Update pip & apt sources
ARG TARGETARCH
ARG CANN_VERSION
ARG DEVICE_TYPE
ARG arch
ARG PIP_INDEX_URL="https://pypi.org/simple/"
ARG APTMIRROR=""
# torch_npu 2.10.0.post6 requires torch==2.10.0, so PYTORCH_VERSION stays at 2.10.0
ARG PYTORCH_VERSION="2.10.0"
ARG TORCHVISION_VERSION="0.25.0"
ARG TORCHAUDIO_VERSION="2.10.0"
ARG TORCH_NPU_VERSION="2.10.0.post6"
ARG TORCH_NPU_INDEX_URL="https://ascend.devcloud.huaweicloud.com/pypi/simple/"
ARG PTA_URL_ARM64="https://gitcode.com/Ascend/pytorch/releases/download/v26.0.0-pytorch2.10.0/torch_npu-2.10.0-cp311-cp311-manylinux_2_28_aarch64.whl"
ARG PTA_URL_AMD64="https://gitcode.com/Ascend/pytorch/releases/download/v26.0.0-pytorch2.10.0/torch_npu-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl"
ARG SGLANG_TAG=main
ARG ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit
ARG SGLANG_KERNEL_NPU_TAG=2026.9.0.post5
ARG SGLANG_KERNEL_NPU_TAG=main
ARG PIP_INSTALL="python3 -m pip install --no-cache-dir"
ARG DEVICE_TYPE
ARG MODELSCOPE_VERSION=""
ARG EVALSCOPE_VERSION=""
ARG MF_VERSION="1.2.1"
ARG MF_WHEEL_URL_AARCH64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl"
ARG MF_WHEEL_URL_X86_64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl"
ARG MC_WHEEL_URL_AARCH64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/memcache/v1.2.1/20260923.4/memcache_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl"
ARG MC_WHEEL_URL_X86_64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/memcache/v1.2.1/20260923.4/memcache_hybrid-1.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl"
RUN if [ "$TARGETARCH" = "amd64" ]; then \
echo "Using x86_64 dependencies"; \
echo "PTA_URL=$PTA_URL_AMD64" >> /etc/environment_new; \
elif [ "$TARGETARCH" = "arm64" ]; then \
echo "Using aarch64 dependencies"; \
echo "PTA_URL=$PTA_URL_ARM64" >> /etc/environment_new; \
else \
echo "Unsupported TARGETARCH: $TARGETARCH"; exit 1; \
fi
# memfabric-zbal: 950 与 a3 使用不同版本
ARG ZBAL_VERSION_950="1.2.21004.post1"
ARG ZBAL_VERSION_A3="1.1.3"
# Later RUN steps source /etc/environment_new, so make sure it exists
RUN touch /etc/environment_new
WORKDIR /workspace
# Set workspace directory
WORKDIR /sgl-workspace/sglang
# Define environments
ENV DEBIAN_FRONTEND=noninteractive
@@ -62,6 +56,7 @@ RUN apt-get update -y && apt upgrade -y && apt-get install -y \
clang \
locales \
ccache \
ffmpeg \
openssl \
libssl-dev \
pkg-config \
@@ -77,30 +72,7 @@ ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
### Install MemFabric and MemCache (按 TARGETARCH 直接取对应架构的 OBS wheel)
RUN set -eux; \
case "$TARGETARCH" in \
arm64) \
MF_URL="$MF_WHEEL_URL_AARCH64"; \
MC_URL="$MC_WHEEL_URL_AARCH64"; \
;; \
amd64) \
MF_URL="$MF_WHEEL_URL_X86_64"; \
MC_URL="$MC_WHEEL_URL_X86_64"; \
;; \
*) \
echo "Unsupported architecture: $TARGETARCH" >&2; \
exit 1; \
;; \
esac; \
${PIP_INSTALL} "$MF_URL" --force-reinstall; \
mfcli kernel install; \
${PIP_INSTALL} "$MC_URL" --force-reinstall --no-deps
### Install memfabric-zbal
RUN if [ "$DEVICE_TYPE" = "950" ]; then ZBAL_PKG="memfabric-zbal==${ZBAL_VERSION_950}"; \
else ZBAL_PKG="memfabric-zbal==${ZBAL_VERSION_A3}"; fi; \
${PIP_INSTALL} "$ZBAL_PKG" -i https://pypi.org/simple/
### Install SGLang Model Gateway
RUN ${PIP_INSTALL} sglang-router
@@ -108,46 +80,20 @@ RUN ${PIP_INSTALL} sglang-router
### Install PyTorch and PTA
RUN . /etc/environment_new && \
(${PIP_INSTALL} torch==${PYTORCH_VERSION} torchvision==${TORCHVISION_VERSION} torchaudio==${TORCHAUDIO_VERSION} --index-url https://download.pytorch.org/whl/cpu) \
&& (${PIP_INSTALL} torch-npu==${TORCH_NPU_VERSION} --extra-index-url ${TORCH_NPU_INDEX_URL})
### Install ModelScope & EvalScope
# Installed right after torch/torch-npu so their dependencies resolve against the pinned torch.
# MODELSCOPE_VERSION / EVALSCOPE_VERSION are empty by default -> latest release.
RUN . /etc/environment_new && \
MS_PKG="modelscope" && \
ES_PKG="evalscope" && \
if [ -n "${MODELSCOPE_VERSION}" ]; then MS_PKG="modelscope==${MODELSCOPE_VERSION}"; fi && \
if [ -n "${EVALSCOPE_VERSION}" ]; then ES_PKG="evalscope==${EVALSCOPE_VERSION}"; fi && \
${PIP_INSTALL} "${MS_PKG}" "${ES_PKG}"
&& (${PIP_INSTALL} ${PTA_URL})
## Install triton-ascend
RUN . /etc/environment_new && \
${PIP_INSTALL} pybind11 && \
if [ "$TARGETARCH" = "arm64" ]; then \
${PIP_INSTALL} https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ta/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl; \
elif [ "$TARGETARCH" = "amd64" ]; then \
${PIP_INSTALL} https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; \
else \
echo "Unsupported architecture: $TARGETARCH"; \
exit 1; \
fi
RUN (${PIP_INSTALL} pybind11) && \
(${PIP_INSTALL} triton-ascend==3.2.1.dev20260530 --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly --trusted-host triton-ascend.osinfra.cn)
# Install SGLang
RUN git clone https://github.com/sgl-project/sglang --branch ${SGLANG_TAG} /sgl-workspace/sglang && \
# Install SGLang (editable mode to preserve source and git history)
RUN git clone https://github.com/sgl-project/sglang --branch $SGLANG_TAG /sgl-workspace/sglang && \
cd /sgl-workspace/sglang/python && rm -rf pyproject.toml && mv pyproject_npu.toml pyproject.toml && \
sed -i '/"memfabric-hybrid==1.1.4"/d; /"memfabric-zbal==1.1.2"/d' pyproject.toml && \
${PIP_INSTALL} -v -e .[all_npu]
ENV ASCEND_HOME_PATH=/usr/local/Ascend/cann-${CANN_VERSION}
ENV LD_LIBRARY_PATH=/usr/local/Ascend/cann-${CANN_VERSION}/lib64:/usr/local/Ascend/cann-${CANN_VERSION}/lib:/usr/local/Ascend/cann-${CANN_VERSION}/x86_64-linux/devlib/device:/usr/local/Ascend/driver/lib64:/usr/local/lib:${LD_LIBRARY_PATH}
RUN mkdir cann-custom-ops && \
cd cann-custom-ops && \
source /usr/local/Ascend/cann-${CANN_VERSION}/set_env.sh && \
wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/custom-ops-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip && \
wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/ops-transformer-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip && \
unzip custom-ops-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip && \
@@ -155,11 +101,7 @@ RUN mkdir cann-custom-ops && \
chmod +x *.run && \
./CANN-custom_ops-none-linux.$(arch).run --install-path=/usr/local/Ascend/cann-${CANN_VERSION}/opp && \
./cann-ops-transformer-custom_linux-$(arch).run --install-path=/usr/local/Ascend/cann-${CANN_VERSION}/opp && \
source /usr/local/Ascend/cann-${CANN_VERSION}/opp/vendors/customize/bin/set_env.bash && \
source /usr/local/Ascend/cann-${CANN_VERSION}/opp/vendors/custom_transformer/bin/set_env.bash && \
source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh && \
source /usr/local/Ascend/nnal/atb/set_env.sh && \
${PIP_INSTALL} custom_ops-1.0-cp312-cp312-linux_$(arch).whl && \
${PIP_INSTALL} custom_ops-1.0-cp311-cp311-linux_$(arch).whl && \
cd .. && rm -rf cann-custom-ops
# Install Deep-ep
@@ -167,9 +109,9 @@ RUN mkdir cann-custom-ops && \
RUN ${PIP_INSTALL} wheel==0.45.1 pybind11 pyyaml decorator scipy attrs psutil \
&& mkdir sgl-kernel-npu \
&& cd sgl-kernel-npu \
&& wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py312-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \
&& unzip sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py312-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \
&& ${PIP_INSTALL} deep_ep*.whl sgl_kernel_npu*.whl torch_memory_saver*.whl \
&& wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py311-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \
&& unzip sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py311-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \
&& ${PIP_INSTALL} deep_ep*.whl sgl_kernel_npu*.whl attentions*.whl \
&& cd .. && rm -rf sgl-kernel-npu \
&& cd "$(python3 -m pip show deep-ep | awk '/^Location:/ {print $2}')" && ln -sf deep_ep/deep_ep_cpp*.so
+1 -12
View File
@@ -113,7 +113,7 @@ from pathlib import Path
project = tomllib.loads(Path("python/pyproject.toml").read_text())["project"]
extras = project["optional-dependencies"]
requirements = list(project["dependencies"])
for group in ("runtime_base", "runtime_common", "srt_hip", "diffusion_common"):
for group in ("runtime_base", "runtime_common", "srt_hip"):
requirements.extend(
"compressed-tensors==0.15.0"
if requirement == "compressed-tensors"
@@ -121,17 +121,6 @@ for group in ("runtime_base", "runtime_common", "srt_hip", "diffusion_common"):
for requirement in extras[group]
if not requirement.startswith("sglang[") and requirement != "torch"
)
# diffusion_hip's pure-Python packages, needed by sglang.multimodal_gen
# (Wan2.2/video-gen) at import time. st_attn/vsa are deliberately left out:
# they're prebuilt CUDA-oriented wheels of unverified gfx1151 compatibility,
# same posture as skipping aiter's compiled CDNA kernels below.
requirements.extend(
requirement
for requirement in extras["diffusion_hip"]
if not requirement.startswith("sglang[")
and not requirement.startswith("st_attn")
and not requirement.startswith("vsa")
)
requirements = list(dict.fromkeys(requirements))
subprocess.check_call(
[sys.executable, "-m", "pip", "install", "--no-cache-dir", *requirements]
+26 -8
View File
@@ -586,12 +586,6 @@ RUN git clone ${AITER_REPO} \
git revert --no-edit --no-commit e708f6c15; \
fi
# ROCm/aiter#5195's .co with a mask assembled in between the q rows sharing a
# tile, which the grouped dsv4 decode needs; bit-identical at max_seqlen_q=1.
COPY --from=local_src \
/src/python/sglang/kernels/ops/attention/dsv4/asm/gfx950/mla_v4/mla_a8w8_qh64_qseqlen1_gqaratio64_nm.co \
/sgl-workspace/aiter/hsa/gfx950/mla_v4/
# The pinned AITER revision uses std::optional in topk_per_row_kernels.cu without
# including <optional>: ROCm/aiter#4702 removed the torch headers that previously
# supplied it transitively. ROCm 7.0 therefore fails module_top_k_per_row, while
@@ -793,7 +787,7 @@ RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment
RUN /bin/bash -lc 'set -euo pipefail; \
echo "[TileLang] Building TileLang for ${GPU_ARCH}"; \
# System dependencies
# System dependencies (NO llvm-dev to avoid llvm-config-16 shadowing)
apt-get update && apt-get install -y --no-install-recommends \
build-essential git wget curl ca-certificates gnupg \
libgtest-dev libgmock-dev \
@@ -822,6 +816,30 @@ RUN /bin/bash -lc 'set -euo pipefail; \
"$VENV_PIP" install --upgrade "setuptools>=77.0.3,<80" wheel "cmake==4.3.4" ninja scikit-build-core && \
"$VENV_PIP" cache purge || true; \
\
# Locate ROCm llvm-config; fallback to installing LLVM 18 if missing
LLVM_CONFIG_PATH=""; \
for p in /opt/rocm/llvm/bin/llvm-config /opt/rocm/llvm-*/bin/llvm-config /opt/rocm-*/llvm*/bin/llvm-config; do \
if [ -x "$p" ]; then LLVM_CONFIG_PATH="$p"; break; fi; \
done; \
if [ -z "$LLVM_CONFIG_PATH" ]; then \
echo "[TileLang] ROCm llvm-config not found; installing LLVM 18..."; \
curl -fsSL --retry 5 --retry-delay 3 --retry-all-errors https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor -o /etc/apt/keyrings/llvm.gpg; \
echo "deb [signed-by=/etc/apt/keyrings/llvm.gpg] http://apt.llvm.org/jammy/ llvm-toolchain-jammy-18 main" > /etc/apt/sources.list.d/llvm.list; \
apt-get update; \
apt-get install -y --no-install-recommends llvm-18; \
rm -rf /var/lib/apt/lists/*; \
LLVM_CONFIG_PATH="$(command -v llvm-config-18)"; \
if [ -z "$LLVM_CONFIG_PATH" ]; then echo "ERROR: llvm-config-18 not found after install"; exit 1; fi; \
fi; \
echo "[TileLang] Using LLVM_CONFIG at: $LLVM_CONFIG_PATH"; \
export PATH="$(dirname "$LLVM_CONFIG_PATH"):/usr/local/bin:${PATH}"; \
export LLVM_CONFIG="$LLVM_CONFIG_PATH"; \
\
# Optional shim for tools that expect llvm-config-16
mkdir -p /usr/local/bin && \
printf "#!/usr/bin/env bash\nexec \"%s\" \"\$@\"\n" "$LLVM_CONFIG_PATH" > /usr/local/bin/llvm-config-16 && \
chmod +x /usr/local/bin/llvm-config-16; \
\
# TVM Python bits need Cython + z3 before configure.
# Pin z3-solver==4.15.4.0: 4.15.4.0 has a manylinux wheel; 4.15.5.0 has no wheel and builds from source (fails: C++20 <format> needs GCC 14+, image has GCC 11).
"$VENV_PIP" install --no-cache-dir "cython>=0.29.36,<3.0" "apache-tvm-ffi @ git+https://github.com/apache/tvm-ffi.git@37d0485b2058885bf4e7a486f7d7b2174a8ac1ce" "z3-solver==4.15.4.0"; \
@@ -837,7 +855,7 @@ RUN /bin/bash -lc 'set -euo pipefail; \
else \
export ROCM_PATH=/opt/rocm; \
fi; \
export CMAKE_ARGS="-DUSE_CUDA=OFF -DUSE_ROCM=ON -DROCM_PATH=${ROCM_PATH} -DSKBUILD_SABI_VERSION= ${CMAKE_ARGS:-}" && \
export CMAKE_ARGS="-DUSE_CUDA=OFF -DUSE_ROCM=ON -DROCM_PATH=${ROCM_PATH} -DLLVM_CONFIG=${LLVM_CONFIG} -DSKBUILD_SABI_VERSION= ${CMAKE_ARGS:-}" && \
"$VENV_PIP" install -e . -v --no-build-isolation --no-deps; \
if [ -f pyproject.toml ]; then sed -i "/^[[:space:]]*\"torch/d" pyproject.toml || true; fi; \
"$VENV_PIP" cache purge || true; \
Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

@@ -388,14 +388,9 @@ Thus for all positive \(a,b,c\) with \(abc=1\),
**Benchmark Command:**
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
**Test Results:**
```text Output
@@ -977,14 +977,12 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 \
--num-threads 1316
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
**Test Results:**
```text Output
@@ -597,13 +597,9 @@ Max ITL (ms): 43465.60
- **Benchmark Command:**
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:8000/v1 \
--num-examples 200
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- **Test Results**:
- DeepSeek-V3
```text Output
@@ -618,13 +614,11 @@ The command above uses sgl-eval. Historical scores from other harnesses are not
- **Benchmark Command:**
```shell Command
sgl-eval run mmlu \
--base-url http://127.0.0.1:8000/v1 \
--num-threads 64 --max-tokens 2048
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000
```
The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above.
- **Test Results**:
- DeepSeek-V3
```text Output
@@ -973,14 +973,12 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 \
--num-threads 1316
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
**Test Results:**
```text Output
@@ -709,13 +709,9 @@ Max ITL (ms): 975.03
- **Benchmark Command:**
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- **Test Results**:
- DeepSeek-V3.2-Exp
```
@@ -725,17 +721,12 @@ The command above uses sgl-eval. Historical scores from other harnesses are not
Output throughput: 965.919 token/s
```
- **Full GSM8K (1319 questions)** — evaluate the complete test set with sgl-eval:
- **Full GSM8K (1319 questions)** — for a stricter accuracy check, run the full set 8-shot:
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 \
--num-threads 1319
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- 8-shot:
```
Accuracy: 0.956
@@ -756,13 +747,11 @@ Historical results below use the removed harness and are not measurements of the
- **Benchmark Command:**
```shell Command
sgl-eval run mmlu \
--base-url http://127.0.0.1:30000/v1 \
--num-threads 64 --max-tokens 2048
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 30000
```
The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above.
- **Test Results**:
- DeepSeek-V3.2-Exp
```
@@ -785,15 +774,15 @@ The command evaluates all 57 subjects. Historical results below use the removed
- **Benchmark Command:**
```shell Command
sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --num-examples 198 --max-tokens 128000 --n-repeats 8 --thinking
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --thinking-mode deepseek-v3
```
- **Historical Test Results** (model: `deepseek-ai/DeepSeek-V3.2-Exp`, 8×B200; measured with the earlier `sglang.test.run_eval` GPQA command):
- **Test Results** (model: `deepseek-ai/DeepSeek-V3.2-Exp`, 8×B200):
- Default (`temperature=0`): mean **0.797** over 8 runs — closely matches the official GPQA-Diamond score of **79.9** for DeepSeek-V3.2-Exp reported in its [model card](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)
- With `temperature=1.0, top_p=0.95` (as recommended by DeepSeek):
```shell Command
sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --num-examples 198 --max-tokens 128000 --n-repeats 8 --top-p 0.95 --temperature 1.0 --thinking
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3
```
```
@@ -51,11 +51,11 @@ docker run --gpus all \
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
For example:
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260926`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi30x-20260926`
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260914`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260914`
```bash Command
docker pull lmsysorg/sglang-rocm:v0.5.20-rocm720-{mi35x,mi30x}-20260926
docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260914
docker run \
--device=/dev/kfd --device=/dev/dri \
@@ -65,7 +65,7 @@ docker run \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<your-hf-token>" \
lmsysorg/sglang-rocm:v0.5.20-rocm720-{mi35x,mi30x}-20260926 \
lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260914 \
sglang serve <use args below>
```
@@ -391,8 +391,8 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000.
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The 0813 Pro Official checkpoint keeps its MTP head too, so the same flags work there as a fallback to DSpark.
- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Prefer it over EAGLE on 0813 for ordinary serving: enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. DSpark also runs under **PD disaggregation** — see the Pro Official pairs in [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io).
- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The 0813 Pro Official checkpoint keeps its MTP head too, so the same flags are the speculative option under PD disaggregation — see the DSpark note below.
- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Prefer it over EAGLE on 0813 for ordinary serving: enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. The one exception is **PD disaggregation**, where DSpark cannot run at all and the Playground strips it — there, fall back to the MTP 3-1-4 shape above, as in [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io).
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
- **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe.
@@ -692,7 +692,7 @@ Larger blocks can improve decode latency when acceptance stays high, but they al
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
DSpark requires `pp_size == 1`. Selecting a prefill or decode role in the Playground keeps the inherited DSpark flags on both roles; for PD-disaggregated configurations that run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X Pro Official PD pairs](#3-8-pd-disaggregation-on-mi355x-mori-io). The MI355X Flash Official recipes run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
DSpark requires `pp_size == 1`. Selecting a prefill or decode role in the Playground removes the inherited DSpark flags; for a PD-disaggregated configuration that runs DSpark on both roles, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark). The MI355X Flash Official recipes run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
### 3.5 Vision (Image Inputs)
@@ -739,6 +739,7 @@ Pending update...
**TP8, concurrency 1–8:**
```bash Command
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
@@ -764,6 +765,7 @@ DSv4 HiCache sizes the host tier with `--hicache-ratio` (host/device token ratio
**DEP8 (DP Attention), concurrency 64–160:**
```bash Command
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
@@ -877,17 +879,12 @@ Concurrency 256 uses 2P1D — two prefill workers, one node each, added with a s
### 3.7 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark)
DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7). Keep `SGLANG_DSV4_UNIFIED_KV_FP8=1` on every arm.
- **BCG**: conc 1–4 — `--cuda-graph-backend-prefill breakable`
- **Grouped ASM**: TP-only, conc ≥ 16 — `SGLANG_DSV4_GROUPED_ASM=1`
- **EP**: MegaMoE; at concurrency 384 lower `--prefill-decode-interval` to `15`
DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7).
**TP8, concurrency 1–16:**
```bash Command
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
@@ -923,11 +920,10 @@ Concurrency 32–48 keeps the same TP-only command and adds HiCache DRAM offload
--hicache-mem-layout page_first_direct
```
**DP8 (DP Attention), concurrency 128:**
**DP8 (DP Attention), concurrency 128–256:**
```bash Command
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
@@ -950,7 +946,6 @@ python3 -m sglang.launch_server \
--stream-interval 20 \
--prefill-decode-interval 20 \
--prefill-delayer-token-usage-low-watermark 0.7 \
--load-balance-method total_requests \
--attention-backend dsv4 \
--enable-deepseek-v4-fp4-indexer \
--page-size 256 \
@@ -973,65 +968,6 @@ python3 -m sglang.launch_server \
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware` so multi-turn sessions land on the rank holding the longest radix/HiCache prefix; above concurrency 160 add `--balance-abs-threshold 32`.
**DP8 EP8 MegaMoE, concurrency 128–384:**
```bash Command
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=8192 \
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
MORI_SHMEM_HEAP_SIZE=17179869184 \
SGLANG_AITER_MEGA_RANK_SYNC=1 \
SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 \
SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 \
SGLANG_DP_USE_REDUCE_SCATTER=0 \
GPU_MAX_HW_QUEUES=5 \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--ep 8 \
--enable-dp-attention \
--enable-dp-lm-head \
--enable-prefill-delayer \
--enable-dp-attention-local-control-broadcast \
--tokenizer-worker-num 8 \
--stream-interval 20 \
--prefill-decode-interval 20 \
--prefill-delayer-token-usage-low-watermark 0.7 \
--load-balance-method total_requests \
--attention-backend dsv4 \
--enable-deepseek-v4-fp4-indexer \
--page-size 256 \
--swa-full-tokens-ratio 0.1 \
--kv-cache-dtype fp8_e4m3 \
--moe-a2a-backend megamoe \
--moe-dense-tp-size 1 \
--disable-shared-experts-fusion \
--enable-eplb \
--eplb-rebalance-num-iterations 200 \
--expert-distribution-recorder-mode stat \
--mem-fraction-static 0.85 \
--chunked-prefill-size 65536 \
--tool-call-parser deepseekv4 \
--reasoning-parser deepseek-v4 \
--speculative-algorithm DSPARK \
--speculative-dspark-block-size 6 \
--enable-hierarchical-cache \
--hicache-ratio 1.5 \
--hicache-write-policy write_through \
--hicache-io-backend direct \
--hicache-mem-layout page_first_direct \
--watchdog-timeout 3600
```
### 3.8 PD Disaggregation on MI355X (MORI-IO)
On ROCm the prefill and decode roles talk over **MORI-IO** rather than Mooncake or NiXL, so the **PD Disagg** card in the [Playground above](#playground) hides the CUDA-only transports once MI300X / MI355X is selected and defaults the IB device list to the node's `rdmaN` NICs instead of ConnectX `mlx5_*` names.
@@ -1047,50 +983,47 @@ docker run \
--ulimit nofile=1048576:1048576 \
--network host --ipc=host --shm-size 32g \
-v ~/.cache/huggingface:/root/.cache/huggingface \
lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260923 \
lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260914 \
python3 -m sglang.launch_server <role args below>
```
The Pro Official (0813) PD pairs ran end-to-end on this `v0.5.20` build, and the Playground's Docker output uses it for that variant. `--network host` replaces the single-node `-p` mapping because the two roles rendezvous across hosts. Long agentic runs on the 1.6T checkpoint may need more than `--shm-size 32g`; raise it if the workers die during weight load.
`--network host` replaces the single-node `-p` mapping because the two roles rendezvous across hosts. Long agentic runs on the 1.6T checkpoint may need more than `--shm-size 32g`; raise it if the workers die during weight load.
Selecting either role with **MORI** on MI355X emits the shared MORI-IO transport env — `SGLANG_MORI_COMBINE_DTYPE` plus the RDMA send-queue tuning `MORI_IO_SQ_BACKOFF_TIMEOUT_US` / `MORI_IO_QP_MAX_SEND_WR` — together with that role's sizing. `--max-running-requests` is deliberately small on both: the pair streams KV between workers rather than holding a large running batch.
The roles are sized per **Strategy**, so the Deploy panel's operating point carries through to the PD command:
- **Low-Latency** — TP-only, `--mem-fraction-static 0.86`, and a running-request ceiling of 8. Decode captures graphs for batches 1–8. No offload tier: the KV pool is all there is.
- **Pro Official (0813)** has its own low-latency pair: a **TP4 prefill** in front of a **TP8 decode**, both keeping the bundled **DSpark** head with `--speculative-dspark-block-size 6` (SGLang derives steps 1 / topk 1 / 7 draft tokens from it). The ceiling is `--max-running-requests 32` on both roles and decode ladders 1–32. Prefill adds a 16384 chunked-prefill budget, `--optimistic-prefill-attempts 2`, `--enable-cache-report`, and a **UMBP** tier (see [§3.9](#3-9-umbp-direct-external-kv-store)) that the Playground turns on by default. Both roles drop the base cell's `--prefill-decode-interval`, which only applies to aggregated serving.
- **Balanced** — still TP-only, with the ceiling at `--max-running-requests 96` and a 16384 chunked-prefill budget. Decode ladders all the way to 96, since without DP the server-wide ceiling *is* the per-rank batch. Prefill runs a **HiCache** host tier (`--hicache-ratio 2.5`, `page_first`, write-through, best-effort prefetch); decode does not, because HiCache is not recommended on the decode role with MORI.
- **Pro Official (0813)** runs balanced as the same asymmetric pair as its low-latency one, re-sized: a **TP4 prefill** in front of a **TP8 decode**, both with **DSpark** at `--speculative-dspark-block-size 3` (steps 1 / topk 1 / 4 draft tokens), a ceiling of `--max-running-requests 96` on both roles, and a decode ladder of 1–96. The balanced base cell is target-only DP, so the roles add `--speculative-algorithm DSPARK` back. Prefill keeps the 16384 chunked-prefill budget and adds `--optimistic-prefill-attempts 2` and `--enable-cache-report`. Its offload tier is **UMBP** rather than HiCache, on by default in the Playground; switch the UMBP card off to fall back to the HiCache tier above.
- **High-Throughput** — inherits TP8 / DP8 and the 65536 chunked-prefill budget from the base cell, raises the ceiling to `--max-running-requests 256` and `--mem-fraction-static 0.92`, and ladders decode to 32 — the per-rank batch 256 leaves across DP 8. It adds `--enable-cache-report`, which surfaces the prefix hit rate you need to judge whether an offload tier is paying for itself at that concurrency.
- **Pro Official (0813)** keeps TP8 / DP8 on both roles and adds **DSpark** at `--speculative-dspark-block-size 3`, the same draft shape as its balanced pair, plus `--enable-dp-lm-head`. The ceiling is `--max-running-requests 512` on both roles at `--mem-fraction-static 0.92`, which is 64 per DP rank, so decode ladders 1–64. Prefill keeps the base cell's 65536 chunked-prefill budget and `--swa-full-tokens-ratio 0.15`, adds `--optimistic-prefill-attempts 2`, and runs **UMBP** on by default in the Playground against the same standalone tier server as the other two Pro Official pairs. Decode drops `--chunked-prefill-size`, and both roles drop `--prefill-decode-interval`.
The offload tier is what actually separates balanced from high-throughput: balanced pairs prefill with HiCache's tiered GPU → host cache, while high-throughput pairs it with UMBP, which links the radix tree straight to MORI with no host tier. Pro Official is the exception, running UMBP under all three of its pairs, including the two TP-only ones. HiCache and UMBP cannot both be on, and the Playground enforces that.
The offload tier is what actually separates balanced from high-throughput: balanced pairs prefill with HiCache's tiered GPU → host cache, while high-throughput pairs it with UMBP, which links the radix tree straight to MORI with no host tier. They cannot both be on, and the Playground enforces that.
<Note>
The low-latency and balanced roles are TP-only, and that is exactly why their decode ladders run to the full ceiling. Selecting either role forces DP-Attention off and grays the control: `--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so switching DP on would cut the per-rank batch below the captured graphs without changing either flag in the command. With PD off, the balanced cell stays a DP recipe for aggregated serving.
</Note>
<Warning>
**Speculative decoding changes how you size the ceilings.** For a target concurrency of N, set `--max-running-requests N*2` on **both** roles. Sizing them at N caps the served batch below the concurrency you are aiming for. This applies to DSpark and EAGLE/MTP alike: the Pro Official low-latency pair's ceiling of 32 is sized for a concurrency of 16, its balanced pair's 96 for 48, and its high-throughput pair's 512 for 256.
**Speculative decoding (MTP) changes how you size the ceilings.** For a target concurrency of N, set `--max-running-requests N*2` on **both** roles. Sizing them at N caps the served batch below the concurrency you are aiming for. PD disaggregation requires EAGLE/MTP (DSpark cannot run under it), so this applies to every PD deployment.
`--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so the decode graph ladder should cover the **per-rank** batch that leaves — `N*2 / dp_size`. That is why the two points above ladder to different places from ceilings that differ by less: balanced is TP-only, so 96 slots are 96 per rank, while high-throughput spreads 256 over DP 8 and only ever sees 32 at a time (Pro Official's 512 over DP 8 is 64).
`--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so the decode graph ladder should cover the **per-rank** batch that leaves — `N*2 / dp_size`. That is why the two points above ladder to different places from ceilings that differ by less: balanced is TP-only, so 96 slots are 96 per rank, while high-throughput spreads 256 over DP 8 and only ever sees 32 at a time.
</Warning>
The high-throughput prefill role is where [UMBP](#3-9-umbp-direct-external-kv-store) belongs — DP attention is already on, which UMBP otherwise requires, and the concurrency is high enough for the pool to matter. On Pro Official, all three prefill roles enable UMBP by default, and the low-latency and balanced ones do so without DP as the TP-only exceptions. Elsewhere, turn on the **UMBP** card alongside the role; it is a separate card rather than part of the role because the linker flags are its to own, and leaving them implicit would hide the dependency.
The high-throughput prefill role is where [UMBP](#3-9-umbp-direct-external-kv-store) belongs — DP attention is already on, which is UMBP's hard requirement, and the concurrency is high enough for the pool to matter. Turn on the **UMBP** card alongside the role; it is a separate card rather than part of the role because the linker flags are its to own, and leaving them implicit would hide the dependency.
Whichever point you pick, the two roles diverge in two places:
- **Cuda graphs.** Prefill runs eager (`--disable-cuda-graph`) because the FP4 indexer's prefill path is not captured. Decode captures the small batch ladder that matches its running-request ceiling (`--cuda-graph-bs-decode 1 2 3 4 5 6 7 8`).
- **MORI dispatch budget.** `SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK` is `16384` on prefill, sized for a whole chunked-prefill batch, and `128` on decode, where a step never dispatches more than a token per running request.
Picking a role keeps an inherited DSpark head on both roles (see [§3.4](#3-4-dspark-speculative-decoding)); keep `--speculative-dspark-block-size` the same on both roles. For the originals, which have no DSpark head, a speculative PD recipe uses the bundled MTP head through the EAGLE path instead:
Note that DSpark is incompatible with PD disaggregation, so picking a role strips the inherited DSpark flags (see [§3.4](#3-4-dspark-speculative-decoding)). For a speculative PD recipe use the bundled MTP head through the EAGLE path instead:
```bash Command
--speculative-algorithm EAGLE --speculative-eagle-topk 1 \
--speculative-num-steps 3 --speculative-num-draft-tokens 4
```
Run the two roles on separate nodes with the same transport and IB device selection, then front them with the router the card prints. On MI355X that router is cache-aware rather than round-robin: `--policy consistent_hashing` keeps a conversation on the worker that already holds its prefix, which is what makes the agentic reuse pay off, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` keep that affinity from starving the peer given how small the running-request ceiling is. Decode holds no reusable prefix, so `--decode-policy round_robin` spreads the decode side evenly. Health checks stay on with a long timeout so a wedged worker is still caught without a multi-minute prefill being mistaken for a failure.
Run the two roles on separate nodes with the same transport and IB device selection, then front them with the router the card prints. On MI355X that router is cache-aware rather than round-robin: `--policy consistent_hashing` keeps a conversation on the worker that already holds its prefix, which is what makes the agentic reuse pay off, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` keep that affinity from starving the peer given how small the running-request ceiling is. Health checks stay on with a long timeout so a wedged worker is still caught without a multi-minute prefill being mistaken for a failure.
`--dp-aware` only does something when the workers run DP attention — turn on the Playground's **DP-Attention** card if you want it to take effect, otherwise it is inert on the TP-only recipe above.
@@ -1107,23 +1040,4 @@ The **Store** knob picks the linker backend. `mori` is the UMBP pool; `mooncake`
**DP Attention is required**, and the card's Enable chip stays greyed out until you turn it on. This is not a soft preference. The linker keys its objects by rank, and MLA KV is replicated across TP, so a TP8 prefill worker under pure TP opens eight separate keyspaces holding eight copies of the same tokens — the pool's effective distinct-token capacity would be an eighth of what its byte budget suggests, which is easy to mistake for a real measurement. DP attention collapses the keys onto one shared keyspace.
The exception is the MI355X Pro Official prefill role in the low-latency and balanced pairs of [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io). At TP4 it holds four copies rather than eight, and that TP-only shape has run end-to-end, so the card turns UMBP on for those roles without DP. Every Pro Official prefill role (low-latency, balanced, and the DP8 high-throughput one) uses a **standalone** UMBP tier: a separate `umbp_standalone_server` process on the prefill node owns the DRAM pool, and the prefill server attaches to it over a Unix socket. Start the tier server first; the 1500 GB DRAM budget below is what the verified runs used:
```bash Command
mkdir -p /tmp/umbp_sa
UMBP_DRAM_CAPACITY=1500000000000 \
UMBP_DRAM_USE_HUGEPAGES=1 \
UMBP_SSD_ENABLED=0 \
MORI_UMBP_LOG_LEVEL=info \
<mori>/umbp_standalone_server unix:///tmp/umbp_sa/sa.grpc.sock
```
Then launch the prefill role with the socket in its environment. The Playground adds both lines: the address has to come from the environment, and the extra-config gives the tier server up to two minutes to bind the socket before prefill gives up.
```bash Command
UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp_sa/sa.grpc.sock \
sglang serve ... \
--hicache-storage-backend-extra-config '{"standalone_startup_timeout_ms":120000}'
```
Two further limits are worth knowing before you size a run. The pool is a per-node process, so a prefill worker that spans nodes would shard its keyspace by node. And only the prefill worker offloads KV — the decode side is untouched.
@@ -134,7 +134,7 @@ print("Answer:", msg.content)
</Accordion>
On the request, `reasoning_effort` accepts the tiers `low` (50), `high` (75), `xhigh` (75), and `max` (100), or a float in `[0.0, 0.99]` that maps onto the model's 1–100 budget. An integer budget is reachable only through `chat_template_kwargs.reasoning_effort`, and that route does not switch thinking on by itself — pair it with `chat_template_kwargs.thinking`. `none` turns thinking off. The remaining OpenAI tiers, which have no V4.1 counterpart (`minimal`, `medium`), log a warning and fall back to the server-side default rather than erroring; that default is `high` (75), and `SGLANG_DSV41_REASONING_EFFORT` overrides it.
On the request, `reasoning_effort` accepts the tiers `low`, `high`, `xhigh` and `max`, or a float in `[0.0, 0.99]` that maps onto the model's 1–100 budget. An integer budget is reachable only through `chat_template_kwargs.reasoning_effort`, and that route does not switch thinking on by itself — pair it with `chat_template_kwargs.thinking`. `none` turns thinking off. The remaining OpenAI tiers, which have no V4.1 counterpart (`minimal`, `medium`), log a warning and fall back to the server-side default rather than erroring; that default is `high`, and `SGLANG_DSV41_REASONING_EFFORT` overrides it.
### 3.2 Tool Calling
@@ -680,14 +680,12 @@ Document model accuracy on standard benchmarks:
- Benchmark Command:
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 \
--num-threads 1316
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Results:
- ERNIE-4.5-21B-A3B-PT
```
+3 -5
View File
@@ -510,9 +510,7 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
@@ -48,7 +48,8 @@ import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deploymen
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
- **Additional Multimodal Parameters:**
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
- `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency.
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
**Example with full multimodal optimizations:**
```bash Command
+3 -5
View File
@@ -900,13 +900,11 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- Test Result
```text Output
Accuracy: 0.975
@@ -76,7 +76,8 @@ import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deploymen
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
- **Additional Multimodal Parameters:**
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
- `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency.
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
**Example with full multimodal optimizations:**
```bash Command
@@ -388,13 +389,8 @@ print(response)
#### Command
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 ./benchmark/gsm8k/bench_sglang.py
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
#### Result Output
```text Output
Accuracy: 0.925
@@ -920,13 +920,11 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- Result
```text Output
+4 -5
View File
@@ -941,13 +941,12 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319
python -m sglang.test.few_shot_gsm8k \
--num-shots 5 \
--num-questions 1319 \
--port 30000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- Test Result (NVIDIA B200, TP=2 NVFP4)
```text Output
Accuracy: 0.946
+3 -17
View File
@@ -644,14 +644,9 @@ The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not i
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Result
```text Output
Accuracy: 0.955
@@ -664,13 +659,9 @@ Output throughput: 642.044 token/s
- Benchmark Command
```bash Command
sgl-eval run mmlu \
--base-url http://127.0.0.1:30000/v1 \
--num-threads 64 --max-tokens 2048
python3 benchmark/mmlu/bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Result
```text Output
subject: abstract_algebra, #q:100, acc: 0.860
@@ -741,14 +732,9 @@ Average accuracy: 0.877
- MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends)
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
```text Output
Accuracy: 0.970
Invalid: 0.000
+1 -1
View File
@@ -124,7 +124,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
</Note>
- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260924`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions.
- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260920`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions.
## 3. Advanced Usage
+3 -17
View File
@@ -575,14 +575,9 @@ The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Result
```text Output
Accuracy: 0.955
@@ -595,13 +590,9 @@ Output throughput: 642.044 token/s
- Benchmark Command
```bash Command
sgl-eval run mmlu \
--base-url http://127.0.0.1:30000/v1 \
--num-threads 64 --max-tokens 2048
python3 benchmark/mmlu/bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Result
```text Output
subject: abstract_algebra, #q:100, acc: 0.860
@@ -672,14 +663,9 @@ Average accuracy: 0.877
- MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends)
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
```text Output
Accuracy: 0.970
Invalid: 0.000
@@ -815,13 +815,10 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python -m sglang.test.few_shot_gsm8k \
--num-questions 200
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- Test Result
```text Output
@@ -1,232 +0,0 @@
---
title: GigaChat 3.5 Reasoning
description: "Deploy GigaChat 3.5 Reasoning with SGLang — the 432B-A28B hybrid MLA + GDN Mixture-of-Experts thinking model with reasoning parsing, MTP speculative decoding and tool calling, on H100 and H200 in FP8."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
<Tabs>
<Tab title="Python (pip / uv)">
GigaChat 3.5 support is on `main` ([PR #29189](https://github.com/sgl-project/sglang/pull/29189), merged 2026-09-21) and not yet in a `pip` release — install from source:
```bash Command
pip install --upgrade pip
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
```
Then run the **Python** output of the command panel below.
</Tab>
<Tab title="Docker">
<Note>No released `lmsysorg/sglang` tag carries GigaChat 3.5 yet. The nightly `lmsysorg/sglang:dev` is rebuilt from `main` and carries the model in builds after 2026-09-21, but that path has not been run end-to-end, so the command panel shows only the Python run mode. Run the Python command inside the nightly image; the Docker tab returns to the panel after a verified run.</Note>
```bash Command
docker pull lmsysorg/sglang:dev
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
This page covers the **Reasoning** checkpoint (always thinks before answering). For the direct-answer checkpoint, see [GigaChat 3.5](/cookbook/autoregressive/GigaChat/GigaChat3.5).
Pick your hardware and recipe to generate the launch command. Two serving strategies cover the operating points:
- **Low-Latency** — MTP speculative decoding on, Mamba state pool pinned to 80 requests. Fastest reply per user: measured 2× decode speed at batch 1 and 1.5× at 16 concurrent 8k-token requests. Pick for chat and agent loops.
- **High-Throughput** — MTP off, pools left to the default ratio, so the KV pool is ~2.7× larger. Past a few dozen concurrent long-context requests the pinned recipe starts queueing prompts on KV space, and at saturation the draft + verify overhead no longer pays; this is the recipe for batch jobs and many concurrent users.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/ai-sage/gigachat3.5-reasoning.jsx";
import { benchmarks } from "/src/snippets/configs/ai-sage/gigachat3.5-reasoning-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
<Note>
H100 numbers were measured on 8×H100 SXM at `b63f8416b3` (the PR #29189 merge on `main`) with `bench_serving`, random
ISL 8192 / OSL 1024, `--random-range-ratio 1.0`, `--warmup-requests 4`, `--flush-cache`; latencies are
P50. GSM8K was measured with the 5-shot chat harness `sglang.test.run_eval --eval-name gsm8k --api chat` (1314 scored questions,
`max_tokens` 8192), so the model thinks before every answer exactly as in production. That harness was
deprecated upstream after these runs; the **Reproduce** button emits its successor, `sgl-eval run gsm8k`
(single-shot), which can land a point or two away from the numbers shown.
</Note>
## Playground
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been run end-to-end; the Playground lets you turn on additional knobs — the reasoning and tool-call parsers, the MTP preset, parallelism — on top of whichever cell the Deploy panel is currently showing. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model Introduction
**GigaChat 3.5 Reasoning** is the thinking release of GigaChat's hybrid Mixture-of-Experts model — **432B** total parameters, **28B** active per token, **256K** context (YaRN, factor 8 over a 32K base), released under the **MIT** license. It shares the base architecture with [GigaChat 3.5](/cookbook/autoregressive/GigaChat/GigaChat3.5): **40** layers interleaving **10** full-attention **MLA** layers (DeepSeek-style latent attention, every 4th layer) with **30** **GDN** linear-attention layers (Qwen3-Next gated delta-net, constant-size state), and an MoE block routing to **8 of 256** experts plus **1** shared expert. Only the MLA layers hold a KV cache; the linear layers keep their state in a Mamba-style pool.
What differs: the chat template opens every assistant turn with `<think>`, so the model always reasons before answering (there is no template switch to turn it off), and the checkpoint ships **3** stacked **MTP** draft heads instead of 2. Tool calls use the same **GCML** format. The model card reports the largest gains over the instant release in mathematics, code, instruction following and structured output.
**Recommended generation:** the model card suggests `temperature=0.6` with a generous `max_tokens` (thinking alone can run to thousands of tokens). Informational only — the sample code below relies on the checkpoint's `generation_config.json` defaults.
**Resources:** HuggingFace — [GigaChat3.5-432B-A28B-Reasoning](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-Reasoning) (FP8, served here) · [GigaChat3.5-432B-A28B-Reasoning-bf16](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-Reasoning-bf16) (BF16, for fine-tuning; no single-node recipe).
## 2. Configuration Tips
**Two pools.** GDN layers keep per-request state in a Mamba-style pool, MLA layers use a paged KV pool; `--mamba-full-memory-ratio` (default 0.9) splits the post-weight budget. On 8×H100 at `--mem-fraction-static 0.8` that is ~51 GB of weights per GPU, 348 state slots and 542k KV tokens. The resolver forces `--mamba-radix-cache-strategy no_buffer` (3 slots per request, overlap scheduler off), so the unpinned server admits 116 concurrent requests.
**MTP needs a pinned pool and 0.83.** Each running request holds extra state copies under speculative decoding, and the default ratio then admits only a few dozen. The Low-Latency recipe pins `--max-mamba-cache-size 240` with `--max-running-requests 80` and runs at `--mem-fraction-static 0.83` (KV pool 199k tokens); at 0.85 the server ran out of memory in eager prefill at 16 concurrent 8k-token prompts. To size by workload instead of a pin: `r = (3 + draft_tokens) × 16 MB / (11.5 KB × L)` for an average request length `L`, and with MTP keep running 8k-token requests under ~20, since the EAGLE worker holds each prompt's hidden states outside both pools.
**Capacity levers** (toggles in the [Playground](#playground)). `--mamba-ssm-dtype bfloat16` halves every state slot at no cost in accuracy or accept length — on the Low-Latency recipe the KV pool grows from 199k to 496k tokens. `--kv-cache-dtype fp8_e4m3` doubles the KV pool but decodes ~2.3× slower on H100 and cannot be combined with MTP on 80 GB.
**MTP flags.** `--speculative-algorithm EAGLE` uses GigaChat's own NextN heads through the multi-layer EAGLE worker, selected automatically. This checkpoint has **3** heads: `--speculative-num-steps 3`, `--speculative-num-draft-tokens 4`.
**Parsers.** `--reasoning-parser gigachat35` splits thinking from the answer (§3.1), `--tool-call-parser gigachat35` gives structured tool calls (§3.2); `auto` resolves to `gigachat35` for both. Do not use the reasoning parser on the Instant checkpoint.
## 3. Advanced Usage
### 3.1 Reasoning
The chat template opens every assistant turn with `<think>`, and there is no template switch to turn thinking off. Enable the `gigachat35` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split the thinking from the final answer into `reasoning_content` vs `content`. Without the parser, the raw think block is returned inline in `content`.
<Accordion title="Reasoning Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="ai-sage/GigaChat3.5-432B-A28B-Reasoning",
messages=[{"role": "user", "content": "What is 17 times 24?"}],
max_tokens=4096,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Reasoning: I need to calculate 17 times 24. I can do this by breaking it down into smaller parts. First, I know that 10 times 24 is 240, and 7 times 24 is... let me figure that out. 7 times 20 is 140, and 7 times 4 is 28, so 140 plus 28 is 168. Then, I add the two results: 240 plus 168.
240 plus 168: 240 plus 100 is 340, plus 60 is 400, plus 8 is 408. Wait, that doesn't seem right. Let me add them properly. 240 + 168: 0 + 8 is 8, 4 + 6 is 10, carry over the 1, so 2 + 1 is 3, plus the carried over 1 is 4, so it should be 408. But I think I remember that 17 times 24 is 408, but let me double-check.
[... ~1,400 more thinking tokens ...]
Answer:
17 multiplied by 24 equals 408. This can be calculated in several ways:
### Method 1: Breaking Down the Numbers
- $17 \times 24 = 17 \times (20 + 4)$
- First, $17 \times 20 = 340$
- Then, $17 \times 4 = 68$
- Add the results: $340 + 68 = 408$
### Method 2: Using the Distributive Property
- $17 \times 24 = (10 + 7) \times 24$
- $10 \times 24 = 240$
- $7 \times 24 = 168$
- Add the results: $240 + 168 = 408$
[...]
```
The full response was 1,641 completion tokens, 24.6 s at batch size 1 on 8×H100 without MTP.
</Accordion>
<Note>
The model can spend hundreds of tokens thinking, so keep `max_tokens` generous. A response with an empty `content` and `finish_reason: "length"` means the think block was cut off before it closed — raise the limit.
</Note>
### 3.2 Tool Calling
Enable the `gigachat35` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured calls via `message.tool_calls`. GigaChat emits tool calls in its **GCML** format — a `<|GCML|tool_calls>` … `</|GCML|tool_calls>` block holding one `<|GCML|invoke name="…">` per call — inside the assistant turn; the parser reads them out and strips the block from `content`. `--tool-call-parser auto` also resolves to `gigachat35` for this checkpoint, since its chat template carries the GCML marker.
<Accordion title="Tool Calling Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string", "description": "The city name"}},
"required": ["city"],
},
},
}
]
resp = client.chat.completions.create(
model="ai-sage/GigaChat3.5-432B-A28B-Reasoning",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Content:", msg.content)
print("Tool calls:", msg.tool_calls)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Reasoning: Okay, the user is asking about the weather in Paris. Let me check what tools I have available. There's the get_weather function which requires the city name. Since the user provided "Paris" directly, I can use that function. I just need to make sure to format the tool call correctly with the city parameter. No other information is needed here, so I'll call get_weather with city set to Paris.
Content:
Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_ef4e35b959c648fbaeab9afd', function=Function(arguments='{"city": "Paris"}', name='get_weather'), type='function', index=0)]
```
Appending a `tool` message with `{"city": "Paris", "temp_c": 9, "condition": "light rain"}` and re-requesting returns the final answer with no further tool call:
```text Output
Reasoning:
Content: The current weather in Paris is **9°C** with **light rain**.
Tool calls: None
```
</Accordion>
<Note>
The tool-call turn carries the model's thinking in `reasoning_content`, so print it alongside `content` and `tool_calls`. GigaChat tends to call tools **sequentially** — one hop, then the next — so a multi-tool request may come back with a single call. Drive it with an agent loop that appends each `tool` result and re-requests until `finish_reason` is no longer `tool_calls`. Parallel calls (several invokes in one block) are supported by the format when the model chooses them.
</Note>
### 3.3 MTP (Speculative Decoding)
GigaChat 3.5 ships its own NextN draft heads, so speculative decoding needs no separate draft model: `--speculative-algorithm EAGLE` runs the stacked heads through SGLang's multi-layer EAGLE worker, which is selected automatically for this architecture. The **Low-Latency** recipe has it on; in the [Playground](#playground) the **Speculative** card adds the same preset to any other cell.
```bash Flags
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--max-running-requests 80 \
--max-mamba-cache-size 240
```
- **Steps follow the checkpoint.** This checkpoint has 3 draft heads, so use `--speculative-num-steps 3` and `--speculative-num-draft-tokens 4` (steps + 1); the worker rejects other values at startup.
- **Pin the state pool.** Under MTP every running request holds extra GDN state copies for verification. Without `--max-mamba-cache-size` and `--max-running-requests` the resolver clamps concurrency to a few dozen requests; §2 explains what the pin costs in context and why the recipe runs at `--mem-fraction-static 0.83`. The panel shows an amber callout whenever a speculative command lacks `--max-running-requests`.
- **Where it pays off.** Long think blocks are long, low-entropy decode, exactly what draft heads predict well: on 8×H100 single-request decode runs about 2× faster and 16 concurrent 8k-token requests about 1.5× faster, with GSM8K unchanged. At 64 concurrent requests the verify work and the smaller KV pool make it slower than the plain recipe, which is why the High-Throughput recipe leaves MTP off. Measured numbers are in the benchmark card above.
@@ -1,162 +0,0 @@
---
title: GigaChat 3.5
description: "Deploy GigaChat 3.5 with SGLang — the 432B-A28B hybrid MLA + GDN Mixture-of-Experts instant model with MTP speculative decoding and tool calling, on H100 and H200 in FP8."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
<Tabs>
<Tab title="Python (pip / uv)">
GigaChat 3.5 support is on `main` ([PR #29189](https://github.com/sgl-project/sglang/pull/29189), merged 2026-09-21) and not yet in a `pip` release — install from source:
```bash Command
pip install --upgrade pip
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
```
Then run the **Python** output of the command panel below.
</Tab>
<Tab title="Docker">
<Note>No released `lmsysorg/sglang` tag carries GigaChat 3.5 yet. The nightly `lmsysorg/sglang:dev` is rebuilt from `main` and carries the model in builds after 2026-09-21, but that path has not been run end-to-end, so the command panel shows only the Python run mode. Run the Python command inside the nightly image; the Docker tab returns to the panel after a verified run.</Note>
```bash Command
docker pull lmsysorg/sglang:dev
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
This page covers the **instant** checkpoint (direct answers, no thinking). For the thinking checkpoint, see [GigaChat 3.5 Reasoning](/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning).
Pick your hardware and recipe to generate the launch command. Two serving strategies cover the operating points:
- **Low-Latency** — MTP speculative decoding on, Mamba state pool pinned. Fastest reply per user; pick for chat and agent loops.
- **High-Throughput** — MTP off. At saturation the draft + verify overhead outweighs the speedup, so this is the recipe for batch jobs and many concurrent users.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/ai-sage/gigachat3.5.jsx";
import { benchmarks } from "/src/snippets/configs/ai-sage/gigachat3.5-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
## Playground
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been run end-to-end; the Playground lets you turn on additional knobs — the tool-call parser, the MTP preset, parallelism — on top of whichever cell the Deploy panel is currently showing. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model Introduction
**GigaChat 3.5** is a hybrid Mixture-of-Experts model from GigaChat — **432B** total parameters, **28B** active per token, **256K** context (YaRN, factor 8 over a 32K base), released under the **MIT** license. Its **40** layers interleave two attention types: **10** full-attention **MLA** layers (DeepSeek-style latent attention, every 4th layer) and **30** **GDN** linear-attention layers (Qwen3-Next gated delta-net, constant-size state). Only the MLA layers hold a KV cache; the linear layers keep their state in a Mamba-style pool. The MoE block routes to **8 of 256** experts plus **1** shared expert. The checkpoint ships **2** stacked **MTP** draft heads for speculative decoding, and tool calls are emitted in GigaChat's **GCML** format.
This is the **instant** release: it answers directly, without a thinking block. The separately trained **Reasoning** release always thinks first and ships 3 MTP heads — it has [its own page](/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning).
**Resources:** HuggingFace — [GigaChat3.5-432B-A28B](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B) (FP8, served here) · [GigaChat3.5-432B-A28B-bf16](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-bf16) (BF16, for fine-tuning; no single-node recipe).
## 2. Configuration Tips
**Two pools.** GDN layers keep per-request state in a Mamba-style pool, MLA layers use a paged KV pool; `--mamba-full-memory-ratio` (default 0.9) splits the post-weight budget. On 8×H100 at `--mem-fraction-static 0.8` that is ~51 GB of weights per GPU, 347 state slots and 542k KV tokens. The resolver forces `--mamba-radix-cache-strategy no_buffer` (3 slots per request, overlap scheduler off), so the unpinned server admits 115 concurrent requests.
**MTP needs a pinned pool.** Each running request holds extra state copies under speculative decoding, and the default ratio then admits only a few dozen. The Low-Latency recipe pins `--max-mamba-cache-size 240` with `--max-running-requests 80`; the KV pool drops to 185k tokens. Keep `--mem-fraction-static 0.8` on 80 GB — it is already tight in eager prefill at 16 concurrent 8k-token prompts.
**Capacity levers** (toggles in the [Playground](#playground)). `--mamba-ssm-dtype bfloat16` halves every state slot at no cost in accuracy or speed. `--kv-cache-dtype fp8_e4m3` doubles the KV pool but decodes ~2.3× slower on H100 and cannot be combined with MTP on 80 GB.
**MTP flags.** `--speculative-algorithm EAGLE` uses GigaChat's own NextN heads through the multi-layer EAGLE worker, selected automatically. This checkpoint has **2** heads: `--speculative-num-steps 2`, `--speculative-num-draft-tokens 3`.
**Parsers.** `--tool-call-parser gigachat35` for structured tool calls (§3.1). No reasoning parser, not even `auto`: the checkpoint never emits `</think>`, and the forced `gigachat35` splitter would move the whole answer into `reasoning_content`.
## 3. Advanced Usage
### 3.1 Tool Calling
Enable the `gigachat35` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured calls via `message.tool_calls`. GigaChat emits tool calls in its **GCML** format — a `<|GCML|tool_calls>` … `</|GCML|tool_calls>` block holding one `<|GCML|invoke name="…">` per call — inside the assistant turn; the parser reads them out and strips the block from `content`. `--tool-call-parser auto` also resolves to `gigachat35` for this checkpoint, since its chat template carries the GCML marker.
<Accordion title="Tool Calling Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string", "description": "The city name"}},
"required": ["city"],
},
},
}
]
resp = client.chat.completions.create(
model="ai-sage/GigaChat3.5-432B-A28B",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=tools,
)
msg = resp.choices[0].message
print("Content:", msg.content)
print("Tool calls:", msg.tool_calls)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Content:
Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_aa689732d4884654a1f76762', function=Function(arguments='{"city": "Paris"}', name='get_weather'), type='function', index=0)]
```
Appending a `tool` message with `{"city": "Paris", "temp_c": 9, "condition": "light rain"}` and re-requesting returns the final answer with no further tool call:
```text Output
Content: The current weather in Paris is light rain with a temperature of 9°C.
Tool calls: None
```
</Accordion>
<Note>
GigaChat tends to call tools **sequentially** — one hop, then the next — so a multi-tool request may come back with a single call. Drive it with an agent loop that appends each `tool` result and re-requests until `finish_reason` is no longer `tool_calls`. Parallel calls (several invokes in one block) are supported by the format when the model chooses them.
</Note>
### 3.2 MTP (Speculative Decoding)
GigaChat 3.5 ships its own NextN draft heads, so speculative decoding needs no separate draft model: `--speculative-algorithm EAGLE` runs the stacked heads through SGLang's multi-layer EAGLE worker, which is selected automatically for this architecture. The **Low-Latency** recipe has it on; in the [Playground](#playground) the **Speculative** card adds the same preset to any other cell.
```bash Flags
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--max-running-requests 80 \
--max-mamba-cache-size 240
```
- **Steps follow the checkpoint.** This checkpoint has 2 draft heads, so use `--speculative-num-steps 2` and `--speculative-num-draft-tokens 3` (steps + 1); the worker rejects other values at startup.
- **Pin the state pool.** Under MTP every running request holds extra GDN state copies for verification. Without `--max-mamba-cache-size` and `--max-running-requests` the resolver clamps concurrency to a few dozen requests; §2 explains what the pin costs in context. The panel shows an amber callout whenever a speculative command lacks `--max-running-requests`.
- **Where it pays off.** Low concurrency: accepted draft tokens skip target forwards, so single-request decode runs about 2× faster with the output distribution unchanged. At high concurrency the verify work and the smaller KV pool eat the gain, which is why the High-Throughput recipe leaves MTP off. Measured numbers are in the benchmark card above.
@@ -1443,7 +1443,7 @@ Median ITL (ms): 15.11
</tbody>
</table>
> **Note**: These GSM8K numbers use the raw few-shot completion harness (now removed). `gemma-4-12B-it` is reasoning-oriented and is under-elicited by raw few-shot prompting; with the chat template it scores **0.950** on the same 1319 GSM8K test questions (also measured before the sgl-eval migration).
> **Note**: These GSM8K numbers use the raw few-shot completion harness (`sglang.test.few_shot_gsm8k`). `gemma-4-12B-it` is reasoning-oriented and is under-elicited by raw few-shot prompting; with the chat template it scores **0.950** on the same 1319 GSM8K test questions (`sglang.test.run_eval --eval-name gsm8k`).
#### gemma-4-12B-it with sgl-eval
@@ -1467,8 +1467,6 @@ sgl-eval run gsm8k --base-url http://localhost:30000/v1
sgl-eval run mmlu --base-url http://localhost:30000/v1 --num-examples 2000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
#### MMMU
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
@@ -1645,7 +1643,7 @@ The command above uses sgl-eval. Historical scores from other harnesses are not
**gemma-4-E2B-it**
```shell Command
$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E2B-it ....
$ python -m sglang.bench_one_batch --correct --model google/gemma-4-E2B-it ....
prefill logits (final): tensor([[-25.3063, -2.5718, -10.3674, ..., -25.3779, -25.5181, -25.2337]],
device='cuda:0')
....
@@ -1660,7 +1658,7 @@ prefill logits (final) tensor([-25.3281, -2.1367, -10.2266, ..., -25.4375, -25
**gemma-4-E4B-it**
```shell Command
$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E4B-it ....
$ python -m sglang.bench_one_batch --correct --model google/gemma-4-E4B-it ....
prefill logits (final): tensor([[-17.6478, 7.9901, -5.6505, ..., -17.5658, -17.6478, -17.7293]],
device='cuda:0')
....
@@ -1674,7 +1672,7 @@ prefill logits (final) tensor([-17.5625, 8.0469, -5.5742, ..., -17.4688, -17
**gemma-4-31B-it**
```shell Command
$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-31B-it ....
$ python -m sglang.bench_one_batch --correct --model google/gemma-4-31B-it ....
prefill logits (final): tensor([[-2.0748, 1.1245, -7.4356, ..., -2.1059, -2.1525, -2.2303]],
device='cuda:0')
....
@@ -1,47 +0,0 @@
---
title: IQuest-Q1
description: "Deploy IQuest-Q1 with SGLang — BF16 recipes for the 320B Mixture-of-Experts model on 8× NVIDIA H200, with optional MTP speculative decoding."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
IQuest-Q1 needs an SGLang build that includes IQuest-Q1 support. The nightly image carries it:
```bash Command
docker pull lmsysorg/sglang:dev
```
For other methods, see the [official SGLang installation guide](../../../docs/get-started/install).
</Accordion>
Download the checkpoint first; the MTP draft lives in its `mtp/` subdirectory:
```bash Command
hf download IQuestLab/IQuest-Q1 --local-dir /model/IQuest-Q1
```
Pick a recipe to generate the launch command. **Low-Latency** enables MTP speculative decoding; **High-Throughput** serves the target model alone.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/IQuestLab/iquest-q1.jsx";
<Deployment config={config} />
## 1. Model Introduction
**IQuest-Q1** is a Mixture-of-Experts language model from IQuestLab with about 320B total parameters and 15B active per token. It has 88 layers that interleave three sliding-window attention layers (4,096-token window) with one full-attention layer, uses learned attention sinks, and routes each token to 8 of 256 experts. It ships with a single-layer MTP draft that SGLang applies recursively for speculative decoding.
**Resources:** [HuggingFace](https://huggingface.co/IQuestLab/IQuest-Q1).
## 2. Configuration Tips
- **Hardware.** IQuest-Q1 requires the FA3 attention backend for both the target and the MTP draft, because the attention sink is applied from the FA3 softmax LSE. SGLang builds FA3 for Hopper, so Blackwell GPUs are not supported. The BF16 weights take about 640 GB, which fits 8× H200 with tensor parallelism 8.
- **Reasoning.** Thinking is on by default. Pass `"chat_template_kwargs": {"thinking": false}` to turn it off for a request. The `iquest_q1` reasoning parser returns the thinking in `reasoning_content`.
- **Tool calling.** The `iquest_q1` tool-call parser supports `tool_choice` values `auto`, `required`, and a named function.
- **MTP depth.** The single draft layer runs 5 sequential steps, one token each, and each round verifies 6 tokens. Keep `--speculative-num-draft-tokens` equal to `--speculative-num-steps` + 1.
@@ -685,13 +685,11 @@ Max ITL (ms): 176.78
#### 5.2.1 GSM8K Benchmark
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:8000/v1 \
--num-examples 200
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 8000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
**Results:**
```text Output
@@ -205,14 +205,9 @@ Output:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Result
```text Output
Accuracy: 0.960
@@ -213,14 +213,9 @@ For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_us
Reference run on a single GB300 node with `--tp 4`:
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
```text Output
Accuracy: 0.9621 (1269 / 1319)
```
@@ -250,18 +250,9 @@ sglang serve \
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64 \
--temperature 1.2 \
--top-p 0.8 \
--max-tokens 32768 \
--chat-template-kwarg enable_thinking=true
python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Result
```text Output
Accuracy: 0.955
@@ -337,8 +337,10 @@ Mean ITL (ms): 47.77
- Benchmark Command:
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:${PORT}/v1 \
python3 -m sglang.test.run_eval \
--eval-name gsm8k \
--host 127.0.0.1 \
--port ${PORT} \
--model auto \
--num-examples 200 \
--num-threads 64 \
@@ -346,8 +348,6 @@ sgl-eval run gsm8k \
--reasoning-effort high
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- Test Results (B200 x8):
```text Output
@@ -635,13 +635,9 @@ Max ITL (ms): 1970.03
- **Benchmark Command:**
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- **Results**:
```text Output
+17 -25
View File
@@ -488,12 +488,8 @@ Max ITL (ms): 268.17
- **Benchmark Command:**
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- Llama-4-Scout-17B-16E-Instruct
```text Output
Accuracy: 0.945
@@ -509,9 +505,9 @@ Latency: 9.739 s
Output throughput: 2405.505 token/s
```
#### 5.3.2 MMLU-Pro with sgl-eval
#### 5.3.2 MMLU Pro with lm-eval
The table below records historical lm-eval measurements on 8×H100 alongside [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)). These scores were not produced by the sgl-eval commands below; rerun the evaluation for results using sgl-eval's prompts and grading.
Accuracy on MMLU Pro matches [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) on 8×H100 (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)):
<table style={{width: "100%", borderCollapse: "collapse"}}>
<thead>
@@ -535,12 +531,6 @@ The table below records historical lm-eval measurements on 8×H100 alongside [Me
</tbody>
</table>
Install the evaluation package before running either command:
```bash
pip install sgl-eval==0.1.0
```
**Scout:**
```bash Command
@@ -552,12 +542,13 @@ python -m sglang.launch_server \
--mem-fraction-static 0.8 \
--context-length 65536
# Run MMLU-Pro
sgl-eval run mmlu_pro \
--base-url http://localhost:30000/v1 \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--max-tokens 2048 \
--num-threads 128
# Run lm_eval
lm_eval --model local-chat-completions \
--model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
--tasks mmlu_pro \
--batch_size 128 \
--apply_chat_template \
--num_fewshot 0
```
**Maverick:**
@@ -571,10 +562,11 @@ python -m sglang.launch_server \
--mem-fraction-static 0.8 \
--context-length 65536
# Run MMLU-Pro
sgl-eval run mmlu_pro \
--base-url http://localhost:30000/v1 \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--max-tokens 2048 \
--num-threads 128
# Run lm_eval
lm_eval --model local-chat-completions \
--model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
--tasks mmlu_pro \
--batch_size 128 \
--apply_chat_template \
--num_fewshot 0
```
@@ -960,13 +960,8 @@ Max ITL (ms): 376.34
#### 5.2.1 GSM8K Benchmark
- Benchmark Command:
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python benchmark/gsm8k/bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Results:
```text Output
Accuracy: 0.950
@@ -977,13 +972,10 @@ Output throughput: 1130.161 token/s
#### 5.2.2 MMLU Benchmark
- Benchmark Command:
```shell Command
sgl-eval run mmlu \
--base-url http://127.0.0.1:30000/v1 \
--num-threads 64 --max-tokens 2048
cd benchmark/mmlu
bash download_data.sh
python3 bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
- Test Results:
```text Output
subject: abstract_algebra, #q:100, acc: 0.620
@@ -601,12 +601,8 @@ sglang serve \
- **Benchmark Command**:
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
- **Result**:
- MiniMax-M2
```text Output
@@ -501,14 +501,13 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:8888/v1 \
--num-examples 1319 \
--num-threads 1316
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316 \
--port 8888
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
**Test Results:**
```text Output
@@ -272,14 +272,12 @@ Document model accuracy on standard benchmarks:
- Benchmark Command
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 \
--num-threads 1316
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
**Test Results:**
```text Output
@@ -336,14 +336,9 @@ Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions`
#### GSM8K
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
**Results:**
```text Output
@@ -333,14 +333,9 @@ shapes to create a distinct symbol which signifies copyright protection.
#### GSM8K
```bash Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 200 \
--num-threads 64
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.
**Results:**
```text Output
@@ -350,9 +345,7 @@ TODO
#### MMLU
```bash Command
sgl-eval run mmlu \
--base-url http://127.0.0.1:30000/v1 \
--num-threads 64 --max-tokens 2048
python3 benchmark/mmlu/bench_sglang.py --port 30000
```
**Results:**
@@ -597,17 +597,15 @@ The following results are from the official Kimi-K2.7-Code model card. They were
#### 5.2.1 Accuracy Benchmark
Evaluate GSM8K on the deployed MXFP4 checkpoint across client parallelism levels (all 1319 test questions via sgl-eval):
GSM8K accuracy of the MXFP4 checkpoint, measured on the deployed server across several client parallelism levels (2,000 questions each):
```shell Command
sgl-eval run gsm8k \
--base-url http://127.0.0.1:30000/v1 \
--num-examples 1319 \
--num-threads '<PARALLEL>'
python3 benchmark/gsm8k/bench_sglang.py \
--num-questions 2000 \
--parallel <PARALLEL> \
--port 30000
```
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
<table>
<thead>
<tr>

Some files were not shown because too many files have changed in this diff Show More