Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4b20bf4ea2 |
@@ -1,774 +0,0 @@
|
||||
---
|
||||
name: add-jit-kernel
|
||||
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang.kernels JIT infrastructure and public operator groups
|
||||
---
|
||||
|
||||
# Tutorial: Adding a New JIT Kernel to SGLang
|
||||
|
||||
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||
The implementation tutorial below does not replace that API contract.
|
||||
|
||||
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||
|
||||
## Goal
|
||||
|
||||
Add a new operation that scales each element of a tensor by a scalar factor:
|
||||
|
||||
- Input: tensor `x` (CUDA) and scalar `factor` (float, passed at runtime)
|
||||
- Output: `x * factor` (element-wise), allocated internally
|
||||
- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
|
||||
|
||||
## When to use JIT vs AOT (`sgl-kernel`)
|
||||
|
||||
- **JIT (`jit_kernel`)**: prefer this first for kernels that do **not** depend on CUTLASS or another large C++ project. It is the default choice for lightweight kernels that benefit from rapid iteration and first-use compilation.
|
||||
- **AOT (`sgl-kernel`)**: prefer this when the kernel **does** depend on CUTLASS or another large C++ project, or when it should live in `python/sglang/kernels/aot/` and participate in the wheel build / torch op registration flow.
|
||||
- **Exception**: kernels that depend on `flashinfer`, or on CUTLASS that is already provided through `flashinfer`, can still be implemented as `jit_kernel`.
|
||||
|
||||
---
|
||||
|
||||
## Conventions
|
||||
|
||||
These hold for every step below.
|
||||
|
||||
- **`namespace sglang` is where JIT code lives.** Open it after the include block and close it at the end of the file, with the device kernels, traits and host wrapper inside. The shared `host::` / `device::` helpers are nested in it too, so they resolve unqualified. `load_jit` emits the `TVM_FFI_DLL_EXPORT_TYPED_FUNC` wrapper inside `namespace sglang` as well, so the `kernel_name` you pass from Python needs no `sglang::` prefix.
|
||||
- **Tuning policy belongs in Python; C++ gets the answer, not the decision.** For a hyperparameter — split factor, threads per item, block size, vector width, an algorithm variant — prefer (not required) making it a **template parameter** and letting the `@cache_once` module factory choose the value, over a runtime `if`/`switch` in the launcher that picks among pre-instantiated kernels. The kernel then compiles exactly one specialisation and asserts its own preconditions with `static_assert`, while the heuristic that produced the value sits in Python where it is readable, adjustable, and inspectable without recompiling CUDA. `store_cache`'s `num_threads` is the worked example: a `get_kernel(num_split)` ladder guarded by byte-alignment `if constexpr`s became one template argument plus a heuristic in `_jit_kvcache_module`. The exception is a knob that genuinely varies per call with the runtime shape — that has to stay a kernel argument.
|
||||
- **Check where the check is cheapest: `static_assert` > C++ host check > cached Python > per-call Python.** Anything fixed at compile time is a `static_assert`. Anything about the tensors is a `TensorMatcher` / `CHECK_HOST` in the C++ launcher, free next to a kernel launch. A check Python cannot delegate goes inside the `@cache_once` module factory, where it runs once per specialisation. What remains in the per-call entry point costs interpreter time on *every* forward, so it should be nothing but picking the module and allocating `out`.
|
||||
- **Fixed-width integer types.** Prefer `int32_t` / `int64_t` / `uint32_t` / `size_t` over `int`, `long`, or `long long`, so an index has the same width on both sides of the FFI boundary. Bare `int` is fine only where the width plainly cannot matter — an unrolled loop counter over a `constexpr` bound, a template `int` parameter. Shapes arrive as `int64_t` (`SymbolicSize::unwrap()`); narrowing to `uint32_t` for in-kernel indexing is a deliberate act, so write the `static_cast` explicitly and only where the range is known.
|
||||
- **Doxygen comments in C++.** Document exported entities with `///` or `/** ... */` blocks using `\brief`, `\param`, `\tparam`, `\return`, the way `include/sgl_kernel/` does. `python -m sglang.kernels.jit` writes `CommentFormat: Doxygen` into `.clangd` when clangd is 21 or newer, so these render on hover in the editor. Plain `//` remains fine for implementation notes inside a function body.
|
||||
- **ASCII only in C++ and CUDA sources.** Write `--`, `->`, `<=` instead of `—`, `→`, `≤`, including in comments. `grep -nP '[^\x00-\x7F]' <file>` before committing.
|
||||
- **`namespace details` is private.** Anything inside one is an implementation detail of its own header, free to change without notice. Do not name `details::` from another module, and never `using namespace details`. If you find yourself wanting something in there, that is the signal to promote it to a documented name instead.
|
||||
- **`const T* __restrict__` for read-only pointers.** This is what `csrc/` does throughout, and it lets the compiler emit non-coherent (`LDG`) loads.
|
||||
- **Watch the register budget.** For memory-bound kernels, keep to roughly 64 registers per thread so occupancy does not become the limit; prefer recomputing a value over letting it spill. Run `SGLANG_JIT_LOG_RESOURCE_USAGE=1 SGLANG_JIT_FORCE_RECOMPILE=1` to log per-kernel registers, spills and shared memory at INFO. Both halves are load-bearing: a cache hit has no compiler output to report, and passing `-Xptxas -v` by hand does nothing on its own because the build captures compiler output and replays it only on failure.
|
||||
- **Alignment is a contract you enforce, not a property you hope for.** A vectorized copy derives its access width from the row *size* (`load_bytes` picks 16B for a 1024B row), but the address it lands on is `base + index * row_stride` — and a stride is not constrained by the size. `TensorMatcher` admits any stride unless you say otherwise, so a padded cache row silently produces `misaligned address`. Every kernel that vectorizes is highly recommended to end its validation with `.ensure_alignment(w)`, where `w` is the width the kernel actually uses.
|
||||
|
||||
---
|
||||
|
||||
## Common Abstractions in `python/sglang/kernels/jit/include/sgl_kernel/`
|
||||
|
||||
**Always prefer these abstractions over raw CUDA primitives.** They provide safety, readability, and consistency with the rest of the codebase. The only reason to drop to raw primitives is performance the abstraction cannot reach — a trade you make deliberately, and justify in a comment.
|
||||
|
||||
### `utils.h` — Host-side utilities
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/utils.h>
|
||||
```
|
||||
|
||||
- **`CHECK_HOST(cond) << "msg " << value`** — **Preferred** runtime check: stream-style, throws `PanicError` with file/line info on failure. Zero overhead on the true path — the message expressions are only evaluated when the check fails.
|
||||
- **`host::RuntimeCheck(cond, args...)`** — Function-style alternative to `CHECK_HOST`. Note its message args are always evaluated (even when the check passes), so prefer `CHECK_HOST` — especially on hot paths.
|
||||
- **`host::Panic(args...)`** — Unconditionally throw a `PanicError` with a descriptive message.
|
||||
- **`host::div_ceil(a, b)`** — Integer ceiling division `(a + b - 1) / b`.
|
||||
- **`host::irange(n)`** / **`host::irange(start, end)`** — Range views for cleaner loops.
|
||||
- **`host::pointer::offset(ptr, offsets...)`** — Byte-safe pointer arithmetic on `void*`. Use this instead of raw casts.
|
||||
|
||||
### `utils.cuh` — Device-side utilities + `LaunchKernel`
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/utils.cuh>
|
||||
```
|
||||
|
||||
- **Type aliases**: `fp16_t`, `bf16_t`, `fp32_t`, `fp8_e4m3_t`, `fp8_e5m2_t` and their packed variants `fp16x2_t`, `bf16x2_t`, `fp32x2_t`, etc.
|
||||
- **`SGL_DEVICE`** — Expands to `__forceinline__ __device__`. Use on all device functions.
|
||||
- **`SGL_DEVICE_HOST`** — `__forceinline__ __device__ __host__`. Use it when a `constexpr` helper must be callable from both the kernel and its launcher, so the two cannot drift.
|
||||
- **`device::kWarpThreads`** — Constant `32` (the codebase's *logical* warp width). `kWarpSize` is an alias. Note this is a group size, not the hardware wave: on gfx950 `warpSize` is 64, and `warp::kFullWidth` reflects that.
|
||||
- **`device::get_lane_id<kNumThreads = kWarpThreads>()`** — this thread's index within its logical group. **Prefer it over `threadIdx.x % kNumThreads` whenever the value feeds an address.** On CUDA the lane register is one read that folds into the address computation, whereas the modulo is a derived value the compiler re-materialises at *every* address scale — 8 instructions and 2 registers on a two-tile warp copy, measured on sm_100a. It picks the cheaper form per platform, so callers never need to care. Only equals the true in-warp lane when `blockDim.x` is a multiple of `kNumThreads`.
|
||||
- **`device::load_as<T>(ptr, offset)`** / **`device::store_as<T>(ptr, val, offset)`** — Type-safe loads/stores from `void*`.
|
||||
- **`device::pointer::offset(ptr, offsets...)`** — Pointer arithmetic on device.
|
||||
- **`host::LaunchKernel(grid, block, device_or_stream [, smem])`** — RAII kernel launcher that:
|
||||
- Resolves the CUDA stream from a `DLDevice` via TVM-FFI automatically.
|
||||
- Checks the CUDA error with file/line info after launch via `operator()(kernel, args...)`.
|
||||
- Supports `.enable_pdl(bool)` for PDL (Programmatic Dependent Launch, SM90+).
|
||||
- **`device::PDLWaitPrimary<kUsePDL>()`** / **`device::PDLTriggerSecondary<kUsePDL>()`** — The two halves of PDL, on sm_90+ (no-ops on older archs and ROCm). Their guarantees are **not** symmetric:
|
||||
- `PDLTriggerSecondary` (`griddepcontrol.launch_dependents`) only lets the next kernel in the stream *start* early. It carries no memory ordering and publishes nothing — matching that, the header's asm has no `"memory"` clobber.
|
||||
- `PDLWaitPrimary` (`griddepcontrol.wait`) is the ordering point: it waits until the preceding kernel has fully finished and its writes are visible.
|
||||
|
||||
So every read of data the preceding kernel produced must come after `PDLWaitPrimary()`. What overlaps with the primary's tail is whatever you put *before* the wait — loading parameters, computing indices, touching buffers the primary never wrote — so a kernel that waits on its first line gains nothing. Neither call is a barrier: threads may reach or skip them independently. See "Programmatic Dependent Launch and Synchronization" in the CUDA C++ Programming Guide.
|
||||
- **`CHECK_CUDA(expr) << "context"`** — Stream-style CUDA error check; evaluates `expr` once and throws `PanicError` with `cudaGetErrorString` + file/line info if it is not `cudaSuccess`. Extra streamed context is optional.
|
||||
- **`host::RuntimeDeviceCheck(cudaError_t)`** — Function-style alternative to `CHECK_CUDA`. It takes no context message, so prefer `CHECK_CUDA`, which builds its error object only on the failure path.
|
||||
|
||||
### `tensor.h` — Tensor validation (`TensorMatcher`, Symbolic types)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/tensor.h>
|
||||
```
|
||||
|
||||
This is the **primary validation API** for all kernel launchers. Use it to validate every `tvm::ffi::TensorView` argument.
|
||||
|
||||
- **`host::SymbolicSize{"name"}`** — A named symbolic dimension. Call `.set_value(n)` to pin it, `.unwrap()` to extract after verification.
|
||||
- **`host::SymbolicDType`** — Symbolic dtype. Use `.set_options<Ts...>()` to restrict allowed types.
|
||||
- **`host::SymbolicDevice`** — Symbolic device. Use `.set_options<kDLCUDA>()` to restrict to CUDA.
|
||||
- **`host::TensorMatcher({dims...})`** — Fluent builder for tensor validation:
|
||||
- `.with_dtype<T>()` — require a specific C++ type (e.g. `fp16_t`)
|
||||
- `.with_dtype<T1, T2, ...>()` — allow a set of types
|
||||
- `.with_device<kDLCUDA>(device_sym)` — require CUDA and bind the checked device to a `SymbolicDevice`
|
||||
- `.with_strides({strides...})` — validate strides (omit to require contiguous)
|
||||
- `.ensure_alignment(bytes)` — require the data pointer **and every stride except the innermost** to be a multiple of `bytes` (a power of two). Size-1 dimensions are skipped, since their strides are arbitrary. This is how a vectorized kernel states its precondition; see the alignment convention above.
|
||||
- `.verify(tensor_view)` — execute the check; throws `PanicError` with full context on failure; **chainable** (`verify(a).verify(b)` to check multiple tensors with the same shape)
|
||||
- **`host::is_type<T>(dtype)`** — whether a `DLDataType` denotes the C++ type `T` (e.g. `fp16_t`).
|
||||
|
||||
**Typical pattern:**
|
||||
```cpp
|
||||
auto N = SymbolicSize{"num_elements"};
|
||||
auto device = SymbolicDevice{};
|
||||
device.set_options<kDLCUDA>();
|
||||
TensorMatcher({N}) //
|
||||
.with_dtype<fp16_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.ensure_alignment(16) // e.g. for 128-bit vectorized load/store
|
||||
.verify(dst)
|
||||
.verify(src); // same shape, dtype, device as dst
|
||||
const int64_t n = N.unwrap();
|
||||
const DLDevice dev = device.unwrap();
|
||||
const int64_t last_dim = 128;
|
||||
TensorMatcher({N, last_dim}) // a fixed dimension can be a plain integer
|
||||
.with_dtype<fp16_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(tensor_2d);
|
||||
```
|
||||
|
||||
### `ffi.h` — Tensor allocation and blob wrapping (`host::ffi::`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/ffi.h>
|
||||
```
|
||||
|
||||
The counterpart to `tensor.h`: that one validates what came in, this one produces new `tvm::ffi::Tensor` values. Allocation goes through the environment allocator (`TVMFFIEnvTensorAlloc`), so buffers come from PyTorch's caching allocator rather than a raw `cudaMalloc`.
|
||||
|
||||
- **`host::alloc_workspace_tensor(nbytes, device)`** (declared in `utils.cuh`) — **the way to get scratch memory**: a 1-D `uint8` tensor of `nbytes`, or an empty tensor when `nbytes == 0`. Hold the returned `Tensor` in a local across every launch that touches it — it frees on destruction.
|
||||
- **`host::ffi::empty(shape, dtype, device)`** — Uninitialized tensor; `shape` accepts a braced list, so `ffi::empty({rows, sizeof(Plan)}, dtype, device)` works for a typed scratch array.
|
||||
- **`host::ffi::empty_like(tensor_view)`** — Same shape, dtype, and device as an existing tensor.
|
||||
- **`host::ffi::from_blob(data, shape, dtype, device[, deleter, stride, byte_offset])`** / **`from_blob_like(data, tensor_view, ...)`** — View memory you already own as a `Tensor`, no copy. The default deleter does nothing, so ownership stays with the caller; pass one only when the `Tensor` should own the block. Strides default to contiguous.
|
||||
|
||||
### `type.cuh` — `DTypeTrait<T>`, `packed_t<T>`, and reduction traits
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/type.cuh>
|
||||
```
|
||||
|
||||
- **`DTypeTrait<T>`** — Static trait struct, specialized for integral types, `fp32_t`, `fp16_t`, `bf16_t`, `fp8_e4m3_t`, and their packed x2/x4 variants. Provides:
|
||||
- `DTypeTrait<T>::from(value)` — convert from another type via the right CUDA intrinsic (e.g. `fp32_t` → `fp16_t`)
|
||||
- `DTypeTrait<T>::abs/max/min` — type-dispatched math (fp32, fp16/bf16 scalar and x2, integrals)
|
||||
- `DTypeTrait<T>::sqrt/rsqrt/exp/sin/cos(x)` — `fp32_t` only
|
||||
- Metadata: `packed_t` / `unpacked_t` / `kVecSize` (packed layout), `kFloatMax` (dtype max as float, e.g. 448.0f for fp8-e4m3), `kZeroBits`
|
||||
- **`packed_t<T>`** — Two-element packed alias: `packed_t<fp16_t>` = `fp16x2_t`, `packed_t<bf16_t>` = `bf16x2_t`, `packed_t<fp32_t>` = `fp32x2_t`. Use for vectorized loads/stores.
|
||||
- **`device::cast<To, From>(value)`** — Type-safe cast using `DTypeTrait`, e.g. `cast<fp32x2_t, fp16x2_t>(v)`.
|
||||
- **`device::unpack(value)`** — View a packed value as an `unpacked_t[kVecSize]` array reference (e.g. `fp32x2_t` → `fp32_t[2]`); element writes propagate back to the packed value.
|
||||
- **`device::ReductionOp` (`SUM`/`MAX`/`MIN`) and `device::ReductionTrait<Op, T>::reduce(x, y)`** — One binary reduction step, dispatched through `DTypeTrait` (packed types reduce elementwise). This is the engine behind `warp::reduce`; use it directly when writing custom reductions.
|
||||
|
||||
### `vec.cuh` — Vectorized memory access (`AlignedVector`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/vec.cuh>
|
||||
```
|
||||
|
||||
- **`device::AlignedVector<T, N>`** — Aligned storage for N elements of type T. N must be a power of two, `sizeof(T)*N <= 32`. Enables vectorized loads/stores for bandwidth efficiency. In terms of API/codegen constraints, the upper bound is 256-bit; in practice, 128-bit is the portable default, while 256-bit vectorization is typically only viable on `SM100+` and should be gated by an architecture check when needed.
|
||||
- `.load(ptr, offset)` — vectorized load from `ptr[offset]`
|
||||
- `.store(ptr, offset)` — vectorized store to `ptr[offset]`
|
||||
- `.fill(value)` — fill all N elements with `value`
|
||||
- `operator[](i)` — element access
|
||||
- **`device::LoadStoreBytes`** — named widths to pass around instead of bare integers: `MAX_GMEM` (arch-dependent, 32 on Blackwell), `MAX_SMEM` (16), `MAX_PORTABLE` (16, safe on CUDA and HIP), `MIN_COALALESCED` (4), and `RAW_1B` .. `RAW_32B`.
|
||||
|
||||
### `tile.cuh` — `tile::Memory` (strided memory access pattern)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/tile.cuh>
|
||||
```
|
||||
|
||||
- `tile::Memory<T>` is fundamentally a **1D cooperative accessor** over a contiguous region.
|
||||
- **`device::tile::Memory<T>::cta(blockDim.x)`** — Creates a tile accessor where each thread handles `tid = threadIdx.x` with stride `tsize` (for `cta(blockDim.x)`, this is `blockDim.x`). Common for loops over a 1D array.
|
||||
- **`device::tile::Memory<T>::warp()`** — the 32-lane flavour, and what `warp::load_bytes` is built on. It takes its lane index from `get_lane_id()` rather than `threadIdx.x % 32`, for the addressing reason described under `utils.cuh`. Use `warp(int n)` for a narrower sub-group; that overload keeps the modulo, since a hardware lane index is the wrong grouping below 32.
|
||||
- **`device::tile::Memory<T>::thread()`** — single-thread accessor, no cooperation.
|
||||
- **`.load(ptr, offset)`** — loads `ptr[tid + offset * tsize]`
|
||||
- **`.store(ptr, val, offset)`** — stores to `ptr[tid + offset * tsize]`
|
||||
- **`.in_bound(n, offset)`** — boundary check
|
||||
|
||||
For a **2D tile**, either flatten `(row, col)` into a linear tile index first, or compute the address manually with `ptr[row * stride + col]` using your thread/block coordinates.
|
||||
|
||||
### `bits.h` — Compile-time bit helpers (`host::`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/bits.h>
|
||||
```
|
||||
|
||||
`constexpr` wrappers over `<bit>`, usable in `static_assert` and in template arguments: `host::is_pow2(x)`, `log2_floor(x)`, `log2_ceil(x)` (both `-1` for `x == 0`), `round_up_pow2(x)`, `round_down_pow2(x)`. They live in `host::` but are constant-expression-only, so device-side `static_assert` can call them.
|
||||
|
||||
### `math.cuh` — Device math (`device::math::`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/math.cuh>
|
||||
```
|
||||
|
||||
- `device::math::max/min<T>(a, b)` — type-dispatched binary math via `DTypeTrait`
|
||||
- `device::math::abs/sqrt/rsqrt/exp/sin/cos<T>(x)` — type-dispatched unary math via `DTypeTrait`
|
||||
|
||||
### `warp.cuh` — Warp-level primitives
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/warp.cuh>
|
||||
```
|
||||
|
||||
**Vectorized row copy — reach for this before hand-rolling.** A warp cooperatively moving one contiguous row is the single most common shape in this tree, and it used to be re-implemented per kernel.
|
||||
|
||||
- **`warp::load_bytes<kBytes, kPattern>(src)`** / **`warp::store_bytes<kBytes, kPattern>(dst, val)`** — the whole warp moves `kBytes` from/to one contiguous row. `kBytes` is arbitrary, down to 1; the helper picks the widest vector that divides it and handles the ragged tail. The returned value is opaque and encodes the width, so a load and its matching store **must use the same `<kBytes, kPattern>`**.
|
||||
- **`warp::LoadStorePattern`** — the `kPattern` values, and the sign carries the meaning. A **negative** one (`WARP_UNIFORM_16B` and friends: `_GMEM`, `_SMEM`, `_4B`/`_8B`/`_32B`) says *the whole warp splits this row*, so the width is derived from the per-lane share (`gcd(kBytes / 32, |kPattern|)`), falling back to `gcd(kBytes, 4)` when the row is too narrow to give every lane 4 bytes. A **positive** one (a bare `LoadStoreBytes` value) makes no warp-splitting assumption and just caps the per-thread vector at `gcd(kBytes, kPattern)`. `WARP_UNIFORM_16B` is the usual choice for a row copy.
|
||||
- **`LoadStorePattern::get_vec_bytes<kBytes, kPattern>()`** — the width actually chosen. `SGL_DEVICE_HOST`, so **the launcher must call this to feed `.ensure_alignment(...)`**. Pass it the same value the kernel passes `load_bytes` — for a split row that is the *per-warp share*, not the whole row; the full row can resolve to a different width and would under-constrain the strides.
|
||||
|
||||
**Reductions.**
|
||||
|
||||
- `warp::reduce<Op, kStart, kFinish>(value, active_mask)` — `Op` is a `device::ReductionOp` (`SUM`/`MAX`/`MIN`). `kStart` and `kFinish` bound a range of lane-index bits: `<N, 1>` reduces within contiguous groups of `N` lanes, `<kFullWidth, N>` reduces *across* groups at the same offset. The operation is **symmetric** — `<A, B>` and `<B, A>` reduce the same lane set.
|
||||
- `warp::reduce_sum/reduce_max/reduce_min<kStart, kFinish>(value)` — wrappers. Work for any type with a `ReductionTrait`: floats, integers, packed x2. This is more widely used in practice than generic `reduce`.
|
||||
- `warp::inclusive_reduce<Op, kWidth, kStart, kFinish>(val, lane_id, mask)` and `inclusive_sum` / `inclusive_max` / `inclusive_min` — segmented inclusive scan: every lane keeps its own running total, unlike `reduce`. Forward when `kStart < kFinish`, backward when `kStart > kFinish`. `lane_id` defaults to `get_lane_id<kWidth>()`, which is correct by construction — only pass it if you already have the **segment-relative** index (`threadIdx.x % kWidth`); a warp-relative one silently corrupts every segment past the first when `kWidth < 32`.
|
||||
- `warp::broadcast<kWidth>(value, src_lane)` — `src_lane` is **segment-relative**: each `kWidth` segment reads its own lane, not one warp-wide source.
|
||||
- `warp::get_lane_id`, `warp::elect_one_lane()` (one elected lane, for gating a single-thread TMA issue; CUDA-only), `warp::kFullMask` and `warp::kFullWidth` (32 on CUDA, 64 on HIP — note this differs from `kWarpThreads`).
|
||||
|
||||
### `cta.cuh` — CTA-level primitives
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/cta.cuh>
|
||||
```
|
||||
|
||||
- `device::cta::reduce_max<T>(value, smem, min_value)` — CTA-wide max using shared memory + warp reduction. Caller is responsible for a `__syncthreads()` after if the result in `smem[0]` is needed.
|
||||
|
||||
### `atomic.cuh` — Atomic operations
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/atomic.cuh>
|
||||
```
|
||||
|
||||
- `device::atomic::max(float* addr, float value)` — float atomic max (handles negative values correctly via bit tricks).
|
||||
- **`device::atomic::Event`** — a cross-CTA arrive/wait counter in one 32-bit word. Producers call `arrive()`; consumers call `wait(num_producers)` (exactly one consumer) or `wait_multi<kConsumerBits>(num_producers, num_consumers, n)` (several). The last released consumer resets the word, so one `Event` is reusable across launches with no host re-zero. Preconditions, all of them load-bearing:
|
||||
- the word must be **zeroed by the host** before first use, and must live in **global** memory — a shared or local `Event` is an illegal address;
|
||||
- `arrive()` is a release and the waits are acquires, so a producer's writes before `arrive()` are visible after the wait;
|
||||
- **generations must be explicitly ordered** — the word has no phase bit, so overlapping generation N+1 with a consumer still in generation N is undefined behaviour (a lagging consumer never exits);
|
||||
- `wait()` and `wait_multi()` lay the word out incompatibly; mixing them on one `Event` is undefined behaviour.
|
||||
|
||||
### `runtime.cuh` — Occupancy and device info
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/runtime.cuh>
|
||||
```
|
||||
|
||||
- `host::runtime::get_blocks_per_sm(kernel, block_dim)` — max active blocks per SM (occupancy)
|
||||
- `host::runtime::get_sm_count(device_id [, use_cache])` — number of SMs on the device. Memoized per device ordinal, so it is cheap enough to call on a launch path; pass `use_cache=false` to force a driver query.
|
||||
- `host::runtime::get_cc_major` / `get_cc_minor` / `get_sm_version` — compute capability, same caching.
|
||||
|
||||
**Do not query the architecture at runtime.** The JIT compiles for the exact local GPU, so the arch is a *compile-time* fact: `SGL_CUDA_ARCH` is injected by `load_jit`, and `SGL_ARCH_HOPPER_OR_GREATER` / `SGL_ARCH_BLACKWELL_OR_GREATER` / `device::kMaxVecBytes` are derived from it. Reach for `get_cc_*` only for something the arch genuinely does not determine. SM *count* is the opposite case — it varies within an arch (B200 vs B300 vs a MIG slice), so it has to stay a runtime query.
|
||||
|
||||
**Persistent kernel pattern** (cap blocks to SM count × occupancy):
|
||||
```cpp
|
||||
static const uint32_t max_occ = runtime::get_blocks_per_sm(kernel, kBlockSize);
|
||||
static const uint32_t num_sm = runtime::get_sm_count(device.unwrap().device_id);
|
||||
const auto num_blocks = std::min(num_sm * max_occ, div_ceil(n, kBlockSize));
|
||||
LaunchKernel(num_blocks, kBlockSize, device.unwrap())(kernel, params);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 0 (optional): Generate a `.clangd` config for better IDE support
|
||||
|
||||
```bash
|
||||
python -m sglang.kernels.jit -h # for verbose help info about clangd configuration
|
||||
python -m sglang.kernels.jit
|
||||
python -m sglang.kernels.jit --dep cutlass flashinfer # with cutlass/flashinfer dependency
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Implement the CUDA kernel in `kernels/jit/csrc/`
|
||||
|
||||
Create `python/sglang/kernels/jit/csrc/elementwise/scale.cuh`.
|
||||
|
||||
The implementation fully uses the project abstractions described above:
|
||||
|
||||
```cpp
|
||||
// NOTE: Comments for headers are not common in practice.
|
||||
// It is only shown here for tutorial purposes to highlight the key abstractions.
|
||||
#include <sgl_kernel/tensor.h> // For TensorMatcher, SymbolicSize, SymbolicDevice
|
||||
#include <sgl_kernel/type.cuh> // For DTypeTrait, fp16_t, bf16_t, fp32_t
|
||||
#include <sgl_kernel/utils.h> // For CHECK_HOST, div_ceil
|
||||
#include <sgl_kernel/utils.cuh> // For LaunchKernel, SGL_DEVICE
|
||||
#include <sgl_kernel/vec.cuh> // For AlignedVector
|
||||
|
||||
#include <dlpack/dlpack.h>
|
||||
#include <tvm/ffi/container/tensor.h>
|
||||
|
||||
namespace sglang {
|
||||
|
||||
/**
|
||||
* \brief Element-wise scale using vectorized 128-bit loads/stores.
|
||||
*
|
||||
* \tparam T Element type: fp16_t | bf16_t | fp32_t
|
||||
* \tparam kVecN Elements per vector load (e.g. 8 for fp16)
|
||||
* \tparam kUsePDL Whether to emit the PDL wait/trigger pair
|
||||
* \param dst Output buffer, `n_total` elements
|
||||
* \param src Input buffer, `n_total` elements
|
||||
* \param factor Runtime scale factor
|
||||
* \param n_total Number of elements to scale
|
||||
*/
|
||||
template <typename T, int kVecN, bool kUsePDL>
|
||||
__global__ void scale_kernel(T* __restrict__ dst,
|
||||
const T* __restrict__ src,
|
||||
float factor,
|
||||
uint32_t n_total) {
|
||||
using vec_t = device::AlignedVector<T, kVecN>;
|
||||
const uint32_t n_vecs = n_total / kVecN;
|
||||
|
||||
// If using PDL, wait for primary kernel before any global memory load.
|
||||
// This is NOT a synchronization point, which means some threads can early exit before this.
|
||||
device::PDLWaitPrimary<kUsePDL>();
|
||||
|
||||
// --- vectorised body ---
|
||||
const uint32_t vec_stride = blockDim.x * gridDim.x;
|
||||
for (uint32_t vi = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
vi < n_vecs;
|
||||
vi += vec_stride) {
|
||||
vec_t v;
|
||||
v.load(src, vi);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kVecN; ++i) {
|
||||
v[i] = static_cast<T>(static_cast<float>(v[i]) * factor);
|
||||
}
|
||||
v.store(dst, vi);
|
||||
}
|
||||
|
||||
// --- scalar tail ---
|
||||
const uint32_t base = n_vecs * kVecN;
|
||||
const uint32_t scalar_stride = blockDim.x * gridDim.x;
|
||||
for (uint32_t i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
base + i < n_total;
|
||||
i += scalar_stride) {
|
||||
dst[base + i] = static_cast<T>(static_cast<float>(src[base + i]) * factor);
|
||||
}
|
||||
|
||||
// If using PDL, signal for the secondary kernel to start after all threads have finished
|
||||
// This is NOT a synchronization point, which means some threads can early exit before this.
|
||||
device::PDLTriggerSecondary<kUsePDL>();
|
||||
}
|
||||
|
||||
/**
|
||||
* \brief Validate the tensors, select the vector width, launch `scale_kernel`.
|
||||
*
|
||||
* \tparam T Element type: fp16_t | bf16_t | fp32_t
|
||||
* \tparam kUsePDL Whether to launch with PDL enabled
|
||||
* \param dst Output tensor; same shape / dtype / device as `src`
|
||||
* \param src Input tensor on CUDA
|
||||
* \param factor Runtime scale factor
|
||||
*/
|
||||
template <typename T, bool kUsePDL>
|
||||
void scale(tvm::ffi::TensorView dst, tvm::ffi::TensorView src, float factor) {
|
||||
using namespace host;
|
||||
|
||||
// 1. Validate input tensors with TensorMatcher
|
||||
SymbolicSize N = {"num_elements"};
|
||||
SymbolicDevice device_;
|
||||
device_.set_options<kDLCUDA>();
|
||||
|
||||
TensorMatcher({N}) //
|
||||
.with_dtype<T>()
|
||||
.with_device<kDLCUDA>(device_)
|
||||
.verify(dst)
|
||||
.verify(src); // same shape / dtype / device as dst
|
||||
|
||||
const uint32_t n = static_cast<uint32_t>(N.unwrap());
|
||||
const DLDevice device = device_.unwrap();
|
||||
|
||||
CHECK_HOST(n > 0) << "scale: num_elements must be > 0, got " << n;
|
||||
|
||||
// 2. Choose vector width for 128-bit loads (16 bytes)
|
||||
// fp16/bf16: 8 elements x 2 bytes = 16 bytes
|
||||
// fp32: 4 elements x 4 bytes = 16 bytes
|
||||
// We encourage using `device::kMaxVecBytes`, which will change according to
|
||||
// the target architecture and can enable 256-bit vectorization on SM100+ if desired.
|
||||
// But 128-bit is more commonly adapted for better compatibility,
|
||||
// so it's still ok to hardcode 16 here just for simplicity.
|
||||
constexpr int kVecN = 16 / sizeof(T);
|
||||
const uint32_t n_work_items = div_ceil(n, static_cast<uint32_t>(kVecN));
|
||||
|
||||
// 3. Launch
|
||||
constexpr uint32_t kBlockSize = 256;
|
||||
const uint32_t grid = div_ceil(n_work_items, kBlockSize);
|
||||
|
||||
// PDL feature is 100% optional. Without `enable_pdl`, the code should still be correct.
|
||||
// Try to enable it if profiling shows that it can benefit the performance of this kernel.
|
||||
LaunchKernel(grid, kBlockSize, device).enable_pdl(kUsePDL)(
|
||||
scale_kernel<T, kVecN, kUsePDL>,
|
||||
static_cast<T*>(dst.data_ptr()),
|
||||
static_cast<const T*>(src.data_ptr()),
|
||||
factor,
|
||||
n);
|
||||
}
|
||||
|
||||
} // namespace sglang
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Include headers from `sgl_kernel/` — **not** raw CUDA headers for anything already covered
|
||||
- Use `TensorMatcher` for all tensor validation; never manually check shape/dtype/device
|
||||
- Use `AlignedVector` for vectorised 128-bit loads/stores — significant bandwidth win
|
||||
- Use `LaunchKernel` — it resolves the stream and checks errors automatically
|
||||
- Use `CHECK_HOST(cond) << ...` for runtime assertions with useful error messages (zero overhead when the check passes)
|
||||
- Prefer passing runtime scalars like `factor` directly unless compile-time specialisation is genuinely required
|
||||
- `fp16_t` / `bf16_t` / `fp32_t` are the project's type aliases (from `utils.cuh`)
|
||||
- `device::cast<To, From>` or `DTypeTrait<T>::from(val)` for cross-type conversions
|
||||
- `device::math::` functions for device math instead of bare `__` intrinsics if possible.
|
||||
- Consider PDL — it can help when the kernel has prologue work to overlap. Place `PDLWaitPrimary()` right before the first read of upstream data, not at the top of the kernel
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Add the Python wrapper in `kernels/ops/`
|
||||
|
||||
The wrapper lives next to its functional group under `python/sglang/kernels/ops/`, not beside the CUDA source — `kernels/jit/` holds only the JIT infrastructure (`csrc/`, `include/`, `utils/`, `benchmark/`). Create `python/sglang/kernels/ops/elementwise/scale.py`:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.utils import (
|
||||
cache_once,
|
||||
is_arch_support_pdl,
|
||||
load_jit,
|
||||
make_cpp_args,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tvm_ffi.module import Module
|
||||
|
||||
|
||||
@cache_once
|
||||
def _jit_scale_module(dtype: torch.dtype) -> Module:
|
||||
"""Compile and cache the JIT scale module for a given dtype."""
|
||||
# Checks on the compile key live here, not in `scale`: `cache_once` keys on
|
||||
# `dtype`, so this runs once per specialisation instead of once per call.
|
||||
if dtype not in (torch.float16, torch.bfloat16, torch.float32):
|
||||
raise RuntimeError(
|
||||
f"Unsupported dtype {dtype}. Supported: float16, bfloat16, float32"
|
||||
)
|
||||
args = make_cpp_args(dtype, is_arch_support_pdl())
|
||||
return load_jit(
|
||||
"scale",
|
||||
*args,
|
||||
cuda_files=["elementwise/scale.cuh"],
|
||||
cuda_wrappers=[("scale", f"scale<{args}>")],
|
||||
)
|
||||
|
||||
|
||||
def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) -> torch.Tensor:
|
||||
"""
|
||||
Element-wise scale: dst = src * factor.
|
||||
|
||||
Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
src : CUDA tensor (FP16 / BF16 / FP32)
|
||||
factor : scale factor
|
||||
out : optional pre-allocated output tensor (same shape/dtype as src)
|
||||
|
||||
Returns
|
||||
-------
|
||||
Scaled tensor (dst = src * factor).
|
||||
"""
|
||||
# DO NOT add proactive validation here: every check costs interpreter time
|
||||
# on a per-forward path. Tensor invariants belong in the C++ launcher, and
|
||||
# anything about the compile key belongs in `_jit_scale_module`.
|
||||
if out is None:
|
||||
out = torch.empty_like(src)
|
||||
|
||||
module = _jit_scale_module(src.dtype)
|
||||
module.scale(out, src, factor)
|
||||
return out
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Use `cache_once` — **not** `functools.lru_cache` (incompatible with `torch.compile`)
|
||||
- `load_jit` first arg(s) form the unique build marker; same marker = same cached binary
|
||||
- Only include compile-time specialisation knobs in the build marker; runtime values like `factor` should stay runtime unless the kernel truly needs templating
|
||||
- `cuda_wrappers`: `(export_name, kernel_symbol)` — `export_name` is called from Python
|
||||
- `make_cpp_args(dtype, ...)` converts `torch.dtype` to C++ type alias:
|
||||
- `is_arch_support_pdl()` checks if the current architecture supports PDL, which is typically passed as a template argument to the kernel.
|
||||
- Keep the entry point thin (see Conventions). What Python must still check goes in the `@cache_once` module factory, not in the entry point: `cache_once` keys on its arguments, so a check there costs one evaluation per specialisation instead of one per call — that is where the supported-dtype guard lives. Tensor invariants belong in the C++ launcher; if something here never reaches a `.verify(...)`, close that gap on the C++ side rather than in Python
|
||||
|
||||
| `torch.dtype` | C++ type |
|
||||
|--------------------|------------|
|
||||
| `torch.float16` | `fp16_t` |
|
||||
| `torch.bfloat16` | `bf16_t` |
|
||||
| `torch.float32` | `fp32_t` |
|
||||
|
||||
---
|
||||
|
||||
## Register the public entry point (required)
|
||||
|
||||
Add metadata in `python/sglang/kernels/ops/elementwise/__init__.py` so the
|
||||
operator appears in the registry without importing its implementation:
|
||||
|
||||
```python
|
||||
from sglang.kernels.registry import register_kernel
|
||||
from sglang.kernels.spec import CapabilityRequirement, KernelBackend, KernelSpec
|
||||
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="elementwise.scale",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.elementwise.scale:scale",
|
||||
capabilities=frozenset({CapabilityRequirement.CUDA}),
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
Record the devices the implementation actually supports; the CUDA-only scale
|
||||
example above does not imply that every JIT kernel is CUDA-only. Runtime and
|
||||
correctness tests import `scale` from `sglang.kernels.ops.elementwise.scale`.
|
||||
Do not import the implementation eagerly in the group initializer. When an
|
||||
existing `BaseFusedOp` owns the operation, add the implementation there instead
|
||||
of registering a second conflicting op/backend pair.
|
||||
|
||||
## Step 3 (optional): Tune JIT build flags
|
||||
|
||||
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
|
||||
|
||||
```python
|
||||
return load_jit(
|
||||
"scale",
|
||||
*args,
|
||||
cuda_files=["elementwise/scale.cuh"],
|
||||
cuda_wrappers=[("scale", f"scale<{args}>")],
|
||||
extra_cuda_cflags=["-O3", "--use_fast_math"],
|
||||
)
|
||||
```
|
||||
|
||||
If your kernel requires SM90+, raise a clear Python error before calling `load_jit`. Arch gating is one of the checks that has to live in Python — it decides whether to compile at all, so the C++ launcher never gets to run:
|
||||
|
||||
```python
|
||||
if torch.cuda.get_device_capability()[0] < 9:
|
||||
raise RuntimeError("This kernel requires SM90 (Hopper) or later")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Write tests (required)
|
||||
|
||||
JIT kernel correctness tests and benchmarks live under `test/registered/kernels/ops/<group>/` and `test/registered/kernels/benchmark/<group>/`, mirroring the wrapper's group under `python/sglang/kernels/ops/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/kernels/jit/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
|
||||
|
||||
- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`.
|
||||
- **Nightly kernel suite**: register with `stage="nightly"` plus the `runner_config` of the machine it needs (e.g. `1-gpu-large`), giving the `nightly-test-1-gpu-large` suite. `.github/workflows/nightly-test-nvidia.yml` sets `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` for the whole nightly run, so the expanded parameter grids apply automatically (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). There is no separate kernel-only nightly job: every nightly test on one machine type shares that machine's suite.
|
||||
|
||||
Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing):
|
||||
|
||||
```python
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
# Optional B200/SM100 registration for tests that cover Blackwell-specific code paths
|
||||
# register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
|
||||
# Optional second registration: same file also runs nightly, same form,
|
||||
# stage is just "nightly" there (and no `nightly=True`)
|
||||
# register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
|
||||
```
|
||||
|
||||
CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. Nightly uses the same shape with `stage="nightly"`; the single-string `suite=` form is left only for `stress` and non-CUDA pools.
|
||||
|
||||
Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery.
|
||||
|
||||
Use `register_cuda_ci(..., disabled="reason")` if the file must stay in-tree but should be skipped in CI (e.g. multi-GPU only).
|
||||
|
||||
**Run like CI** (from repo root):
|
||||
|
||||
```bash
|
||||
(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-1-gpu-large)
|
||||
# For B200/SM100-specific coverage:
|
||||
(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-4-gpu-b200)
|
||||
```
|
||||
|
||||
For fast iteration you can still run `pytest` on a single file locally; CI coverage is via `run_suite.py`.
|
||||
|
||||
Create `test/registered/kernels/ops/elementwise/test_scale.py`:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
import torch
|
||||
from sglang.kernels.ops.elementwise.scale import scale
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||
@pytest.mark.parametrize("size", [1, 127, 128, 1024, 4097]) # cover tail remainder
|
||||
@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0, 3.0])
|
||||
def test_scale_correctness(dtype, size, factor):
|
||||
src = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = scale(src, factor)
|
||||
expected = src * factor
|
||||
|
||||
rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
|
||||
torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||
def test_scale_out_param(dtype):
|
||||
src = torch.randn(1024, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(src)
|
||||
result = scale(src, 2.0, out=out)
|
||||
assert result is out
|
||||
torch.testing.assert_close(out, src * 2.0, rtol=1e-2, atol=1e-2)
|
||||
|
||||
|
||||
def test_scale_cpu_error():
|
||||
src = torch.randn(128, dtype=torch.float16) # CPU tensor
|
||||
with pytest.raises(RuntimeError, match="CUDA"):
|
||||
scale(src, 2.0)
|
||||
|
||||
|
||||
def test_scale_unsupported_dtype():
|
||||
src = torch.randint(0, 10, (128,), dtype=torch.int32, device="cuda")
|
||||
with pytest.raises(RuntimeError, match="dtype"):
|
||||
scale(src, 2.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(pytest.main([__file__, "-v", "-s"]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Add a benchmark (required)
|
||||
|
||||
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
|
||||
|
||||
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
|
||||
|
||||
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
|
||||
|
||||
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
|
||||
- **`@marker.parametrize(names, vals, ci_vals=None)`** — stackable decorator that adds a row axis (pytest-style). Each `@parametrize` adds one (or more, correlated) parameter the benchmark is swept over (Cartesian product across all `parametrize` decorators). `names` may be a single name (`"size"`) or a comma-separated correlated tuple axis (`"h,d"`, with `vals` then a list of tuples like `[(1, 64), (2, 128)]`). Pass the optional third `ci_vals` for a smaller sweep that is auto-selected under `is_in_ci()` — this is the built-in CI-shrinking mechanism, so you usually don't need `get_benchmark_range` for swept axes.
|
||||
- **`marker.do_bench(fn, *, input_args=(), input_kwargs={}, ...)`** — runs `fn` under CUDA graph (default) or a naive loop, returns a `BenchResult`. Key knobs:
|
||||
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
|
||||
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
|
||||
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
|
||||
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"`; if you narrow it, keep every read, in-place, and output tensor.
|
||||
- `use_cuda_graph=False` for kernels that can't be captured.
|
||||
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
|
||||
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
|
||||
- **`utils.create_random(*shape)` / `utils.create_empty(*shape)`** — shorthand for `torch.randn` / `torch.empty` with `DEFAULT_DTYPE` (`bfloat16`) and `DEFAULT_DEVICE` (`"cuda"`). Override via the `dtype=` / `device=` kwargs.
|
||||
- **`utils.get_benchmark_range(full_range, ci_range)`** — returns the smaller `ci_range` under CI (`is_in_ci()`), the `full_range` locally. Still available for the `benchmark(...)` column axis (which has no `ci_vals`); for `parametrize` row axes prefer the built-in `ci_vals` argument.
|
||||
|
||||
Create `test/registered/kernels/benchmark/elementwise/bench_scale.py`:
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.benchmark import marker
|
||||
from sglang.kernels.jit.benchmark.utils import create_random
|
||||
from sglang.kernels.ops.elementwise.scale import scale as jit_scale
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
|
||||
|
||||
|
||||
@torch.compile()
|
||||
def torch_impl_scale(src: torch.Tensor, factor: float) -> torch.Tensor:
|
||||
return src * factor
|
||||
|
||||
|
||||
FN_MAP = {
|
||||
"jit": jit_scale,
|
||||
"torch": torch_impl_scale,
|
||||
}
|
||||
|
||||
|
||||
# `parametrize(name, full_vals, ci_vals)`: the 3rd arg is the smaller sweep
|
||||
# auto-selected under CI; the full range runs locally.
|
||||
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096, 65536]) # 1K .. 512K
|
||||
@marker.benchmark("impl", ["jit", "torch"])
|
||||
def benchmark(size: int, impl: str):
|
||||
src = create_random(size)
|
||||
factor = 2.0
|
||||
return marker.do_bench(
|
||||
FN_MAP[impl],
|
||||
input_args=(src, factor),
|
||||
# `src` is read -> clone it per iter to avoid L2 reuse; factor is a scalar.
|
||||
graph_clone_args=(0,),
|
||||
# Defaults already report bandwidth: memory_args="all" counts src,
|
||||
# memory_output="out" counts the returned tensor -> bytes(src)+bytes(out).
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
benchmark.run()
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- The `line_arg` name passed to `benchmark` (`"impl"` here) must match a parameter on `benchmark(...)`; same for every `parametrize` name (`"size"`).
|
||||
- Stack `@parametrize` once per swept axis. The required `@marker.benchmark` is the **innermost** decorator (bottom of the stack, directly above the function) — `@parametrize` rows go above it.
|
||||
- Prefer `create_random` / `create_empty` from `utils.py` over open-coding `torch.randn(..., dtype=..., device=...)`.
|
||||
- The GB/s column appears by default (`memory_args="all"` + `memory_output="out"`). For memory-bound kernels it's the most informative number; scope `memory_args` / `memory_output` to the tensors actually touched if the defaults over- or under-count. For compute-bound kernels where bandwidth is misleading, set `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1` (or `disable_log_bandwidth=True`).
|
||||
- For in-place kernels (which return `None`), pass the written tensors via `memory_output=(...)` since the `"out"` default would capture nothing.
|
||||
- Tune `graph_clone_args` / `graph_clone_kwargs` to all the arguments that might be read by the kernel. We can only skip cloning for write-only args. For in-place modified args, we still need to clone them to get accurate timing (reusing the same buffer keeps it L2-hot and skews results).
|
||||
- Call `benchmark.run()` (no `print_data=` kwarg — the marker framework prints directly).
|
||||
|
||||
Run locally:
|
||||
|
||||
```bash
|
||||
python test/registered/kernels/benchmark/elementwise/bench_scale.py
|
||||
```
|
||||
|
||||
Run the benchmark suite the way CI does:
|
||||
|
||||
```bash
|
||||
cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config`; starred args and non-literal values break AST collection
|
||||
- **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations
|
||||
- **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...`
|
||||
- **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way.
|
||||
- **Missing GB/s column**: the column is on by default; check that `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH` is not `1` and `disable_log_bandwidth` is not `True`. For in-place kernels (return `None`) the `memory_output="out"` default counts nothing — pass the written tensors via `memory_output=(...)`
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `docs/docs/developer_guide/development_jit_kernel_guide.mdx`
|
||||
- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
|
||||
- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
|
||||
- `python/sglang/kernels/jit/utils/compile/` — `load_jit` (`loader.py`), `make_cpp_args` (`cpp_args.py`)
|
||||
- `python/sglang/kernels/jit/utils/common.py` — `cache_once`, `should_run_full_tests`, `get_ci_test_range`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`, `is_type`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/ffi.h` — `ffi::empty`, `ffi::empty_like`, `ffi::from_blob`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/utils.cuh` — type aliases, `LaunchKernel`, `SGL_DEVICE`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/vec.cuh` — `AlignedVector`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/tile.cuh` — `tile::Memory`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/bits.h` — compile-time bit helpers
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/type.cuh` — `DTypeTrait`, `packed_t`, `device::cast`, `device::unpack`, `ReductionTrait`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/math.cuh` — `device::math::`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/warp.cuh` — `warp::reduce<Op>` and `reduce_sum/max/min` wrappers
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh` — minimal runnable reference
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
|
||||
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
|
||||
- `python/sglang/kernels/jit/benchmark/utils.py` — `create_random` / `create_empty` / `get_benchmark_range` helpers and `DEFAULT_DTYPE` / `DEFAULT_DEVICE`
|
||||
- `test/registered/kernels/benchmark/layernorm/bench_qknorm.py` — real example: multi-axis `parametrize` (with `ci_vals`) + in-place `memory_output`
|
||||
- `test/registered/kernels/benchmark/kvcache/bench_store_cache.py` — real example: scoped `memory_args` / `memory_output` + selective `graph_clone_args`
|
||||
|
||||
## Summary of Files Created
|
||||
|
||||
```
|
||||
python/sglang/kernels/jit/csrc/elementwise/scale.cuh # NEW: CUDA kernel
|
||||
python/sglang/kernels/ops/elementwise/scale.py # NEW: Python wrapper
|
||||
test/registered/kernels/ops/elementwise/test_scale.py # NEW: Tests
|
||||
test/registered/kernels/benchmark/elementwise/bench_scale.py # NEW: Benchmark
|
||||
```
|
||||
@@ -1,370 +0,0 @@
|
||||
---
|
||||
name: add-sgl-kernel
|
||||
description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel to sgl-kernel (including tests & benchmarks)
|
||||
---
|
||||
|
||||
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
|
||||
|
||||
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||
The implementation tutorial below does not replace that API contract.
|
||||
|
||||
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||
|
||||
## Goal
|
||||
|
||||
Add a new operation that scales each element of a tensor by a scalar factor:
|
||||
|
||||
- Input: tensor `x` (CUDA) and scalar `factor` (float)
|
||||
- Output: `x * factor` (element-wise, in-place or into pre-allocated `out`)
|
||||
- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
|
||||
- Dispatched via `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro (defined in `python/sglang/kernels/aot/include/utils.h`)
|
||||
|
||||
## Two rules of thumb (must follow)
|
||||
|
||||
1. **Prefer `python/sglang/kernels/jit` first** when the kernel does **not** depend on CUTLASS or another large C++ project. This is the default path for lightweight kernels that benefit from rapid iteration.
|
||||
2. **Prefer `sgl-kernel`** when the kernel **does** depend on CUTLASS or another large C++ project, or when it should be part of the AOT wheel / torch op registration flow.
|
||||
3. **Exception**: if the dependency is `flashinfer`, or CUTLASS that is already provided through `flashinfer`, the kernel can still be implemented as `jit_kernel`.
|
||||
|
||||
In addition, every new kernel must ship with:
|
||||
|
||||
- **Tests** (pytest)
|
||||
- **A benchmark script** (`marker.do_bench` from `sglang.kernels.jit.benchmark` — it works for any callable, not only JIT kernels)
|
||||
|
||||
---
|
||||
|
||||
## Repository integration map
|
||||
|
||||
You will typically touch these files/areas:
|
||||
|
||||
- Implementation: `python/sglang/kernels/aot/csrc/elementwise/scale.cu` (pick the right subdirectory)
|
||||
- Public declarations: `python/sglang/kernels/aot/include/sgl_kernel_ops.h`
|
||||
- Torch extension registration: `python/sglang/kernels/aot/csrc/common_extension.cc`
|
||||
- Build: `python/sglang/kernels/aot/CMakeLists.txt` (`set(SOURCES ...)`)
|
||||
- Python API: `python/sglang/kernels/aot/python/sgl_kernel/` and `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||
- Tests: `python/sglang/kernels/aot/tests/test_scale.py`
|
||||
- Benchmarks: `python/sglang/kernels/aot/benchmark/bench_scale.py`
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Implement the kernel in `csrc/`
|
||||
|
||||
Pick the right subdirectory:
|
||||
|
||||
- `csrc/elementwise/` — for element-wise ops (our example)
|
||||
- `csrc/gemm/`, `csrc/attention/`, `csrc/moe/` — for other categories
|
||||
|
||||
Create `python/sglang/kernels/aot/csrc/elementwise/scale.cu`:
|
||||
|
||||
```cpp
|
||||
#include <ATen/cuda/CUDAContext.h>
|
||||
#include <c10/cuda/CUDAGuard.h>
|
||||
#include <torch/all.h>
|
||||
|
||||
#include "utils.h" // DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16
|
||||
|
||||
// scale_kernel: out[i] = input[i] * factor
|
||||
// Supports float, half (__half), __nv_bfloat16 via template T
|
||||
template <typename T>
|
||||
__global__ void scale_kernel(T* __restrict__ out,
|
||||
const T* __restrict__ input,
|
||||
float factor,
|
||||
int64_t n) {
|
||||
int64_t idx = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
|
||||
if (idx < n) {
|
||||
out[idx] = static_cast<T>(static_cast<float>(input[idx]) * factor);
|
||||
}
|
||||
}
|
||||
|
||||
void scale(at::Tensor& out, const at::Tensor& input, double factor) {
|
||||
TORCH_CHECK(input.is_cuda(), "input must be a CUDA tensor");
|
||||
TORCH_CHECK(input.is_contiguous(), "input must be contiguous");
|
||||
TORCH_CHECK(out.is_cuda(), "out must be a CUDA tensor");
|
||||
TORCH_CHECK(out.is_contiguous(), "out must be contiguous");
|
||||
TORCH_CHECK(out.sizes() == input.sizes(), "out and input must have the same shape");
|
||||
TORCH_CHECK(out.scalar_type() == input.scalar_type(),
|
||||
"out and input must have the same dtype");
|
||||
|
||||
const int64_t n = input.numel();
|
||||
const int threads = 256;
|
||||
const int blocks = (n + threads - 1) / threads;
|
||||
|
||||
const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
|
||||
const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
|
||||
|
||||
// Dispatches over float, float16, bfloat16
|
||||
DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16(input.scalar_type(), c_type, [&] {
|
||||
scale_kernel<c_type><<<blocks, threads, 0, stream>>>(
|
||||
static_cast<c_type*>(out.data_ptr()),
|
||||
static_cast<const c_type*>(input.data_ptr()),
|
||||
static_cast<float>(factor),
|
||||
n);
|
||||
cudaError_t status = cudaGetLastError();
|
||||
TORCH_CHECK(status == cudaSuccess,
|
||||
"scale_kernel launch failed: ", cudaGetErrorString(status));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Use `at::Tensor` (PyTorch tensors), `TORCH_CHECK` for validation, `at::cuda::getCurrentCUDAStream()` for stream
|
||||
- Keep Python wrappers thin; do shape/dtype/device validation in C++ right around the launch path
|
||||
- `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` covers `float`, `half` (FP16), `__nv_bfloat16` (BF16)
|
||||
- Add device error checking after every kernel launch
|
||||
- If a kernel only works on certain architectures, enforce that with `TORCH_CHECK` and skip logic in tests
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Add a C++ declaration in `include/sgl_kernel_ops.h`
|
||||
|
||||
Edit `python/sglang/kernels/aot/include/sgl_kernel_ops.h`, add to the elementwise section:
|
||||
|
||||
```cpp
|
||||
void scale(at::Tensor& out, const at::Tensor& input, double factor);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Register the op in `csrc/common_extension.cc`
|
||||
|
||||
Edit `python/sglang/kernels/aot/csrc/common_extension.cc`, inside `TORCH_LIBRARY_FRAGMENT(sgl_kernel, m)`:
|
||||
|
||||
```cpp
|
||||
// From csrc/elementwise
|
||||
m.def("scale(Tensor! out, Tensor input, float factor) -> ()");
|
||||
m.impl("scale", torch::kCUDA, &scale);
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- `Tensor!` means in-place / mutable output argument
|
||||
- The schema is important for `torch.compile` and for consistent call signatures
|
||||
- Keep the torch schema in PyTorch scalar types (`float` here), but note that the C++ launcher signature still needs `double` for scalar arguments accepted by `torch::Library`
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Add the new source file to `CMakeLists.txt`
|
||||
|
||||
Edit `python/sglang/kernels/aot/CMakeLists.txt`, add to `set(SOURCES ...)`:
|
||||
|
||||
```cmake
|
||||
csrc/elementwise/scale.cu
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Keep the list **alphabetically sorted** (the file explicitly requires this)
|
||||
- If the kernel has arch constraints, reflect that in tests/benchmarks via skip logic
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Expose a Python API under `python/sglang/kernels/aot/python/sgl_kernel/`
|
||||
|
||||
Prefer following the existing module organization first. For elementwise kernels, the usual pattern is:
|
||||
|
||||
- implement the Python wrapper in `python/sglang/kernels/aot/python/sgl_kernel/elementwise.py`
|
||||
- then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||
|
||||
For example, in `python/sglang/kernels/aot/python/sgl_kernel/elementwise.py`, add:
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
def scale(
|
||||
input: torch.Tensor,
|
||||
factor: float,
|
||||
out: torch.Tensor | None = None,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Element-wise scale: out = input * factor.
|
||||
|
||||
Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input : CUDA input tensor
|
||||
factor : scale factor (float)
|
||||
out : optional pre-allocated CUDA output tensor (same shape/dtype as input)
|
||||
"""
|
||||
if out is None:
|
||||
out = torch.empty_like(input)
|
||||
torch.ops.sgl_kernel.scale.default(out, input, factor)
|
||||
return out
|
||||
```
|
||||
|
||||
Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py` following the existing import style used by other kernels.
|
||||
|
||||
---
|
||||
|
||||
## SGLang integration (required for runtime use)
|
||||
|
||||
After exposing the AOT wheel symbol, add a lazy wrapper and a `KernelSpec` under
|
||||
`python/sglang/kernels/ops/<group>/`. Set `backend=KernelBackend.AOT`, record
|
||||
actual device/architecture support, and keep `sgl_kernel` imports inside the
|
||||
implementation path. SGLang runtime and integration tests import that wrapper.
|
||||
For this example, use `sglang.kernels.ops.elementwise.scale`.
|
||||
|
||||
The wheel-level tests below validate its standalone API/build. They do not
|
||||
replace CI-registered SGLang correctness tests in
|
||||
`test/registered/kernels/ops/elementwise/test_scale.py` and benchmarks in
|
||||
`test/registered/kernels/benchmark/elementwise/bench_scale.py`. Follow
|
||||
`write-sglang-test` for their registration and CI budget; do not register tests
|
||||
under the shipped `python/sglang/` package.
|
||||
|
||||
## Step 6: Write tests (required)
|
||||
|
||||
Create `python/sglang/kernels/aot/tests/test_scale.py`:
|
||||
```python
|
||||
import pytest
|
||||
|
||||
import torch
|
||||
import sgl_kernel
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||
@pytest.mark.parametrize("size", [128, 1024, 4096, 65536])
|
||||
@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0])
|
||||
def test_scale_correctness(dtype, size, factor):
|
||||
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(input)
|
||||
|
||||
result = sgl_kernel.scale(input, factor, out=out)
|
||||
assert result is out
|
||||
|
||||
expected = input * factor
|
||||
rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
|
||||
torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
|
||||
|
||||
|
||||
def test_scale_shape_mismatch():
|
||||
input = torch.randn(128, dtype=torch.float16, device="cuda")
|
||||
out = torch.empty(256, dtype=torch.float16, device="cuda")
|
||||
with pytest.raises(RuntimeError, match="same shape"):
|
||||
sgl_kernel.scale(input, 2.0, out=out)
|
||||
|
||||
|
||||
def test_scale_cpu_input():
|
||||
input = torch.randn(128, dtype=torch.float16) # CPU
|
||||
out = torch.empty_like(input)
|
||||
with pytest.raises(RuntimeError, match="CUDA"):
|
||||
sgl_kernel.scale(input, 2.0, out=out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(pytest.main([__file__, "-q"]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Add a benchmark (required)
|
||||
|
||||
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
|
||||
|
||||
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
import sgl_kernel
|
||||
from sglang.kernels.jit.benchmark import marker
|
||||
|
||||
|
||||
def sglang_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
|
||||
sgl_kernel.scale(input, factor, out=out)
|
||||
|
||||
|
||||
def torch_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
|
||||
torch.mul(input, factor, out=out)
|
||||
|
||||
|
||||
FN_MAP = {"sglang": sglang_scale, "torch": torch_scale}
|
||||
|
||||
|
||||
@marker.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32], [torch.float16])
|
||||
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096]) # 1K .. 512K
|
||||
@marker.benchmark("provider", ["sglang", "torch"])
|
||||
def benchmark(dtype: torch.dtype, size: int, provider: str):
|
||||
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(input)
|
||||
return marker.do_bench(
|
||||
FN_MAP[provider],
|
||||
# Pass every tensor through input_args (not a closure) so marker rotates
|
||||
# them across CUDA-graph calls to defeat L2 reuse.
|
||||
input_args=(input, 2.0, out),
|
||||
# Bandwidth = bytes(input) + bytes(out), both already in input_args.
|
||||
memory_output=None,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
benchmark.run()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Build
|
||||
|
||||
Build:
|
||||
|
||||
```bash
|
||||
cd python/sglang/kernels/aot
|
||||
make build -j16
|
||||
```
|
||||
|
||||
If you need to limit host resource usage:
|
||||
|
||||
```bash
|
||||
cd python/sglang/kernels/aot
|
||||
make build -j1 MAX_JOBS=2 CMAKE_ARGS="-DSGL_KERNEL_COMPILE_THREADS=1"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Validate
|
||||
|
||||
After building successfully, run the test and benchmark:
|
||||
|
||||
```bash
|
||||
pytest python/sglang/kernels/aot/tests/test_scale.py -q
|
||||
python python/sglang/kernels/aot/benchmark/bench_scale.py
|
||||
```
|
||||
|
||||
PR CI also runs `pr-test-sgl-kernel.yml`, including the B200 job
|
||||
`sgl-kernel-b200-test` when kernel changes are detected. Use that job as the
|
||||
Blackwell coverage signal for AOT `sgl-kernel` changes.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Async CUDA errors**: `CUDA_LAUNCH_BLOCKING=1`
|
||||
- **Memory errors**: `compute-sanitizer --tool memcheck python ...`
|
||||
- **Build is too slow / OOM**: reduce `MAX_JOBS` and `SGL_KERNEL_COMPILE_THREADS`
|
||||
- **Binary bloat**: use `python/sglang/kernels/aot/analyze_whl_kernel_sizes.py`
|
||||
- **CMake sources list**: if your `.cu` file is missing from `SOURCES`, the symbol will be undefined at link time
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `python/sglang/kernels/aot/README.md`
|
||||
- `python/sglang/kernels/aot/include/sgl_kernel_ops.h`
|
||||
- `python/sglang/kernels/aot/csrc/common_extension.cc`
|
||||
- `python/sglang/kernels/aot/CMakeLists.txt`
|
||||
- `python/sglang/kernels/aot/include/utils.h` — `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro and friends
|
||||
- `python/sglang/kernels/aot/csrc/elementwise/activation.cu` — reference for the FP16/BF16/FP32 dispatch pattern
|
||||
|
||||
## Summary of Files Created/Modified
|
||||
|
||||
```
|
||||
python/sglang/kernels/aot/csrc/elementwise/scale.cu # NEW: CUDA kernel + launcher
|
||||
python/sglang/kernels/aot/include/sgl_kernel_ops.h # MODIFIED: C++ declaration
|
||||
python/sglang/kernels/aot/csrc/common_extension.cc # MODIFIED: schema + dispatch registration
|
||||
python/sglang/kernels/aot/CMakeLists.txt # MODIFIED: add source file (alphabetical)
|
||||
python/sglang/kernels/aot/python/sgl_kernel/elementwise.py # MODIFIED: Python wrapper
|
||||
python/sglang/kernels/aot/python/sgl_kernel/__init__.py # MODIFIED: re-export Python API
|
||||
python/sglang/kernels/aot/tests/test_scale.py # NEW: tests
|
||||
python/sglang/kernels/aot/benchmark/bench_scale.py # NEW: benchmark
|
||||
```
|
||||
@@ -1,414 +0,0 @@
|
||||
---
|
||||
name: ci-workflow-guide
|
||||
description: Guide to SGLang CI workflow orchestration — stage ordering, fail-fast, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages.
|
||||
---
|
||||
|
||||
# SGLang CI Workflow Orchestration Guide
|
||||
|
||||
This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and aborted on failure across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md).
|
||||
|
||||
---
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- **Suite**: `base-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `base-b-test-1-gpu-small`)
|
||||
- **Test group**: Directory-level registered test group under `test/registered/` (e.g., `hicache` maps to `test/registered/hicache/test_*.py`)
|
||||
- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions |
|
||||
| `.github/workflows/pr-test-extra.yml` | Extra workflow — gated by BOTH `run-ci` and `run-ci-extra` labels |
|
||||
| `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting |
|
||||
| `.github/actions/check-pr-test-health/action.yml` | Cross-job fail-fast: queries API for any failed job |
|
||||
| `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete |
|
||||
| `.github/actions/check-maintenance/action.yml` | Maintenance mode check |
|
||||
| `.github/scripts/ci-labels.cjs` | Resolves the four CI control labels into dispatch axes |
|
||||
| `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests |
|
||||
| `python/sglang/test/ci/ci_register.py` | Test registration (AST-parsed markers), LPT auto-partition |
|
||||
| `python/sglang/test/ci/ci_utils.py` | `run_unittest_files()`: execution, retry, continue-on-error |
|
||||
| `scripts/ci/utils/slash_command_handler.py` | Handles slash commands from PR comments |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ build kernel │
|
||||
└──────┬───────┘
|
||||
│
|
||||
├─ check-changes ──── detects which packages changed
|
||||
│ (main_package, sgl_kernel, jit_kernel, multimodal_gen)
|
||||
│
|
||||
├─ call-gate ──────── pr-gate.yml (draft? label? rate limit?)
|
||||
│
|
||||
├─────────────────────────────────────────────────────┐
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────────────────────────┐ │
|
||||
│ Base A (~3 min) │ │
|
||||
│ pre-flight check │ │
|
||||
│ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ base-a-test-1-gpu-small │ │ │
|
||||
│ │ (small GPUs) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ base-a-test-cpu │ │ │
|
||||
│ │ (CPU) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
└──────┬──────────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────┐ ┌──────────────────────────┐
|
||||
│ Base B (~30 min) │ │ kernel test │
|
||||
│ base tests │ └──────────────────────────┘
|
||||
│ │ ┌──────────────────────────┐
|
||||
│ ┌─────────────────────────────┐ │ │ multimodal gen test │
|
||||
│ │ base-b-test-1-gpu-small │ │ └──────────────────────────┘
|
||||
│ │ (small GPUs, e.g. 5090) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-b-test-1-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-b-test-2-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Base C (~30 min) │
|
||||
│ advanced tests │
|
||||
│ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-c-test-4-gpu-h100 │ │
|
||||
│ │ (H100 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-c-test-8-gpu-h200 │ │
|
||||
│ │ (8 x H200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-c-test-4-gpu-b200 │ │
|
||||
│ │ (4 x B200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ Other advanced tests │ │
|
||||
│ │ (DeepEP, PD Disagg, GB300) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ pr-test-finish │
|
||||
│ aggregates all results, fails if │
|
||||
│ any job failed/cancelled │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fails fast (red X) with a root cause annotation.
|
||||
|
||||
**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fail-fast is also disabled.
|
||||
|
||||
---
|
||||
|
||||
## Fail-Fast Layers
|
||||
|
||||
4 layers of fail-fast, from fine to coarse:
|
||||
|
||||
| Layer | Mechanism | Granularity | Disabled on schedule? |
|
||||
|-------|-----------|-------------|----------------------|
|
||||
| **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes |
|
||||
| **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes |
|
||||
| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fail-fast (red X) | Yes |
|
||||
| **4. Stage → stage (cross-stage)** | `wait-for-base-*` + `needs` | Base A fails → base B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
|
||||
|
||||
- **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py`
|
||||
- **Layer 2**: `--continue-on-error` flag in `run_suite.py` — off for PRs, on for scheduled runs
|
||||
- **Layer 3**: `check-pr-test-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
|
||||
- **Layer 4**: `wait-for-base-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
|
||||
|
||||
---
|
||||
|
||||
## Execution Modes
|
||||
|
||||
| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | Manual dispatch (`workflow_dispatch`) |
|
||||
|--------|---------------------|------------------------------|--------------------------------------|
|
||||
| **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only |
|
||||
| **Cross-job fail-fast** | Yes (`check-pr-test-health`) | Yes | Yes |
|
||||
| **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No |
|
||||
| **Retry** | Enabled | Enabled | Enabled |
|
||||
| **max_parallel** | 3 (default), 14 if `max-concurrency` label | 14 | 3 (default), 14 if `max-concurrency` |
|
||||
| **PR gate** | Yes (draft, label, rate limit) | Skipped | Skipped |
|
||||
| **Concurrency** | `cancel-in-progress: true` per PR | Queue (no cancel) | Isolated per stage+SHA |
|
||||
|
||||
Four labels relax these limits for one PR: `bypass-fail-fast`, `parallel-stages`,
|
||||
`max-concurrency`, and `highest-priority` (all three). `.github/scripts/ci-labels.cjs`
|
||||
resolves them; the [contribution guide](https://docs.sglang.io/developer_guide/contribution_guide.html#ci-control-labels)
|
||||
describes what each one does.
|
||||
|
||||
---
|
||||
|
||||
## Stage Gating (`wait-for-jobs` action)
|
||||
|
||||
`wait-for-base-a` and `wait-for-base-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
|
||||
|
||||
**How it works:**
|
||||
1. Calls `listJobsForWorkflowRun` to list all jobs in the current run
|
||||
2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `base-b-test-1-gpu-small (3)`)
|
||||
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
|
||||
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
|
||||
|
||||
**Job specs example** (base-b):
|
||||
```json
|
||||
[
|
||||
{"prefix": "base-b-test-1-gpu-small", "expected_count": 8},
|
||||
{"prefix": "base-b-test-1-gpu-large", "expected_count": 14},
|
||||
{"prefix": "base-b-test-2-gpu-large", "expected_count": 4},
|
||||
{"prefix": "base-b-test-4-gpu-b200", "expected_count": 1}
|
||||
]
|
||||
```
|
||||
|
||||
> **Critical**: `expected_count` must match the matrix size. If you add/remove matrix entries, update the wait job's spec accordingly.
|
||||
|
||||
**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and manual dispatches skip these entirely, allowing parallel execution.
|
||||
|
||||
---
|
||||
|
||||
## Cross-Job Fail-Fast (`check-pr-test-health` action)
|
||||
|
||||
Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`).
|
||||
|
||||
**How it works:**
|
||||
1. Queries `listJobsForWorkflowRun` for the current workflow run
|
||||
2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-pr-test-health` (excludes cascade failures)
|
||||
3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names
|
||||
4. If none → does nothing (step succeeds)
|
||||
|
||||
**Cascade filtering**: When job A fails fast due to the health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fail-fast. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list.
|
||||
|
||||
**Usage pattern:**
|
||||
```yaml
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
...
|
||||
|
||||
- uses: ./.github/actions/check-pr-test-health
|
||||
id: pr-test-health
|
||||
|
||||
- name: Install dependencies # skipped automatically if health check failed
|
||||
... # (default if: success() is false)
|
||||
|
||||
- name: Run test # also skipped
|
||||
...
|
||||
```
|
||||
|
||||
**Visual effect**: Job shows **red X** (failure) with error annotation showing root cause job names. Subsequent steps are naturally skipped (default `if: success()` is false after a failed step). No per-step `if` guards needed.
|
||||
|
||||
**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fail-fast.
|
||||
|
||||
**Error message example:**
|
||||
```
|
||||
Fail-fast: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Within-Suite Failure Handling
|
||||
|
||||
Controlled by `run_unittest_files()` in `python/sglang/test/ci/ci_utils.py`.
|
||||
|
||||
### Flags
|
||||
|
||||
| Flag | PR default | Scheduled default | Effect |
|
||||
|------|------------|-------------------|--------|
|
||||
| `--continue-on-error` | Off | On | Off: stop at first failure. On: run all files, report all failures at end |
|
||||
| `--enable-retry` | On | On | Retry retriable failures (accuracy/perf assertions) |
|
||||
| `--max-attempts` | 2 | 2 | Max attempts per file including initial run |
|
||||
|
||||
### Retry Classification
|
||||
|
||||
When a test fails and retry is enabled, the output is classified:
|
||||
|
||||
**Non-retriable** (checked first — real code errors):
|
||||
`SyntaxError`, `ImportError`, `ModuleNotFoundError`, `NameError`, `TypeError`, `AttributeError`, `RuntimeError`, `CUDA out of memory`, `OOM`, `Segmentation fault`, `core dumped`, `ConnectionRefusedError`, `FileNotFoundError`
|
||||
|
||||
**Retriable** (accuracy/performance):
|
||||
`AssertionError` with comparison patterns (`not greater than`, `not less than`, `not equal to`), `accuracy`, `score`, `latency`, `throughput`, `timeout`
|
||||
|
||||
**Default**: Unknown `AssertionError` → retriable. Other unknown failures → not retriable.
|
||||
|
||||
### How `continue_on_error` is set
|
||||
|
||||
In `pr-test.yml`'s `check-changes` job:
|
||||
- `schedule` runs or `run_all_tests` flag → `continue_on_error = 'true'`
|
||||
- PR runs → `continue_on_error = 'false'`
|
||||
|
||||
Each test job propagates via:
|
||||
```yaml
|
||||
env:
|
||||
CONTINUE_ON_ERROR_FLAG: ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||
run: |
|
||||
python3 run_suite.py --hw cuda --suite <name> $CONTINUE_ON_ERROR_FLAG
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Partitioning
|
||||
|
||||
Large suites are split across matrix jobs using the **LPT (Longest Processing Time) heuristic** in `ci_register.py:auto_partition()`:
|
||||
|
||||
1. Sort tests by `est_time` descending, filename as tie-breaker (deterministic)
|
||||
2. Greedily assign each test to the partition with smallest cumulative time
|
||||
3. Result: roughly equal total time per partition
|
||||
|
||||
**Partition table** (CUDA per-commit suites):
|
||||
|
||||
| Suite | Partitions | Runner | max_parallel |
|
||||
|-------|-----------|--------|-------------|
|
||||
| `base-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
|
||||
| `base-a-test-cpu` | 4 | `ubuntu-latest` | — |
|
||||
| `base-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
|
||||
| `base-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
|
||||
| `base-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
|
||||
| `base-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `base-b-kernel-unit-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `base-b-kernel-unit-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `base-b-kernel-unit-test-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `base-b-kernel-benchmark-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `base-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
|
||||
| `base-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
|
||||
| `base-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
|
||||
| `base-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
|
||||
| `base-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
|
||||
| `base-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
|
||||
| `base-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
|
||||
| `base-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
|
||||
|
||||
> **Suite names are generated**, not hand-written: each comes from a test's `register_*_ci(stage=..., runner_config=...)` as `{stage}-test-{runner_config}`, and `runner_config` maps to the `Runner` column via `scripts/ci/runner_configs.yml`.
|
||||
>
|
||||
> **Note**: Kernel suites (`base-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `base-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
**Workflow usage:**
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
partition: [0, 1, 2, 3, 4, 5, 6, 7]
|
||||
steps:
|
||||
- run: python3 run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
|
||||
--auto-partition-id ${{ matrix.partition }} --auto-partition-size 8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## check-changes Job
|
||||
|
||||
Determines which test suites to run based on file changes.
|
||||
|
||||
### Detection Methods
|
||||
|
||||
| Trigger | Method | Details |
|
||||
|---------|--------|---------|
|
||||
| `pull_request` | `dorny/paths-filter` | Detects changes via GitHub diff |
|
||||
| `workflow_dispatch` (with `pr_head_sha`) | GitHub API | `repos/{repo}/compare/main...{sha}` |
|
||||
| `schedule` / `run_all_tests` | Force all true | Runs everything |
|
||||
|
||||
### Output Flags
|
||||
|
||||
| Output | Triggers |
|
||||
|--------|----------|
|
||||
| `main_package` | Base A/B/C test suites |
|
||||
| `sgl_kernel` | Kernel wheel builds + kernel test suites; also switches B200 jobs to kernel-build runner labels outside `target_stage` mode |
|
||||
| `jit_kernel` | JIT kernel test workflow |
|
||||
| `multimodal_gen` | Multimodal-gen test workflow |
|
||||
|
||||
> **Note**: In `target_stage` mode, `sgl_kernel` is only active when `include_wheel_build=true`. Without that opt-in, kernel-change reruns fail validation instead of running a target stage without freshly built wheels. Outside `target_stage`, `sgl_kernel=true` switches B200 jobs from `4-gpu-b200` / `4-gpu-b200-low-disk` to `4-gpu-b200-kernel` / `4-gpu-b200-kernel-low-disk`.
|
||||
|
||||
---
|
||||
|
||||
## Concurrency Control
|
||||
|
||||
```
|
||||
group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
```
|
||||
|
||||
| Segment | Source | Purpose |
|
||||
|---------|--------|---------|
|
||||
| `event_name` | `github.event_name` | Prevents scheduled runs colliding with fork PRs named `main` |
|
||||
| `branch` | `github.head_ref \|\| github.ref_name` | Per-branch isolation |
|
||||
| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates manual dispatches from main runs |
|
||||
| `stage` | `inputs.target_stage \|\| 'all'` | Allows parallel stage dispatches |
|
||||
|
||||
`cancel-in-progress: true` for `pull_request` events (new push cancels old run), `false` for `workflow_call`.
|
||||
|
||||
---
|
||||
|
||||
## How To: Add a New Stage Job
|
||||
|
||||
1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-base-X, ...]`
|
||||
2. Copy the `if:` condition pattern from an existing same-stage job (handles `target_stage`, `schedule`, `main_package`)
|
||||
3. Add `checkout` step
|
||||
4. Add `check-pr-test-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
|
||||
5. Add `check-maintenance` step
|
||||
6. Add `download-artifact` step if `sgl_kernel` changed
|
||||
7. Add `install dependencies` step
|
||||
8. Add `run test` step with `$CONTINUE_ON_ERROR_FLAG`
|
||||
9. Add `upload-cuda-coredumps` step with `if: always()`
|
||||
10. Register the suite name in `PER_COMMIT_SUITES` in `test/run_suite.py`
|
||||
11. If using matrix, add `--auto-partition-id` and `--auto-partition-size` to the run command
|
||||
12. **Update `wait-for-base-X`** job spec with the new job name and `expected_count` (if matrix)
|
||||
13. **Add the job to `pr-test-finish.needs`** list
|
||||
|
||||
---
|
||||
|
||||
## How To: Debug CI Failures
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---------|-------------|---------------|
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
|
||||
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
|
||||
|
||||
---
|
||||
|
||||
## Slash Commands
|
||||
|
||||
| Command | Effect |
|
||||
|---------|--------|
|
||||
| `/tag-run-ci-label` | Adds `run-ci` label to PR |
|
||||
| `/tag-run-ci-label extra` | Adds both `run-ci` and `run-ci-extra` labels |
|
||||
| `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run |
|
||||
| `/tag-and-rerun-ci` | Adds `run-ci` label + reruns failed |
|
||||
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
|
||||
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
| `/run-cpu-ci` | Dispatches `run-cpu-ci.yml` on main: only the `base-a-test-cpu` suite, at the PR head SHA, no caches saved. Same gate as `/rerun-test`. Alias: `/rerun-cpu-ci` |
|
||||
|
||||
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||
|
||||
### Label-gated workflow dispatch (pr-test, pr-test-extra)
|
||||
|
||||
`pr-test.yml` and `pr-test-extra.yml` both listen for `pull_request.labeled` (in addition to `opened`/`synchronize`/`reopened`). The `check-changes.if` gate has two clauses:
|
||||
|
||||
1. **For `labeled` events**: the just-added label must be one of the gating labels (`run-ci` for pr-test, `run-ci` or `run-ci-extra` for pr-test-extra) — otherwise every unrelated label addition would dispatch a full CI run.
|
||||
2. **All events**: the PR must currently carry the required labels.
|
||||
|
||||
This is what lets `/tag-run-ci-label` (and the `extra` variant) trigger a fresh CI run without an extra push.
|
||||
|
||||
**Caveat — skipped runs cannot be un-skipped by `run.rerun()`:** GitHub's rerun API reuses the original event payload, so rerunning a `pull_request`-event run that was skipped because of missing labels will skip again (label set in the frozen payload doesn't update). The only way to recover a label-skipped run is to add the missing label, which fires a fresh `labeled` event with the current label set. `handle_rerun_failed_ci` in the slash handler is for rerunning failed/non-label-skipped runs; it cannot revive label-skipped ones.
|
||||
@@ -1,268 +0,0 @@
|
||||
---
|
||||
name: cookbook-add-model
|
||||
description: Add a new model to the SGLang Cookbook (docs/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the homepage vendor card. Interactive, multi-phase. Run with /cookbook-add-model.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Add a model to the SGLang Cookbook
|
||||
|
||||
> Migrating an **existing legacy-template page** (one that imports a monolithic
|
||||
> `…/autoregressive/<slug>-deployment.jsx` generator)? Use the
|
||||
> `cookbook-migrate-model` skill instead — same target format, but the legacy
|
||||
> page (not the user) is the source of truth.
|
||||
|
||||
The cookbook is **config-driven**: two shared engines contain NO model-specific code —
|
||||
`docs/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
|
||||
`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
|
||||
a per-model `config` (+ optional `benchmarks`) consumed by both engines, plus an MDX page
|
||||
that imports them. No engine edits.
|
||||
|
||||
**Instantiate the model-agnostic template** (NOT a clone of any live cookbook — the
|
||||
template is decoupled and covers all hardware + all axes):
|
||||
- `templates/config.jsx.tmpl` → `docs/src/snippets/configs/<hf-org>/<model-slug>.jsx`
|
||||
- `templates/benchmarks.jsx.tmpl` → `…/<model-slug>-benchmarks.jsx` (skip if no numbers)
|
||||
- `templates/page.mdx.tmpl` → `docs/cookbook/<category>/<Vendor>/<ModelName>.mdx`
|
||||
|
||||
The template uses explicit `__TOKEN__` placeholders; you fill them, prune what the model
|
||||
lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a populated
|
||||
*instance* you can consult, but is not the template.
|
||||
|
||||
**Deep references (read on demand, don't inline):**
|
||||
- [references/authoring-reference.md](references/authoring-reference.md) — field-by-field config / cells / playground / MDX contract.
|
||||
- [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose.
|
||||
- [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work).
|
||||
- [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`).
|
||||
- [references/diffusion-authoring.md](references/diffusion-authoring.md) — required opening, tag, command-picker, and feature-overlay contract for diffusion model pages. Read before editing any `docs/cookbook/diffusion/<Vendor>/<Model>.mdx` page.
|
||||
|
||||
## Architecture at a glance
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ cookbook/<category>/<Vendor>/<Model>.mdx │
|
||||
│ import { Deployment } from "/src/snippets/_deployment.jsx"; │
|
||||
│ import { Playground } from "/src/snippets/_playground.jsx"; │
|
||||
│ import { config } from "/src/snippets/configs/.../X.jsx"; │
|
||||
│ <Deployment config={config} /> <Playground config={config} />│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (config passed as React prop)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ src/snippets/configs/<vendor>/<model>.jsx │
|
||||
│ export const config = { │
|
||||
│ supportedHardware, variants, quantizations, strategies, ... │
|
||||
│ cells: [ { match:{hw,variant,quant,strategy,nodes}, │
|
||||
│ env:[...], flags:[...] }, ... ], // 5-dim matrix │
|
||||
│ playgroundFeatures: { attention, moe, parsers, ... }, │
|
||||
│ }; │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (consumed by BOTH engines — no model code in engines)
|
||||
▼
|
||||
┌──────────────────────────────────┬──────────────────────────────┐
|
||||
│ _deployment.jsx │ _playground.jsx │
|
||||
│ Renders the verified matrix; │ Renders override chips + │
|
||||
│ one cell → its env/flags. │ diff against the cell. │
|
||||
└──────────────────────────────────┴──────────────────────────────┘
|
||||
```
|
||||
|
||||
The two widgets stay in sync via: the **URL hash** (deploy mirrors its selection;
|
||||
playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatches on
|
||||
every change; playground listens — `replaceState` doesn't fire `hashchange`), and the
|
||||
shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders).
|
||||
|
||||
> The main template is **autoregressive**. Diffusion pages use
|
||||
> `templates/diffusion-page.mdx.tmpl` plus
|
||||
> [references/diffusion-authoring.md](references/diffusion-authoring.md); do not force the
|
||||
> autoregressive deployment matrix on them. Omni pages follow their own category structure.
|
||||
> All categories still obey the Mintlify / NEW-tag / docs.json / category-card / validation
|
||||
> rules below.
|
||||
|
||||
---
|
||||
|
||||
**Interactive, multi-step workflow. Collect inputs incrementally — don't ask for
|
||||
everything upfront.** The real work is the verified `cells[]` recipes + measured
|
||||
benchmarks (Phases 2 + 4); everything else is filling the template.
|
||||
|
||||
## Phase 1 — Collect inputs
|
||||
|
||||
1. **Model card** — HuggingFace repo/URL. **Fetch the page** and extract description,
|
||||
param count, architecture, context length, **license**. (Fetching guards factual bugs
|
||||
like an off-by-a-few-B param count.) If the model isn't public, ask the user.
|
||||
2. **Variants / quantizations** — keep separate: variants are size/mode (e.g. Flash/Pro,
|
||||
Instruct/Thinking); quantizations come from the HF card / linked repos (BF16/FP8/FP4/…).
|
||||
Default to BF16 when a full-precision repo exists.
|
||||
3. **Tested hardware + parallelism** — which platforms are actually tested, and TP/EP/DP
|
||||
for each. List only tested hw (unlisted greys out).
|
||||
4. **Verified launch recipes** — the full `sglang serve` flags per
|
||||
(hw × variant × quant × strategy × nodes) combo → these become `cells[]`. Rewrite any
|
||||
`python -m sglang.launch_server` to `sglang serve` form.
|
||||
5. **sglang version / image tag** — ask which sglang build the recipes + benchmarks ran on
|
||||
(a release like `0.5.x`, or `main`/nightly). **Never guess or hallucinate it.** This one
|
||||
tag fills `dockerImages` and the benchmarks' `sglang_version`; when the user is unsure,
|
||||
default the image to `lmsysorg/sglang:dev` (nightly) rather than inventing a release.
|
||||
6. **Pre-flight**: `gh pr list --repo sgl-project/sglang --search "<model>"` (dup check).
|
||||
|
||||
**Hardware reference** (the shared `HARDWARE_CATALOG` in `_deployment.jsx`). A GPU **not** in
|
||||
this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.hardware`
|
||||
(`{id,label,vram,vendor}`) — the engine merges it in; don't edit the engine catalog:
|
||||
|
||||
| Platform | Vendor | VRAM | Docker image |
|
||||
|---|---|---|---|
|
||||
| H100 | NVIDIA | 80GB | `lmsysorg/sglang:<ver>` |
|
||||
| H200 | NVIDIA | 141GB | `lmsysorg/sglang:<ver>` |
|
||||
| B200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` |
|
||||
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
|
||||
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:<ver>` — needs a **CUDA 13** build |
|
||||
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
| MI355X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
|
||||
- **Image tag (`<ver>`)**: don't guess — ask the user for the tag the recipes ran on, or
|
||||
default to `dev` (nightly). The same tag goes in `dockerImages` and benchmarks'
|
||||
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
|
||||
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
|
||||
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
|
||||
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300
|
||||
single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so
|
||||
its only multi-GPU topology is TP=2 across 2 nodes.
|
||||
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
|
||||
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
|
||||
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
|
||||
- **EP** (MoE): 8-GPU NVIDIA `--tp 8 --ep 8`; AMD `EP = TP`; small NVIDIA (TP≤4) omit
|
||||
`--ep` unless benchmarked. (The template's AMD example cell shows these.)
|
||||
|
||||
## Phase 2 — Instantiate the template
|
||||
|
||||
For a diffusion model, instantiate `templates/diffusion-page.mdx.tmpl` and keep the
|
||||
shared `DiffusionModelTags` component, plus `templates/diffusion-config.jsx.tmpl` for the
|
||||
opt-in scoped command builder. Put the compact install command and builder in §1 Quick
|
||||
start. The first two paragraphs in §2 Model capabilities are not generic filler: they must
|
||||
state the model's capability range, strongest differentiator, when to choose it, and at
|
||||
least one real deployment or capability boundary. Put orthogonal runtime features in
|
||||
`scope: "serve"` or `scope: "request"`, not in the base recipe; use the schema from
|
||||
`references/diffusion-authoring.md`.
|
||||
|
||||
1. **Copy** the three template files to their target paths (above). Note the two
|
||||
vendor-folder conventions: under `configs/` the folder is the **HuggingFace org**
|
||||
(`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`).
|
||||
2. **Replace every `__TOKEN__`**: `__MODEL_DISPLAY__`, `__MODEL_SLUG__`, `__HF_ORG__`,
|
||||
`__HF_REPO__`, `__REASONING_PARSER__`, `__TOOLCALL_PARSER__`, `__ONE_LINER__`. Verify
|
||||
none remain: `grep -rn '__[A-Z_]*__' <new files>`.
|
||||
3. **Prune** to what the model supports (delete, don't stub) — using
|
||||
[references/authoring-reference.md](references/authoring-reference.md):
|
||||
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
|
||||
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
|
||||
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
|
||||
- `playgroundFeatures` axes: remove the `megamoe` backend option + the
|
||||
`megamoeQuant` block from the `moe` axis (non-Blackwell-MoE), delete `hisparse`
|
||||
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
|
||||
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
|
||||
to single `default` if there's no variant axis (then drop the `variant` half of
|
||||
`modelNames`/`defaultAccuracy` keys).
|
||||
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||
a guessed release; key by `hw`, or a more specific key — `hw|variant|quant`, `variant|quant`,
|
||||
`hw|quant|strategy`, `hw|quant`, resolved in that order before `hw` — when one
|
||||
variant/quant/strategy needs its own image), `multiNodeHints` only for fabric-specific hw
|
||||
(e.g. gb200).
|
||||
|
||||
5. **Diffusion pages: add the ComfyUI section.** Every diffusion cookbook page ends with
|
||||
a `## <n>. Run in ComfyUI` section so a reader never has to guess whether the model is
|
||||
reachable from ComfyUI. It is one component; the per-model facts live in the component,
|
||||
not the page:
|
||||
|
||||
```mdx
|
||||
## <n>. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="<key>" />
|
||||
```
|
||||
|
||||
Pick `model` from the table in
|
||||
`docs/src/snippets/diffusion/comfyui-support.jsx`. Use the model's own key when it has
|
||||
an entry (its executor or dedicated node differs); otherwise use the generic `image` or
|
||||
`video`. A model gets its own key only when the plugin actually treats it specially —
|
||||
an entry in `executor_class_dict`
|
||||
(`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a
|
||||
dedicated node. Adding a key without the matching plugin support makes the page lie.
|
||||
|
||||
### Site-wiring (do all three)
|
||||
|
||||
- **`docs/docs.json`** — add the page under Cookbook → `<category>` → `<Vendor>`, at
|
||||
the **top** of that vendor's `pages` (root-relative, no `.mdx`:
|
||||
`cookbook/<category>/<Vendor>/<Model>`). New vendor group → insert in the section's
|
||||
local ordering.
|
||||
- **NEW-tag hygiene** — the new page keeps `tag: NEW` (from the template). Scan the
|
||||
vendor dir for existing NEW and strip it from siblings; verify ≤1:
|
||||
`grep -rn 'tag: NEW' docs/cookbook/<category>/<Vendor>/` → at most one result. (Scan
|
||||
files; don't assume the first `docs.json` entry holds NEW.)
|
||||
- **Homepage card** — `docs/cookbook/<category>/intro.mdx`: if the org already has a
|
||||
`<Card>`, update only its `href` (keep `img`). If the org is **new**, add a `<Card>`
|
||||
(title = nav-group name; keep card order aligned with `docs.json`) **and create its logo**:
|
||||
ask the user for the brand logo, then generate the conforming **icon-only 940×525 RGBA
|
||||
transparent** PNG → `docs/cards/logos/<org-slug>.png` per
|
||||
[references/vendor-logo.md](references/vendor-logo.md) (track with `git add -f` — `*.png`
|
||||
is gitignored repo-wide). Never invent or copy a logo.
|
||||
|
||||
## Phase 3 — Validate
|
||||
|
||||
```bash
|
||||
cd docs
|
||||
mint validate # frontmatter, missing nav entries, MDX/JSX errors
|
||||
mint broken-links
|
||||
mint dev # visual smoke test at http://localhost:3000/cookbook/<category>/<Vendor>/<Model>
|
||||
```
|
||||
|
||||
Spot-check: cells render sensible commands; URL-hash nav persists across reload; the
|
||||
Playground inherits the Deploy selection live; each axis toggle produces the expected
|
||||
diff; Docker mode wraps in `docker run` with the right image; multi-node cells emit the
|
||||
hints + `--nnodes N`; cURL resolves the model name; the NEW badge shows on the new page
|
||||
and is gone from same-vendor siblings; the homepage card points to the new model.
|
||||
|
||||
## Phase 4 — Interactive testing
|
||||
|
||||
The user deploys each cell, runs the benches, and pastes results; you fill the data:
|
||||
- mark each tested `cells[]` entry `verified: true` (absent = yellow/unverified badge);
|
||||
- fill the `<model>-benchmarks.jsx` entries (one per cell `match`) with measured
|
||||
speed/accuracy + the `sglang_version` the user reports (don't invent one — the template's
|
||||
`0.0.0` is a deliberate TODO); set model-level `defaultAccuracy` per variant. Leave a
|
||||
cell's entry as a bare `match` stub if it has no numbers yet (the card shows "pending").
|
||||
|
||||
## Phase 5 — Prose & config tips
|
||||
|
||||
**Read [references/mintlify-authoring.md](references/mintlify-authoring.md) first** (it
|
||||
carries the parser-output-shape / thinking-mode / Output-Example / no-hardcoded-sampling
|
||||
rules + the Mintlify forbidden-syntax list). Then rewrite the MDX prose from the HF card +
|
||||
user notes: §1 Model Introduction (description, links, params, license, variants table),
|
||||
§2 Configuration Tips (hw-specific tuning, caveats), §3 Advanced Usage (Reasoning /
|
||||
Tool-Calling / HiCache — keep only what applies; match the reasoning example to the
|
||||
parser's output shape; each runnable block gets an `**Output Example:**`).
|
||||
|
||||
For diffusion pages, follow the category-specific Quick start and capability contract in
|
||||
`references/diffusion-authoring.md`. Run `node docs/scripts/check_cookbook_configs.mjs` to
|
||||
verify the tag widget and introduction structure before rendering the page.
|
||||
|
||||
## Phase 6 — Review
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Git workflow
|
||||
|
||||
Always branch — never commit to main directly.
|
||||
|
||||
```bash
|
||||
git checkout -b add-<model>-cookbook
|
||||
git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs/src/snippets/configs/<hf-org>/<slug>-benchmarks.jsx \
|
||||
docs/cookbook/<category>/<Vendor>/<Model>.mdx \
|
||||
docs/docs.json docs/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
@@ -1,241 +0,0 @@
|
||||
---
|
||||
name: cookbook-migrate-model
|
||||
description: Migrate a legacy-template SGLang cookbook page (monolithic per-model generator under docs/src/snippets/autoregressive/) onto the config-driven template (shared _deployment.jsx / _playground.jsx engines + per-model config). Use when asked to migrate, convert, or port an existing cookbook page — NOT for brand-new models (use cookbook-add-model for those). Run with /cookbook-migrate-model <Model page name, e.g. GLM-5.1>.
|
||||
---
|
||||
|
||||
# Cookbook Migrate Model
|
||||
|
||||
Convert one legacy cookbook page to the config-driven format, faithfully. The
|
||||
legacy page — its generator widget and its measured benchmark blocks — is the
|
||||
**single source of truth**. You are transcribing it into the new data model,
|
||||
not improving it.
|
||||
|
||||
Reuses the `cookbook-add-model` skill's assets (read them on demand):
|
||||
- `../cookbook-add-model/templates/config.jsx.tmpl`, `page.mdx.tmpl`, `benchmarks.jsx.tmpl`
|
||||
- `../cookbook-add-model/references/authoring-reference.md` (config/cells/playground contract)
|
||||
- `../cookbook-add-model/references/mintlify-authoring.md` (MDX rules)
|
||||
|
||||
Migration-specific references in this skill:
|
||||
- [references/dimension-mapping.md](references/dimension-mapping.md) — legacy-control → new-dimension mapping rules, command rewrite table, per-family strategy sets, and the Qwen3.5 pilot as a worked example (PR #27848).
|
||||
|
||||
The round's per-model inventory (scope, batch order, quirks, measured-data
|
||||
survey) is tracked by the migration maintainer outside the repo — expect it in
|
||||
your dispatch prompt, or ask for it.
|
||||
|
||||
## Hard rules (non-negotiable)
|
||||
|
||||
1. **Never modernize.** Env vars, flags, TP values, docker tags, version strings
|
||||
are copied verbatim from the legacy page — even when today's defaults differ
|
||||
(e.g. a flag whose behavior has since become the default; keep it anyway). The recipe
|
||||
that was verified is the recipe as written. Allowed normalizations are ONLY
|
||||
the five alias rewrites in dimension-mapping.md §2 (`launch_server`→`sglang
|
||||
serve`, `--model`→`--model-path`, `--tp-size`→`--tp`, abbreviated
|
||||
`--speculative-algo`→`--speculative-algorithm`,
|
||||
`--expert-parallel-size`→`--ep`). **Accuracy-degrading flags**
|
||||
(`--kv-cache-dtype fp8_e4m3`, W4A4-style runtime quant) follow a
|
||||
deterministic rule — enforced in migration, no asking: offered as a
|
||||
legacy **selectable option** → never select it (cells mirror the
|
||||
accuracy-safe side), and the option **survives as a Playground axis** —
|
||||
an existing one where it fits, else add one via a separate prior engine
|
||||
PR (rule 4); a legacy choice never degrades to a tips mention. Baked
|
||||
into the recipe's **default/unconditional command** → keep it verbatim
|
||||
(the recipe was measured with it, and fp8 KV halves KV memory —
|
||||
stripping could OOM it). See dimension-mapping.md §2 caveats.
|
||||
2. **Never invent versions or numbers.** Benchmark numbers only from the
|
||||
legacy page's measured blocks, and a result migrates ONLY when its
|
||||
`sglang_version` is a **reproducible anchor** — the bar is reproducibility,
|
||||
not "must be a release":
|
||||
- ✅ release tag/version (`v0.5.9` / `0.5.9`), commit hash, OR — for
|
||||
**Day-0 support** (the enabling PR isn't merged and no release is cut
|
||||
yet) — a specific **PR (`PR #27944`) or commit** you can `gh pr checkout`
|
||||
/ `git checkout <sha>`. Commit is most precise; a PR pin is fine for day-0.
|
||||
- ❌ a moving ref — `"main branch"`, `"main (2026-06-11)"`, open-ended
|
||||
`"0.5.8+"` — is NOT reproducible: **drop the WHOLE result (speed AND
|
||||
accuracy)**, not just speed. Keep `benchmarkCommands` so ⚡Reproduce still
|
||||
guides re-measurement against a pinned build.
|
||||
**Never inherit cross-model numbers** — measurements the legacy page
|
||||
attributes to a *different model* (e.g. a K2.6 page carrying K2.5-measured
|
||||
speed) are dropped regardless of version. When kept, `sglang_version` is the
|
||||
legacy page's string verbatim. Docker tags only the ones the legacy page
|
||||
pinned (unmapped hw falls back to `:dev`).
|
||||
3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the
|
||||
legacy page has concrete measured data for that exact 5-dim combo AND
|
||||
(b) the cell's flags equal the deployment command used for that measurement
|
||||
— modulo `{{HOST_IP}}`/`{{PORT}}`, the five alias rewrites, and **parser
|
||||
flags**: `--reasoning-parser`/`--tool-call-parser` are stripped from every
|
||||
cell (Playground-only feature; when the measured run had them on, say so in
|
||||
the benchmarks file header). When the measured command diverges from the
|
||||
generator default, **the verified cell follows the measured command**; the
|
||||
generator default stays as the sibling strategy/cell or a tips note. Everything else is unverified (yellow) —
|
||||
including combos that look memory-infeasible; keep them verbatim and list
|
||||
them in the PR body for the re-verification track.
|
||||
4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not
|
||||
change in a migration PR. Model-specific features are config DATA consumed
|
||||
by generic axis handlers (MegaMoE precedent), so they need NO engine change.
|
||||
A **titled single-select that strips a flag family** — KV Cache DType
|
||||
(`--kv-cache-dtype`), mamba (`--mamba-radix-cache-strategy`), … — is already
|
||||
covered by the merged generic **`flagSelects`** axis: declare it in the
|
||||
config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5
|
||||
mamba example), **no engine PR**. Only a genuinely new control *shape* that
|
||||
`flagSelects` can't express would need a one-time generic primitive (never a
|
||||
model-named handler) on a separate prior PR (engine-axis.md).
|
||||
5. **`github.cookbookModel` must be set** (`<hf-org>/<page-slug>`, e.g.
|
||||
`qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels
|
||||
as deepseek-v4. The issue template itself needs NO edits (free-form input).
|
||||
6. **Strategy tiers are signal-driven.** A cell goes under `low-latency` /
|
||||
`high-throughput` ONLY on a signal present in the legacy source (an
|
||||
explicit performance toggle, a named recipe, or prose stating the
|
||||
operating point); **no signal → `balanced`**. Never derive a slant from
|
||||
your own hardware intuition — re-tiering on measured evidence is the
|
||||
hardware owner's follow-up PR, not part of a migration
|
||||
(dimension-mapping.md §4).
|
||||
|
||||
## Workflow (one model = one PR)
|
||||
|
||||
### 1. Inventory the legacy assets
|
||||
- Read the legacy generator (`docs/src/snippets/autoregressive/<slug>-deployment.jsx`)
|
||||
end-to-end: every option dimension (radio vs checkbox vs dynamic), every
|
||||
gate/SUPPORT matrix, the full emitted command per reachable combo (env
|
||||
prefixes, `# Error` pseudo-commands included).
|
||||
- Read the legacy MDX: §2 install docker tags → `dockerImages` (pinned, not
|
||||
upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
|
||||
Output Examples verbatim); §5 benchmark blocks → transcribe each measured
|
||||
block: deploy command used, bench command (dataset/isl/osl/num-prompts/
|
||||
concurrency), P50 (median) TTFT/TPOT, output tok/s, hardware, version string.
|
||||
- Inbound-anchor sweep: `grep -rn "<PageName>" docs/ --include='*.mdx'` —
|
||||
find links/`#fragments` into this page (`mint broken-links` does NOT check
|
||||
fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
|
||||
- Check the maintainer-provided inventory notes for this model's known quirks —
|
||||
but treat them (and the §4 family table) as a **survey snapshot**: re-verify
|
||||
every dimension against the live legacy files before mapping. Pages keep
|
||||
receiving updates (precedent: Kimi-K2.6's live generator has a speculative
|
||||
toggle the 2026-06-10 survey notes lack).
|
||||
|
||||
### 2. Design the 5-dim mapping
|
||||
Apply [references/dimension-mapping.md](references/dimension-mapping.md). Key
|
||||
decision — which legacy toggle becomes the `strategies` dimension: a toggle
|
||||
that **changes other parts of the command** (TP, mem) must (the Playground
|
||||
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||
even when its flags are uncoupled (`--attn-dp-size N` is a pure
|
||||
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||
Playground axis with the flags baked into cells when the legacy default was
|
||||
ON — EXCEPT parsers:
|
||||
`--reasoning-parser`/`--tool-call-parser` are NEVER baked into cells, they are
|
||||
Playground-only (DSv4 convention). **Every legacy control survives as an
|
||||
interactive control** — a dimension or a Playground axis, never a tips-only
|
||||
mention — and a model-specific control is **config data, not engine code**
|
||||
(MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure
|
||||
config whenever it fits an existing axis's data schema. A **titled
|
||||
single-select that strips a flag family** (Nemotron3's "KV Cache DType",
|
||||
mamba `--mamba-radix-cache-strategy`, …) fits the merged generic **`flagSelects`**
|
||||
axis — so it too is config-only (declare a `flagSelects` list). Only a control
|
||||
whose *shape* `flagSelects` still can't express would need a ONE-TIME generic
|
||||
primitive (never a model-named handler) on a separate PRIOR engine PR, keeping
|
||||
the migration PR data-only (hard rule 4, engine-axis.md). The strategy count follows the page's
|
||||
operating points: **one recipe → a single `balanced`; two → `low-latency` +
|
||||
`high-throughput`; three → the full trio (the ideal)**. The tiers apply per
|
||||
(hw × variant × quant) combination — a single-recipe combination on a
|
||||
multi-strategy page parks under its semantically honest tier (no
|
||||
latency/throughput slant → `balanced`; the page's list is the union). When the
|
||||
legacy toggle is MTP / speculative decoding, the direction is a deterministic
|
||||
default — apply without asking: **MTP on → `low-latency`, MTP off →
|
||||
`high-throughput`** (reversed only with maintainer confirmation). Tier
|
||||
placement is signal-driven (hard rule 6). Never invent a recipe just to fill
|
||||
strategy chips (see dimension-mapping.md §4). Record the outcome as a
|
||||
**strategy mapping table** for the PR body — one row per group of
|
||||
combinations sharing the same legacy signal (e.g. "all GPU combos: MTP
|
||||
toggle → low-latency / high-throughput"; "xeon: (none) → balanced"), with a
|
||||
one-line rationale each; don't enumerate 60 identical rows. The table is what
|
||||
hardware owners sign off on at review.
|
||||
|
||||
### 3. Generate the config (codegen, then audit)
|
||||
- For >~30 cells, port the legacy `generateCommand()` into a throwaway Node
|
||||
script that enumerates combos and emits the `cells:[...]` literal (output
|
||||
must stay a pure literal — Mintlify forbids runtime spreads/calls). Apply the
|
||||
verified-cell override in the script. See the pilot scripts embedded in the
|
||||
worked example of dimension-mapping.md §5.
|
||||
- **Independent equivalence audit (required):** extract the ORIGINAL generator
|
||||
from git (`git show main:<path>` — NOT `HEAD:`, the migration branch deletes
|
||||
the file, see dimension-mapping.md §5 item 7), stub React hooks, run it for
|
||||
every combo, and diff token-by-token against the new cells. Expected deltas
|
||||
only: the appended `--host {{HOST_IP}}`/`--port {{PORT}}`, the
|
||||
engine-injected multi-node trio, the §2 alias rewrites (the entrypoint
|
||||
rewrite doesn't appear in cell tokens — cells hold flags only; the audit
|
||||
script normalizes it on the legacy side), and the intentional verified-cell
|
||||
override. Paste the PASS count + the audit script in the PR body
|
||||
(collapsed `<details>`).
|
||||
- Hand-author the non-cells fields per authoring-reference.md. Structural
|
||||
self-checks: every cell resolves a `modelNames` key; no `--nnodes/--node-rank/
|
||||
--dist-init-addr/--host/--port` literals; every `{{KEY}}` declared; every
|
||||
`supportedHardware` id has ≥1 cell.
|
||||
|
||||
### 4. Benchmarks file
|
||||
One entry per measured block only (cells without entries already render
|
||||
"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
|
||||
total (in+out) tok/s/GPU = `output tok/s ÷ (tp × nnodes) ×
|
||||
(isl+osl)/osl` — stored directly (the card shows it as-is). TTFT/TPOT
|
||||
take the P50 (median) rows; set `config.latencyPercentile` (default `"P50"`; use
|
||||
`"Mean"` only for legacy Mean-recorded data — temporary, being migrated to P50; an
|
||||
entry-level `latencyPercentile` overrides the page value per cell).
|
||||
Put the workload's
|
||||
`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
|
||||
the benchmarks carry accuracy data** — the engine ships no default eval set
|
||||
(#27842), so missing labels means the accuracy rows silently don't render;
|
||||
extra context (sample counts, suites that don't fit) goes in the entry's
|
||||
`notes`. Zero-measured-data pages: skip the file and the `benchmarks` prop
|
||||
entirely, but keep `benchmarkCommands` so ⚡Reproduce still guides users.
|
||||
|
||||
### 5. Rewrite the MDX
|
||||
From `page.mdx.tmpl`: keep the original `title` (nav identity), write a fresh
|
||||
SEO `description` (top-level — delete any legacy `metatags.description`), **no
|
||||
`tag: NEW`** (a migration is not a launch), **no `mode:`**. Install accordion
|
||||
carries the legacy install content + pinned images. Keep the template's
|
||||
DSv4-style strategy bullets — serving semantics first (single-user chat /
|
||||
typical multi-user / batch throughput), trimmed to the strategies the page
|
||||
ships, plus at most a one-line note on what each strategy changes on this
|
||||
model; do NOT rewrite them as toggle-/migration-centric explanations.
|
||||
Legacy §5 benchmark prose is deleted (numbers → benchmark card, commands →
|
||||
⚡Reproduce); legacy prose deploy commands are deleted (doc↔config parity —
|
||||
fold their unique flags into §2 tips). Invocation examples + real outputs carry
|
||||
over verbatim, but **wrapped in Accordions** — §3 commands and outputs are
|
||||
collapsible (required, DeepSeek-V4 pattern): code in an
|
||||
`<Accordion title="… (Python)">`, output in a following
|
||||
`<Accordion title="Example Output">`; legacy pages kept them inline.
|
||||
|
||||
### 6. Delete the legacy generator
|
||||
Remove `docs/src/snippets/autoregressive/<slug>-deployment.jsx` and its
|
||||
import. `grep -rn "<slug>-deployment" docs/` must return nothing (config
|
||||
provenance comments must not name the deleted path). Site wiring needs **no
|
||||
changes**: docs.json path/title unchanged, vendor card + logo already exist.
|
||||
|
||||
### 7. Validate
|
||||
- `grep -rn '__[A-Z_]*__'` on the new files (no template tokens).
|
||||
- `cd docs && mint validate && mint broken-links` (pre-existing breaks on
|
||||
main are not yours — say so in the PR).
|
||||
- `mint dev` browser smoke: initial selection = the verified cell (first in
|
||||
`cells[]`) with green badge; multi-node cells show the injected trio +
|
||||
header; AMD cells show env prefixes; Docker mode wraps with the pinned image
|
||||
and passes cell env as `--env`; condition-hidden combos grey out; benchmark
|
||||
card values; NO parser flags in any Deploy command; Playground parser
|
||||
toggles ADD the parser flags (green additions) while spec toggles strike
|
||||
the baked spec flags (red); Submit ↗ prefills this model. **Probe pitfall:** drive at most
|
||||
ONE programmatic click per evaluation and wait for React to settle —
|
||||
multiple clicks in one synchronous script batch and read stale DOM.
|
||||
- Token-level audit from step 3 passes.
|
||||
|
||||
### 8. PR + review
|
||||
One PR per model. PR body: migration framing, verified policy applied, the
|
||||
strategy mapping table (step 2), the audit PASS count + script, any
|
||||
inherited-infeasible combos flagged for re-verification. Then run `/cookbook-review-pr <N>` and fix findings.
|
||||
FYI: docs previews only build for in-repo (`sgl-project/sglang`) branches —
|
||||
a fork-headed PR is perfectly fine but renders no preview; a maintainer can
|
||||
re-push the branch in-repo if a preview is wanted for review.
|
||||
|
||||
### 9. Keep this skill current
|
||||
Any new convention, engine behavior, or pitfall you discover while migrating
|
||||
(naming decisions, audit-script gotchas, review-rule conflicts, …) MUST be fed
|
||||
back into this skill — same PR if it's skill-file-only, or an immediate
|
||||
follow-up commit on the skill's branch/PR. The next agent runs on what's
|
||||
written here, not on your session's context.
|
||||
@@ -1,268 +0,0 @@
|
||||
---
|
||||
name: cookbook-review-pr
|
||||
description: Review a pull request against the SGLang Cookbook (docs/, Mintlify) contribution checklist — the config-driven format (per-model config + benchmarks JSX consumed by the shared _deployment.jsx / _playground.jsx engines). Run with /cookbook-review-pr <PR number>.
|
||||
---
|
||||
|
||||
# Cookbook Review PR
|
||||
|
||||
Fetch the diff, run the checklist, report what you find. The cookbook is **config-driven**:
|
||||
shared engines (`_deployment.jsx`, `_playground.jsx`) with NO model-specific code; each
|
||||
model is a data `config` (+ optional `benchmarks`) under `src/snippets/configs/<vendor>/`
|
||||
plus an MDX page. This checklist targets that layout. Field-schema detail lives in
|
||||
`.claude/skills/cookbook-add-model/references/authoring-reference.md` — defer to it rather
|
||||
than restating.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. `gh pr view <N> --repo sgl-project/sglang --json title,body,files,author,baseRefName,headRefName,commits,reviews`
|
||||
2. `gh pr diff <N> --repo sgl-project/sglang`
|
||||
3. `gh pr list --repo sgl-project/sglang --state open --search "<model name>"` (duplicate check)
|
||||
4. Run every checklist item against the diff.
|
||||
5. Output per-file verdicts + overall recommendation.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. File hygiene
|
||||
- A cookbook PR should only touch: `docs/src/snippets/configs/<vendor>/*.jsx`
|
||||
(config + benchmarks), `docs/cookbook/**/*.mdx`, `docs/docs.json`,
|
||||
`docs/cookbook/<category>/intro.mdx` (vendor card), `docs/cards/logos/<vendor>.png`
|
||||
(new vendor only). Flag stray files (`settings.local.json`, lockfiles, IDE configs).
|
||||
- Pages must be `.mdx`, not `.md`. Files end with a trailing newline. Check commit history
|
||||
for unrelated commits accidentally included.
|
||||
- **Engines untouched**: `_deployment.jsx` / `_playground.jsx` should NOT change in a
|
||||
model-add PR (adding a model is data-only). Engine edits = a separate axis/feature PR
|
||||
(see `cookbook-add-model/references/engine-axis.md`); review them against that checklist.
|
||||
|
||||
### 2. Config quality (the per-model `config`)
|
||||
- Single `export const config = { ... }` literal — **no** function calls, spreads,
|
||||
fragment refs, or IIFE (Mintlify re-evals at hydration → `ReferenceError`).
|
||||
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
|
||||
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
|
||||
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog — unless the platform is
|
||||
genuinely shared, which is an engine change the PR body has to argue (and the
|
||||
`cookbook-add-model` hardware table moves with it). A new catalog entry is inert for
|
||||
other pages: each filters the catalog by its own `supportedHardware`.
|
||||
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||
- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
|
||||
2 → `low-latency` + `high-throughput`; 3 → the full trio. Tiers apply per
|
||||
(hw × variant × quant) combination: a single-recipe combination must park under its
|
||||
semantically honest tier (clear slant → that tier, e.g. a workstation card under
|
||||
`low-latency`; no slant → `balanced`, e.g. a CPU platform) — **flag a no-slant recipe
|
||||
parked under low-latency/high-throughput**. Mixed unions
|
||||
like [low-latency, balanced, high-throughput] with per-selection greying are fine. Also
|
||||
flag model-specific ids (e.g. `mtp`), and flag an INVERTED speculative mapping — the
|
||||
deterministic default is MTP/spec-decoding ON → `low-latency`, OFF → `high-throughput`
|
||||
(at saturation the draft+verify overhead outweighs the speedup); the reverse needs an
|
||||
explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics
|
||||
in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal
|
||||
toggles.
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback); a
|
||||
`hw|quant` key (resolved before the plain `hw`) is valid when one quant on a shared GPU needs
|
||||
a different image (e.g. an FP4 dev build) — don't flag those.
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
Hints render above BOTH run modes, so flag one that reads as docker-only ("add these
|
||||
docker run flags") — `docker run` flags belong in the hardware entry's
|
||||
`multiNodeDockerFlags`, which the engine emits into the Docker command itself.
|
||||
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
|
||||
template's `model` field is a free-form input prefilled from this value; if the config
|
||||
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
|
||||
page's submissions get mislabeled.
|
||||
- `playgroundFeatures` is opt-OUT: the **general axes ship on every cookbook by default**
|
||||
(`attention` TP/CP/DP-Attn, `moe` backend+EP for MoE models, `parsers`, `speculative`,
|
||||
`pdDisagg`, `hicache`) — flag a missing general axis unless the model genuinely cannot
|
||||
use it. Model-specific axes only where applicable (MegaMoE backend + `megamoeQuant`
|
||||
only on Blackwell MoE, gated by `requiresHw`; `hisparse` only DSA-style). Knobs that are
|
||||
meaningless for a subset of variants/hw are `disable`d with a reason, not silently live
|
||||
(e.g. MoE knobs greyed on dense variants). No empty/stub axes.
|
||||
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
|
||||
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
|
||||
nothing).
|
||||
- **All-hardware considered**: every `supportedHardware` id (from the catalog or `config.hardware`) has ≥1 cell OR is a deliberate
|
||||
greyed "coming soon"; AMD was pruned or kept on purpose (not a leftover template family).
|
||||
|
||||
### 3. Cells / 5-dim matrix
|
||||
- Every cell `match` has EXACTLY the 5 keys (`hw`, `variant`, `quant`, `strategy`, `nodes`).
|
||||
- `env` / `flags` are flat literals (only `{{PLACEHOLDER}}` subst) — no shared
|
||||
`commonFlags` reference (Mintlify won't inline it).
|
||||
- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
|
||||
(the renderer injects them from `match.nodes`).
|
||||
- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
|
||||
- NO `--reasoning-parser` / `--tool-call-parser` in any cell — parsers are a
|
||||
Playground-only feature added on top of the base command (DSv4 convention);
|
||||
flag any cell that bakes them in.
|
||||
- Accuracy-degrading flags in cells — runtime quant below the checkpoint
|
||||
(e.g. MegaMoE W4A4 — DSv4 gates it behind the Playground's `megamoeQuant`)
|
||||
and lossy `--kv-cache-dtype` (e.g. `fp8_e4m3` over a higher-precision-KV
|
||||
checkpoint): **flag for explicit maintainer confirmation**. Output quality
|
||||
should be exactly what the quant chip declares, so absent a recorded
|
||||
sign-off in the PR (e.g. carried verbatim from a measured legacy recipe's
|
||||
default command), request the flag move to Playground/tips.
|
||||
- Flag order: `--model-path` first (an optional `--trust-remote-code` may precede it —
|
||||
the DSv4 cells do), then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||
last (the playground's insert anchors assume this).
|
||||
- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
|
||||
(BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
|
||||
|
||||
### 4. Benchmarks
|
||||
- Each `benchmarks[]` entry's `match` tuple corresponds to a real cell.
|
||||
- `accuracyLabels` is present whenever the benchmarks carry accuracy data — the engine
|
||||
ships NO default eval set; without it the accuracy rows silently don't render.
|
||||
`defaultAccuracy` / per-cell `accuracy` / `benchmarkCommands.accuracy` keys all
|
||||
∈ `config.accuracyLabels`.
|
||||
- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
|
||||
that only released FP8/FP4 is a factual bug.
|
||||
- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
|
||||
from the `sglang serve` deploy command, and should carry `--flush-cache`: bench_serving's
|
||||
`random` prompts are deterministic, so a warm rerun hits the radix cache and inflates
|
||||
throughput — speed numbers are measured cache-cold.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
- **Latency percentile**: `config.latencyPercentile` (default `"P50"`, or `"Mean"`) matches the
|
||||
percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. A benchmarks
|
||||
entry may carry its own `latencyPercentile` to override the page value per cell
|
||||
(entry → config → `"P50"`): on a P50 page, kept legacy Mean cells must set it — a
|
||||
`sglang_version` tag alone doesn't convey the percentile. (`"Mean"` is temporary — legacy
|
||||
data is being re-measured to P50.)
|
||||
- **Throughput convention**: `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU**
|
||||
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
|
||||
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||
column must be produced by the SAME harness — flag a page that, say, measures one
|
||||
platform's GSM8K with a historical raw completion harness and another's with
|
||||
`sgl-eval run gsm8k --num-examples 1319` and shows both as one "GSM8K %"
|
||||
(the scores aren't comparable). Either standardize on one harness (matching
|
||||
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
|
||||
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
|
||||
|
||||
### 5. Doc ↔ config parity (the #1 finding)
|
||||
- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
|
||||
equal what the engine emits from the corresponding cell — same flags, same order. Drift
|
||||
here is the most common review miss.
|
||||
|
||||
### 5b. ComfyUI section (diffusion pages)
|
||||
- A diffusion page ends with `## <n>. Run in ComfyUI` rendering `<ComfyUISupport />`. A
|
||||
reader must not have to guess whether the model is reachable from ComfyUI.
|
||||
- The `model` prop is a key that exists in `docs/src/snippets/diffusion/comfyui-support.jsx`.
|
||||
A model-specific key is only correct when the plugin really treats it specially — an entry
|
||||
in `executor_class_dict`
|
||||
(`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a dedicated
|
||||
node. Otherwise the generic `image` / `video` key is the honest one; a model-specific key
|
||||
without matching plugin support makes the page claim support that does not exist.
|
||||
- Prose describing ComfyUI support inline instead of using the component is a finding: the
|
||||
facts drift from the plugin.
|
||||
|
||||
### 6. Commands / port
|
||||
- Launch uses `sglang serve` — flag any `python -m sglang.launch_server` /
|
||||
`python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`;
|
||||
guard against prose/cells reintroducing the old launcher.
|
||||
- Port `30000` everywhere (launch, curl, client `base_url`, bench) — flag `8000`.
|
||||
Launch port must match client/curl port on the same page.
|
||||
|
||||
### 7. Frontmatter
|
||||
- Every new MDX page has `title:` and a **top-level** `description:` (a real one-line value
|
||||
prop, not copied from another vendor) — NOT `metatags.description` (non-canonical; the
|
||||
top-level field is what renders as the subtitle and SEO meta — see mintlify-authoring).
|
||||
- **No `mode: wide` on a model page** — it hides the right-hand "On this page" ToC that every
|
||||
other model page has. Leave `mode` unset (the Deploy/Playground panels self-cap at 900px, so
|
||||
the default column holds them fine). `mode: wide` belongs only on category `intro.mdx` grids.
|
||||
- `tag: NEW` only for genuine new launches; when one is added, stale `tag: NEW` on older
|
||||
pages should be dropped in the same PR (`grep -RlE "^tag: NEW" docs/cookbook/`).
|
||||
- MDX imports BOTH `Deployment` and `Playground` from `/src/snippets/...` (absolute).
|
||||
- Deploy heading slugs to `deployment` (or `deploy`), Playground to `playground` — so
|
||||
"↑ Switch base" and "Open the Playground →" scroll. No numbered headings for these two.
|
||||
|
||||
### 8. Navigation & homepage
|
||||
- New page → `docs/docs.json` updated: under the right vendor group inside
|
||||
`navigation` → Cookbook → Autoregressive Models, root-relative, **no `.mdx`**:
|
||||
`cookbook/<category>/<Vendor>/<Model>`.
|
||||
- Homepage `<Card href>` in `docs/cookbook/<category>/intro.mdx` points to the vendor's
|
||||
flagship; new vendors get a new `<Card>` + a logo at `docs/cards/logos/<vendor>.png` —
|
||||
**940×525 RGBA transparent, icon-only (no wordmark)**, lowercase filename, tracked via
|
||||
`git add -f` (`*.png` is gitignored repo-wide). Card order matches the `docs.json` nav order.
|
||||
- Don't change `docs/cookbook/intro.mdx` for individual model adds (top-level only).
|
||||
|
||||
### 9. Links & factual
|
||||
- HuggingFace URLs resolve to a real model. License section matches the actual HF license
|
||||
(don't copy from another model). Docker images from `lmsysorg/sglang`; no `sgl-project-dev`.
|
||||
The image **tag** is a real build (a release the author ran, or `:dev`/nightly) — not a
|
||||
guessed version.
|
||||
- Internal links root-relative, no extension (`/cookbook/.../<Model>`); flag `.md`/`.mdx`
|
||||
or `../`-relative links. `docs.sglang.io` is canonical.
|
||||
- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
|
||||
not `${VAR}` (a bash no-op).
|
||||
- **Parser ids must exist in the code registries** on the PR's target branch: every
|
||||
`--reasoning-parser X` / `--tool-call-parser Y` named in prose or in
|
||||
`playgroundFeatures.parsers` flags is a registered key in
|
||||
`python/sglang/srt/parser/reasoning_parser.py` (DetectorMap) /
|
||||
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
|
||||
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
|
||||
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
|
||||
(`python/sglang/srt/parser/template_detection.py`) actually resolve THIS model's
|
||||
chat template to the right parser — no rule match means auto silently disables the
|
||||
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
|
||||
|
||||
### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
|
||||
- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
|
||||
blocks (`> [!NOTE]`), markdown **pipe tables** (use JSX `<table>`), inline `<details>`,
|
||||
or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
|
||||
- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
|
||||
the opening fence); a fenced block nested inside another uses four backticks outside.
|
||||
- §3 commands and outputs are **collapsible** (DeepSeek-V4 pattern): every runnable
|
||||
example wrapped in an `<Accordion>`, its real output in a following
|
||||
`<Accordion title="Example Output">` (`Pending update...` only with user
|
||||
acknowledgement). Flag bare/inline example blocks and `**Output Example:**` headings.
|
||||
- Reasoning-parser example matches the parser's **output shape**: separate-field
|
||||
(`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
|
||||
- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
|
||||
`generation_config.json` defaults); listing them in §1 informationally is fine.
|
||||
|
||||
### 10. Quantization rules
|
||||
- **NVFP4** checkpoints are Blackwell-only (B200/B300/GB300) — never AMD. An AMD FP4 cell
|
||||
is legitimate ONLY when the vendor published an **MXFP4** checkpoint for it (e.g.
|
||||
`amd/Qwen3.5-397B-A17B-MXFP4` on MI355X) — verify the HF repo resolves; otherwise the
|
||||
AMD FP4 chip must be absent/`disabled`.
|
||||
- BF16 / FP8 work on NVIDIA and AMD. `--kv-cache-dtype fp8_e4m3` in a cell is an
|
||||
accuracy-degrading flag — see §3 (needs explicit maintainer sign-off; default
|
||||
home is Playground/tips).
|
||||
|
||||
### 11. Scope
|
||||
- Changes match the PR title. Flag global changes hiding behind a platform-specific title
|
||||
(e.g. an "H200 FP8" PR that adds a flag to ALL cells). Unmentioned side-fixes belong in
|
||||
the PR body.
|
||||
|
||||
### 12. Duplicate PRs
|
||||
- Another open PR for the same model? Flag it; compare completeness; note merge-conflict
|
||||
risk on `docs.json` + the vendor card; flag a superseded older PR by the same author.
|
||||
|
||||
### 13. Build / validate
|
||||
```bash
|
||||
cd docs
|
||||
mint validate
|
||||
mint broken-links
|
||||
```
|
||||
Optional: `mint dev` for a visual smoke test.
|
||||
|
||||
### 14. Reviewer feedback
|
||||
- `gh api repos/sgl-project/sglang/pulls/<N>/comments` — have prior reviewer requests been
|
||||
addressed? Unresolved requested-changes should be flagged.
|
||||
|
||||
### 15. Grammar & spelling
|
||||
- Check added/changed prose for typos and grammar (e.g. "recommend" vs "recommended").
|
||||
Flag each with the exact wrong text + correction.
|
||||
|
||||
## Output
|
||||
|
||||
Per file:
|
||||
- ✅ PASS
|
||||
- ⚠️ ISSUE: \<what\>
|
||||
- 🔴 BLOCK: \<what\>
|
||||
|
||||
Overall: **APPROVE** / **REQUEST CHANGES** / **BLOCKED**
|
||||
@@ -1,203 +0,0 @@
|
||||
---
|
||||
name: env-var-conventions
|
||||
description: Conventions for SGLang environment variables — where to define, how to access, how to name, and how to deprecate. Use when adding, renaming, or reviewing any `SGLANG_*` environment variable (or migrating a legacy `SGL_*` alias), or when touching `python/sglang/srt/environ.py`.
|
||||
---
|
||||
|
||||
# Environment Variables — Conventions
|
||||
|
||||
Apply this skill when adding, renaming, or reviewing any sglang-owned environment variable (`SGLANG_*`, or a legacy `SGL_*` alias being phased out), or when touching `python/sglang/srt/environ.py`.
|
||||
|
||||
## Rule 1 — Define in the `Envs` class in `python/sglang/srt/environ.py`
|
||||
|
||||
All sglang-owned env vars live as `EnvField` descriptors on the `Envs` class. Never add a new `os.getenv("SGLANG_...")`, `get_bool_env_var("SGLANG_...")`, or `get_int_env_var("SGLANG_...")` call site — the helpers in `python/sglang/srt/utils/common.py` carry an explicit `FIXME: move your environment variable to sglang.srt.environ` and exist only for pre-existing call sites.
|
||||
|
||||
Group the new entry under an existing section comment (e.g. `# Logging Options`, `# Scheduler: recv interval`, `# Flashinfer`). Add a new section comment only when none fits — never drop a new entry at the bottom of an unrelated block.
|
||||
|
||||
### Decision table: register in `Envs` or use `os.getenv`?
|
||||
|
||||
| Variable | Owner | Goes through `Envs`? |
|
||||
|---|---|---|
|
||||
| `SGLANG_*` | sglang | **Always.** The canonical prefix for all new entries. |
|
||||
| `MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` | Upstream/vendor alias that sglang wants to centralize | **Yes** — register in `Envs` so `.get()` / `.override()` work uniformly. Keep the upstream prefix. |
|
||||
| `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `HF_HUB_*` (raw upstream) | External tooling | **No.** Read with `os.getenv` — they're set by the launcher / driver, not by sglang. |
|
||||
| `RANK`, `LOCAL_RANK`, `WORLD_SIZE`, `MASTER_ADDR`, `MASTER_PORT`, `HOME`, `PATH` | Distributed launcher / OS | **No.** `os.getenv` only. |
|
||||
| Test runner internals (`PYTEST_CURRENT_TEST`, etc.) | Test framework | **No.** `os.getenv` only. |
|
||||
|
||||
`SGL_*` is **not** a parallel valid prefix — it's a deprecated legacy alias. `_convert_SGL_to_SGLANG` rewrites `SGL_*` to `SGLANG_*` at import time with a `DeprecationWarning`. Never define a new `SGL_*` descriptor or `os.getenv("SGL_...")` call site; if you see one in code, it's tech debt to migrate.
|
||||
|
||||
The rule of thumb: if the value's lifecycle is owned by sglang code (we read it, we may want to override it in tests, we may want to rename it), put it in `Envs`. If the value is set by something outside sglang and we only consume it as-is, use `os.getenv`.
|
||||
|
||||
## Rule 2 — Pick the typed descriptor
|
||||
|
||||
| Type | Use for |
|
||||
|---|---|
|
||||
| `EnvBool(default)` | boolean flag |
|
||||
| `EnvInt(default)` | integer |
|
||||
| `EnvFloat(default)` | float |
|
||||
| `EnvStr(default)` | string |
|
||||
| `EnvTuple(())` | comma-separated list, parsed via `s.split(",")` and stripped |
|
||||
|
||||
Default value:
|
||||
- For a knob whose "unset" state must be distinguishable from any concrete value, use `None` as the default (e.g. `EnvStr(None)`, `EnvInt(None)`). The descriptor handles set-to-None correctly via `_set_to_none`.
|
||||
- For a feature flag, the default encodes the production behavior. `ENABLE_FOO = EnvBool(False)` means foo is off in prod; `DISABLE_FOO = EnvBool(False)` means foo is on in prod. See Rule 4 on picking the verb.
|
||||
|
||||
### IntEnum for multi-state knobs
|
||||
|
||||
When a knob has more than two discrete states (e.g. an off / soft / strict ladder), define an `IntEnum` next to `Envs` and pass the enum member as the `EnvInt` default. Callers compare against the enum, not raw integers:
|
||||
|
||||
```python
|
||||
class ToolStrictLevel(IntEnum):
|
||||
OFF = 0
|
||||
FUNCTION = 1
|
||||
PARAMETER = 2
|
||||
|
||||
SGLANG_TOOL_STRICT_LEVEL = EnvInt(ToolStrictLevel.OFF)
|
||||
|
||||
# At the call site:
|
||||
if envs.SGLANG_TOOL_STRICT_LEVEL.get() >= ToolStrictLevel.PARAMETER:
|
||||
...
|
||||
```
|
||||
|
||||
Don't pile `SGLANG_ENABLE_FOO_STRICT` / `SGLANG_ENABLE_FOO_OFF` boolean knobs that fight each other — one ordered integer is cleaner.
|
||||
|
||||
## Rule 3 — Access via the `EnvField` API, never raw `os.environ`
|
||||
|
||||
```python
|
||||
from sglang.srt.environ import envs
|
||||
|
||||
if envs.SGLANG_FOO.get():
|
||||
...
|
||||
```
|
||||
|
||||
`envs.SGLANG_FOO` is the descriptor itself; its `__bool__` and `__len__` raise on purpose so that `if envs.SGLANG_FOO:` fails loudly instead of silently reading as truthy. The `.get()` is mandatory at every read site.
|
||||
|
||||
### Full API surface
|
||||
|
||||
| Method | Use |
|
||||
|---|---|
|
||||
| `.get()` | Read value (parsed). Returns `default` if unset, or `None` if explicitly set to None. |
|
||||
| `.set(value)` | Set value. `set(None)` flips the internal `_set_to_none` flag so the next `.get()` returns `None`, not `default`. |
|
||||
| `.clear()` | Unset entirely. Next `.get()` returns `default`. |
|
||||
| `.is_set()` | True iff the key is present in `os.environ` (regardless of value, including the explicit-None case). |
|
||||
| `.override(value)` | Context manager — set on enter, restore exactly what was there on exit. Use this in tests. |
|
||||
|
||||
The `_set_to_none` distinction matters: `clear()` → `is_set()=False, get()=default`; `set(None)` → `is_set()=True, get()=None`. A few descriptors (e.g. `SGLANG_TEST_MAX_RETRY = EnvInt(None)`, `SGLANG_DISAGGREGATION_THREAD_POOL_SIZE = EnvInt(None)`) rely on this to distinguish "user opted out" from "default applies".
|
||||
|
||||
### Test overrides
|
||||
|
||||
```python
|
||||
with envs.SGLANG_TEST_RETRACT.override(True):
|
||||
...
|
||||
```
|
||||
|
||||
Don't mutate `os.environ` directly in tests — `override` restores the original cleanly, including the explicit-None state, even if the block raises.
|
||||
|
||||
For multiple overrides composed dynamically, use `ExitStack`:
|
||||
|
||||
```python
|
||||
with ExitStack() as stack:
|
||||
for name, value in test_envs.items():
|
||||
stack.enter_context(getattr(envs, name).override(value))
|
||||
run_test()
|
||||
```
|
||||
|
||||
### Subprocess inheritance
|
||||
|
||||
`override` mutates the real `os.environ`, so child processes spawned **inside** the `with` block inherit the override. This is the supported way to seed a `subprocess.Popen`:
|
||||
|
||||
```python
|
||||
with envs.SGLANG_TEST_RETRACT.override(True):
|
||||
subprocess.Popen([...]).wait()
|
||||
```
|
||||
|
||||
A child started outside the `with` block sees the original value.
|
||||
|
||||
### `temp_set_env` is for non-sglang keys only
|
||||
|
||||
The module-level `temp_set_env(**env_vars)` helper exists for overriding **non-sglang** env vars (e.g. `CUDA_LAUNCH_BLOCKING`, `NCCL_DEBUG`) in tests. It explicitly rejects `SGLANG_*` / `SGL_*` keys:
|
||||
|
||||
```python
|
||||
# Wrong — raises ValueError
|
||||
with temp_set_env(SGLANG_TEST_RETRACT="true"): ...
|
||||
|
||||
# Right — sglang keys go through the descriptor
|
||||
with envs.SGLANG_TEST_RETRACT.override(True): ...
|
||||
|
||||
# Right — non-sglang keys go through temp_set_env
|
||||
with temp_set_env(CUDA_LAUNCH_BLOCKING="1"): ...
|
||||
```
|
||||
|
||||
The `allow_sglang=True` escape hatch exists for the rare case where you must bypass `Envs` (e.g. setting an env var **name** that's only constructed at runtime); don't use it just to skip writing a descriptor.
|
||||
|
||||
## Rule 4 — Naming: `SGLANG_` prefix + verb category
|
||||
|
||||
Prefix is always `SGLANG_` for new entries. `SGL_*` is auto-translated to `SGLANG_*` with a `DeprecationWarning` in `_convert_SGL_to_SGLANG`; never add a new `SGL_*` key.
|
||||
|
||||
The second token signals intent. Pick the right verb up front — renames require an alias entry (Rule 5).
|
||||
|
||||
| Verb | Meaning | Example |
|
||||
|---|---|---|
|
||||
| `ENABLE_FOO` | Knob that turns feature foo on/off. Default in the `EnvBool` encodes prod behavior. | `SGLANG_ENABLE_TORCH_COMPILE`, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM` |
|
||||
| `DISABLE_FOO` | Kill-switch. `DISABLE_FOO=True` turns foo off. | `SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP` |
|
||||
| `USE_FOO` | Selects which implementation / backend | `SGLANG_USE_AITER`, `SGLANG_USE_DEEPGEMM_BMM` |
|
||||
| `FORCE_FOO` | Overrides autodetection | `SGLANG_FORCE_FP8_MARLIN`, `SGLANG_FORCE_STREAM_INTERVAL` |
|
||||
| `LOG_FOO` | Logging-only knob | `SGLANG_LOG_GC`, `SGLANG_LOG_MS` |
|
||||
| `TEST_FOO` | Test-only hook | `SGLANG_TEST_RETRACT`, `SGLANG_TEST_MAX_RETRY` |
|
||||
| `DEBUG_FOO` | Debug-only instrumentation | `SGLANG_DEBUG_MEMORY_POOL`, `SGLANG_DEBUG_SYMM_MEM` |
|
||||
| `OPT_FOO` | Perf-optimization toggle (heavily used by DSV4 work) | `SGLANG_OPT_USE_FUSED_HASH_TOPK`, `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2` |
|
||||
|
||||
Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only forbidden combination is `DISABLE_FOO = EnvBool(True)`, because it produces a true double-negative at the call site (`if not envs.SGLANG_DISABLE_FOO.get():` reads as "if not disabled"). All other combinations are fine:
|
||||
|
||||
| Pattern | Call site | Verdict |
|
||||
|---|---|---|
|
||||
| `ENABLE_FOO = EnvBool(False)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — opt-in feature |
|
||||
| `ENABLE_FOO = EnvBool(True)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — on in prod, user opts out via `False` |
|
||||
| `DISABLE_FOO = EnvBool(False)` | `if not envs.SGLANG_DISABLE_FOO.get():` | OK — single negation, reads as "if enabled" |
|
||||
| `DISABLE_FOO = EnvBool(True)` | `if not envs.SGLANG_DISABLE_FOO.get():` | **Forbidden** — true double-negative |
|
||||
|
||||
`SGLANG_*` is the canonical sglang prefix. Vendor-integration keys (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`) keep their upstream prefix and live in the same `Envs` class — these are integration aliases, not sglang-owned feature flags.
|
||||
|
||||
## Rule 5 — Renames go through `*WithAlias` or `_print_deprecated_env`
|
||||
|
||||
For a rename where the old key must keep working with a warning:
|
||||
|
||||
```python
|
||||
SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME")
|
||||
```
|
||||
|
||||
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
|
||||
|
||||
For a full removal where the env var is going away, add to `_convert_SGL_to_SGLANG`:
|
||||
|
||||
```python
|
||||
_print_deprecated_env("SGLANG_OLD_NAME", "SGLANG_NEW_NAME") # mapped to a replacement
|
||||
_print_deprecated_env("SGLANG_OLD_NAME") # no replacement, gone
|
||||
```
|
||||
|
||||
For env-var to CLI-flag migration, add at module top-level:
|
||||
|
||||
```python
|
||||
_warn_deprecated_env_to_cli_flag(
|
||||
"SGLANG_FOO",
|
||||
"Please use '--foo' instead.",
|
||||
)
|
||||
```
|
||||
|
||||
Don't silently flip a default during a rename. If the new default disagrees with the old one, that's a behavior change — call it out in the PR body separately from the rename.
|
||||
|
||||
## Rule 6 — Env var vs CLI flag
|
||||
|
||||
| If the knob is… | Goes in |
|
||||
|---|---|
|
||||
| User-facing (documented, expected to flip per deployment) | `server_args.py` CLI flag |
|
||||
| Expert toggle, A-B kill-switch, vendor integration | `environ.py` env var |
|
||||
| Test / debug hook | `environ.py` env var with `TEST_` / `DEBUG_` prefix |
|
||||
| Temporary env var rolling out to a CLI flag | env var first, then migrate via `_warn_deprecated_env_to_cli_flag` |
|
||||
|
||||
Don't add a CLI flag that just forwards to an env var, and don't add an env var that duplicates an existing CLI flag. Pick one surface.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **External / vendor env vars consumed raw** (`HF_HUB_*`, `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `RANK`, `MASTER_ADDR`, etc.): see the decision table in Rule 1 — `os.getenv` is correct, don't pull them into `Envs`.
|
||||
- **Pre-existing `get_bool_env_var(...)` / `get_int_env_var(...)` call sites**: leave them as is; new code shouldn't add more, but mass-migration is out of scope for a feature PR.
|
||||
- **Upstream-aliased keys already in `Envs`** (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` — see Rule 1 decision table): the `SGLANG_` prefix rules in Rule 4 don't apply — the upstream prefix is the canonical name.
|
||||
@@ -1,142 +0,0 @@
|
||||
---
|
||||
name: generate-profile
|
||||
description: Generate an e2e profiling trace of an SGLang server run. Launches a server, validates accuracy, captures a Chrome-compatible trace, and returns the profile path.
|
||||
---
|
||||
|
||||
# Generate an E2E Profile of an SGLang Server Run
|
||||
|
||||
This skill launches an SGLang server, validates it with a quick accuracy test, generates a profiling trace, and returns the profile file path.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||
- At least one available CUDA GPU
|
||||
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
|
||||
|
||||
## Step-by-step Workflow
|
||||
|
||||
### Step 1: Launch the server
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=<gpu_id> sglang serve --model-path <model> --port <port> &
|
||||
```
|
||||
|
||||
- Default model: `Qwen/Qwen3-8B` (good balance of speed and quality)
|
||||
- Default port: `30000`
|
||||
- The server runs in the background. Save the PID for cleanup.
|
||||
- Use the GPU specified by the user's preferences (check memory files for GPU preferences).
|
||||
|
||||
### Step 2: Wait for server readiness
|
||||
|
||||
Poll the health endpoint until the server is ready:
|
||||
|
||||
```bash
|
||||
for i in $(seq 1 120); do
|
||||
if curl -s http://127.0.0.1:<port>/health 2>/dev/null | grep -q "ok\|healthy"; then
|
||||
echo "Server ready"
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
```
|
||||
|
||||
The server prints **"The server is fired up and ready to roll!"** to stdout when ready. The health endpoint returns 200 once the server can accept requests.
|
||||
|
||||
Typical startup time: 30-90 seconds depending on model size and whether CUDA graphs are being compiled.
|
||||
|
||||
### Step 3: Validate accuracy (sanity check)
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
|
||||
```
|
||||
|
||||
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
### Step 4: Generate the profile
|
||||
|
||||
```bash
|
||||
python3 -m sglang.test.send_one --profile
|
||||
```
|
||||
|
||||
This command:
|
||||
1. Sends a request to the server
|
||||
2. Triggers the profiler for 5 steps (default)
|
||||
3. Generates a trace file under `/tmp/<timestamp>/`
|
||||
4. The trace directory contains:
|
||||
- `<timestamp>-TP-0.trace.json.gz` — Chrome trace format (open in `chrome://tracing` or Perfetto)
|
||||
- `server_args.json` — the server configuration used
|
||||
|
||||
**Output format:**
|
||||
```
|
||||
Dump profiling traces to /tmp/<timestamp>
|
||||
```
|
||||
|
||||
The profile path is printed to stdout. Parse it from the output.
|
||||
|
||||
**Optional flags:**
|
||||
- `--profile-steps N` — number of profiling steps (default: 5)
|
||||
- `--profile-by-stage` — profile by stage (prefill/decode separately)
|
||||
- `--profile-prefix <path>` — custom output prefix
|
||||
|
||||
### Step 5: Kill the server
|
||||
|
||||
```bash
|
||||
pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
|
||||
```
|
||||
|
||||
Wait a moment and verify no sglang processes remain:
|
||||
```bash
|
||||
sleep 2 && pgrep -af "sglang serve" || echo "Server killed"
|
||||
```
|
||||
|
||||
### Step 6: Report the profile path
|
||||
|
||||
Return the profile directory path (e.g., `/tmp/1773999986.4769795`) and list its contents so the user knows what files were generated.
|
||||
|
||||
## Example Full Run
|
||||
|
||||
```bash
|
||||
# 1. Launch server
|
||||
source cleanup/bin/activate
|
||||
CUDA_VISIBLE_DEVICES=1 sglang serve --model-path Qwen/Qwen3-8B --port 30000 &
|
||||
|
||||
# 2. Wait for ready
|
||||
for i in $(seq 1 120); do
|
||||
curl -s http://127.0.0.1:30000/health | grep -q "ok" && break
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
|
||||
# Expected: score > 80%
|
||||
|
||||
# 4. Profile
|
||||
python3 -m sglang.test.send_one --profile
|
||||
# Output: "Dump profiling traces to /tmp/1773999986.4769795"
|
||||
|
||||
# 5. Cleanup
|
||||
pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
|
||||
sleep 2
|
||||
|
||||
# 6. Check output
|
||||
ls -la /tmp/1773999986.4769795/
|
||||
# 1773999986.4851577-TP-0.trace.json.gz (Chrome trace)
|
||||
# server_args.json (server config)
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
- **Different port**: Pass `--port <port>` and use `--host 127.0.0.1 --port <port>` for test commands
|
||||
- **Multi-GPU**: Use `--tp <N>` for tensor parallelism; trace files will be generated per TP rank
|
||||
- **Longer profile**: Use `--profile-steps 10` for more steps in the trace
|
||||
- **Stage profiling**: Use `--profile-by-stage` to separate prefill and decode phases
|
||||
|
||||
## Viewing the Profile
|
||||
|
||||
Open the `.trace.json.gz` file in:
|
||||
- **Perfetto UI**: https://ui.perfetto.dev/ (drag and drop the file)
|
||||
- **Chrome tracing**: `chrome://tracing` (load the file)
|
||||
|
||||
Both support the gzipped Chrome trace format natively.
|
||||
@@ -1,105 +0,0 @@
|
||||
---
|
||||
name: kernel-organization
|
||||
description: Apply the SGLang kernels RFC when adding, moving, splitting, or reviewing kernel APIs, registry metadata, kernel tests, benchmarks, and model-specific implementations. Use with add-jit-kernel, add-sgl-kernel, and write-sglang-test for placement and migration checks.
|
||||
---
|
||||
|
||||
# Kernel organization
|
||||
|
||||
Read `python/sglang/kernels/README.md` and `test/README.md` before choosing a
|
||||
location. [RFC #29630](https://github.com/sgl-project/sglang/issues/29630)
|
||||
establishes the namespace; subsequent migrations #32148 and #40922 clarify
|
||||
logical grouping and remove the old `_jit_` filename prefix.
|
||||
|
||||
## Choose the owner
|
||||
|
||||
- Put callable SGLang kernel APIs under `sglang.kernels.ops.<group>`. Runtime
|
||||
and integration tests import from that namespace, including its submodules.
|
||||
The `sgl_kernel` wheel retains its own public API and packaging tests.
|
||||
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
|
||||
routing in `moe`, attention index selection in `attention`, sampling in
|
||||
`sampling`, normalization in `layernorm`.
|
||||
- Classify a fused operator by its complete contract; do not split a fused
|
||||
kernel into separate launches just to separate its parts. Split unrelated
|
||||
public entry points that happen to share a source file. Decided rules:
|
||||
- Any cross-rank collective or peer-memory access (all-reduce, all-gather,
|
||||
reduce-scatter, a2a, multicast/NVLS) puts the op in `communication`,
|
||||
whatever compute it fuses.
|
||||
- Producer + output quantization stays with the producer (norm + quant in
|
||||
`layernorm`, activation + quant in `activation`, GEMM with a quant epilogue
|
||||
in `gemm`, MoE dispatch/permute + quant in `moe`). `quantization` holds only
|
||||
pure format conversion: quant, dequant, scale-layout conversion, repack.
|
||||
- RoPE stays in `attention` (standalone RoPE, position ids, QK-norm + RoPE).
|
||||
- `kvcache` owns ops that read/write KV-cache contents; `attention` owns the
|
||||
index/shape/schedule tables (page tables, `cu_seqlens`, `kv_indices`, split
|
||||
metadata) and index selection. A fused compute + store stays with its
|
||||
compute group.
|
||||
- Model-specific files/subpackages and tuning data are allowed **inside** a
|
||||
logical group. Do not add a model bundle or an implementation file directly
|
||||
under `ops/`. Propose a new logical group only for a distinct responsibility,
|
||||
and update `ops._GROUPS`, the README group list, and tests together
|
||||
(`test_kernels_namespace.py` checks the README against `_GROUPS` and the
|
||||
README layout against the directories under `kernels/`).
|
||||
- Agent-generated implementations live in `kernels/kda_kernels/` (Kernel Design
|
||||
Agents, `KernelBackend.KDA`) and `kernels/cake_kernels/` (Cake adapters over
|
||||
FlashInfer). They are backends, not groups: register their public ops in the
|
||||
logical group with a `KernelSpec` whose target points into them, and do not
|
||||
reorganize those directories.
|
||||
- Keep shared build/runtime infrastructure (including the Helion runtime
|
||||
helper) in `kernels/jit`; operator wrappers call it from their logical group.
|
||||
Keep process groups, communicator state, workspace classes, model dispatch,
|
||||
and buffer ownership in `srt` (`srt/layers/communication/` for the K3 and Muse
|
||||
Spark adapters); kernels must not import srt runtime modules to inspect
|
||||
model objects, so pass flags in. Adapters there need not pretend to be generic
|
||||
interfaces.
|
||||
|
||||
## Register the public entry point
|
||||
|
||||
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
|
||||
existing `BaseFusedOp` registration when the operation has interchangeable
|
||||
backends (a `BaseFusedOp` built per layer registers `module:Class.forward_<backend>`
|
||||
targets). Group imports must not import GPU implementations or compile kernels.
|
||||
|
||||
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
|
||||
existing backend dispatch and describe actual device/architecture restrictions
|
||||
with `CapabilityRequirement`; JIT/AOT are not device types. Input shape/dtype
|
||||
checks remain part of the entry point's contract. Registering a torch custom op
|
||||
does not register it in the SGLang kernel inventory. Predicates, private JIT
|
||||
factories, reference helpers, and runtime classes are not separate kernel APIs.
|
||||
|
||||
## Place and preserve tests
|
||||
|
||||
- Kernel numerical tests: `test/registered/kernels/ops/<group>/`.
|
||||
- CI microbenchmarks: `test/registered/kernels/benchmark/<group>/`.
|
||||
- Inventory/selector tests: `test/registered/unit/kernels/`.
|
||||
- Runtime unit tests: `test/registered/unit/<subsystem>/`, following the tested
|
||||
runtime module. A CUDA allocation does not make a runtime test a kernel test.
|
||||
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
|
||||
standalone test/benchmark entry points in production operator modules.
|
||||
- Shared test helpers and reference oracles: `sglang.test.kernels`. Preserve
|
||||
vendored upstream trees and AOT wheel packaging boundaries instead of
|
||||
reorganizing them incidentally.
|
||||
|
||||
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
|
||||
execution entry points, and CI stages/runners. Apportion existing time estimates
|
||||
across split files; do not duplicate the original budget for every output file
|
||||
or silently drop a registration. Follow `write-sglang-test` for CI registration.
|
||||
|
||||
## Verify a migration
|
||||
|
||||
1. Search all runtime, test, benchmark, documentation, patch-string, and lazy
|
||||
registry references before deleting the old path. Check relative imports and
|
||||
package re-exports as well as direct imports. Do not leave forwarding shims.
|
||||
2. Keep tuning files with the GEMM that loads them and preserve relative lookup
|
||||
behavior. Check wheel/package inclusion as well as source-tree execution.
|
||||
3. Preserve module singleton state: every runtime caller must import the same
|
||||
new communication adapter, not a second copy of its buffer registry.
|
||||
4. Separate relocation commits from semantic changes. Use
|
||||
`mechanical-refactor-verify` to reproduce moves, and compare test inventories
|
||||
and CI registration coverage before/after. Only delete an experimental path
|
||||
after checking its call sites, flags, source, and dedicated tests together.
|
||||
5. Run the namespace/dispatch CPU tests, the registered-test validation hook,
|
||||
pre-commit, and relevant GPU tests when available. State exactly which GPU
|
||||
checks ran; import/AST checks do not prove numerical or performance parity.
|
||||
|
||||
Do not infer violations solely from a model name inside a group, a missing
|
||||
`_jit_` prefix, use of a runtime utility, or absence of `BaseFusedOp` inheritance.
|
||||
@@ -1,249 +0,0 @@
|
||||
---
|
||||
name: processor-model-parity
|
||||
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
|
||||
---
|
||||
|
||||
# Processor model parity
|
||||
|
||||
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
|
||||
server, sgl-router, the renderer). A model is supported only when, for the same
|
||||
request body, the processor produces **the same prompt text and the same prompt
|
||||
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
|
||||
the implementation, and SGLang code exists only where the two differ.
|
||||
|
||||
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
|
||||
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
|
||||
Multimodal preprocessing is not covered here: `render_request` fails on media
|
||||
content parts, so hosts send those chats to the engine. The fixture generator
|
||||
serves every checkpoint text-only, so a media case records that rejection.
|
||||
|
||||
## Done means
|
||||
|
||||
- **All text should work**: every text chat the engine accepts renders and
|
||||
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
|
||||
the processor cannot reproduce goes to the engine; list it in the PR.
|
||||
- **Chat and completions go through the engine's `/generate`**, with the router
|
||||
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
|
||||
engine's OpenAI route is only the `Unsupported` fallback. That needs the
|
||||
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
|
||||
against a live engine.
|
||||
|
||||
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
|
||||
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
|
||||
to `rust/sglang-processor/` unless they start with `python/`.
|
||||
|
||||
## Where a model's code goes
|
||||
|
||||
| Piece | File | Needed when |
|
||||
|---|---|---|
|
||||
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
|
||||
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
|
||||
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
|
||||
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
|
||||
| README row | the `render` table in `README.md` | always |
|
||||
|
||||
No new test code: `tests/parity.rs` checks every fixture in the directory.
|
||||
|
||||
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
|
||||
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
|
||||
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
|
||||
renders the model:
|
||||
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
|
||||
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
|
||||
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
|
||||
`PromptFormatter`, replace that branch.
|
||||
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
|
||||
variant holding Dynamo's Jinja `PromptFormatter`, picked in
|
||||
`ChatFormatter::from_tokenizer_config` from the template text (as
|
||||
`parser/template_detection.py` picks it, where it has a detector), and build the
|
||||
template context that `_process_messages` and `_apply_jinja_template` build, like
|
||||
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
|
||||
rewrites shapes its load-time probes find the template rejects (a later system
|
||||
turn, for one); SGLang raises on them, so check those before rendering.
|
||||
|
||||
## Method
|
||||
|
||||
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
|
||||
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
|
||||
Note whether its tokenizer ships a `chat_template`; that can change the spec.
|
||||
1. **Trace the Python path for the model.** Write down every transformation
|
||||
between the HTTP body and `prompt_ids`, in order:
|
||||
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
|
||||
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
|
||||
Reuse `render/reasoning.rs` for this rather than porting it again.
|
||||
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
|
||||
replaces the request effort; server `--default-chat-template-kwargs` merge in.
|
||||
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
|
||||
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
|
||||
It also reads the server's `--tool-call-parser` and whether the tokenizer has
|
||||
a chat template. If the deployment depends on the parser, set
|
||||
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
|
||||
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
|
||||
effort profile reads `encoding/encoding_dsv4.py`).
|
||||
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
|
||||
message dumping, content flattening, `_handle_last_assistant_message`, system
|
||||
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
|
||||
call, and how the prompt is tokenized.
|
||||
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
|
||||
rules live there: dropped turns, `</think>` on empty reasoning, argument
|
||||
formatting, and the errors it raises.
|
||||
|
||||
This list is the **divergence inventory**.
|
||||
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
|
||||
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
|
||||
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
|
||||
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
|
||||
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
|
||||
which often differ from SGLang's. Diff the low-level encoder against Python's
|
||||
for the rules in step 1.
|
||||
3. **Write one request case per inventory item** in the fixture (`name` +
|
||||
`request` only), then generate the expected outputs from Python (below). Cover
|
||||
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
|
||||
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
|
||||
each failing case, add the smallest SGLang step that fixes it, with a one-line
|
||||
doc comment naming the Python source it mirrors. Never patch the fixture to
|
||||
match Rust.
|
||||
5. **Verify** text offline, then token ids with the checkpoint cached (commands
|
||||
below). If sgl-router or the renderer uses the formatter, run their suites too.
|
||||
|
||||
## Checklist (DeepSeek-V4 case that covers each)
|
||||
|
||||
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
|
||||
reduced to role and content; null content becomes `""`; history tool calls keep
|
||||
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
|
||||
- **Content parts**: which part types survive and the join separator. (`parts`)
|
||||
- **Tools**: all request tools or only those `tool_choice` allows; field order
|
||||
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
|
||||
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
|
||||
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
|
||||
`tool_schema_type_aliases`)
|
||||
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
|
||||
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
|
||||
- **Pydantic coercion**: typed request fields are coerced before rendering. For
|
||||
example, `strict: 1` and `defer_loading: "false"` become booleans, and
|
||||
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
|
||||
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
|
||||
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
|
||||
JSON string; other encoders format them their own way. Keep key order
|
||||
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
|
||||
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
|
||||
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
|
||||
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
|
||||
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
|
||||
truthiness (`1` is on); strings use Python's yes-word set.
|
||||
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
|
||||
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
|
||||
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
|
||||
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
|
||||
a separately tokenized prefix. Check whether content is flattened before the
|
||||
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
|
||||
assistant turn's tool calls are discarded, so object checks on their arguments
|
||||
belong after the split. Hosts reach the model through `render_prompt` too, so
|
||||
check their continuation handling as well.
|
||||
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
|
||||
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
|
||||
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
|
||||
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
|
||||
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
|
||||
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
|
||||
continuation prefix is encoded alone and loses a leading BOS
|
||||
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
|
||||
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
|
||||
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
|
||||
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
|
||||
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
|
||||
includes pydantic rejections and Python crashes (a 500 is still a rejection).
|
||||
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
|
||||
PR that hosts fall back to the engine for it.
|
||||
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
|
||||
reach, fix it upstream and mark the case `known_gap` with the reason. The test
|
||||
reports the case without failing, and fails once it matches so the marker gets
|
||||
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
|
||||
|
||||
## Generating fixtures
|
||||
|
||||
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
|
||||
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
|
||||
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
|
||||
renders it through the real `_process_messages`, under the case's
|
||||
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
|
||||
writes:
|
||||
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
|
||||
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
|
||||
are kept as written). An `AttributeError` aborts instead: the stub server lacks
|
||||
something, so extend it;
|
||||
- the fixture-level `config` (the `config.json` fields the processor reads, plus
|
||||
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
|
||||
checkpoint its `chat_template`, so the test renders offline.
|
||||
|
||||
```sh
|
||||
cd rust/sglang-processor
|
||||
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
|
||||
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
|
||||
```
|
||||
|
||||
When a new spec needs more, extend `serving_chat()` in the script (one place):
|
||||
- **More server state:** set the extra attribute that `__init__` resolves, for
|
||||
example `_dsv41_default_reasoning_effort`.
|
||||
|
||||
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
|
||||
recorded by the template's text output. Its `render_request` returns the
|
||||
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
|
||||
|
||||
Pin `revision` to a commit hash and regenerate only on purpose.
|
||||
|
||||
## Verify
|
||||
|
||||
```sh
|
||||
cd rust
|
||||
cargo test -p sglang-processor --locked # text, offline
|
||||
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
|
||||
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
|
||||
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
|
||||
```
|
||||
|
||||
The token check loads the pinned commit's snapshot from the cache. When the
|
||||
snapshot is missing, the test prints `token ids not checked` (visible with
|
||||
`-- --nocapture`), so confirm that line is absent before claiming token parity.
|
||||
With the snapshot cached, the test also checks that the checkpoint resolves the
|
||||
fixture's recorded DeepSeek-V4 profile.
|
||||
|
||||
## OpenAI layer
|
||||
|
||||
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
|
||||
once its render parity holds. Two more fixtures then cover the rest of Python's
|
||||
OpenAI layer:
|
||||
- `tests/fixtures/reasoning_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
|
||||
chunked outputs. A parser is served once `src/think/models/` ports it.
|
||||
- `tests/fixtures/tool_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
|
||||
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
|
||||
parser is served once `src/tool_call/models/` ports it.
|
||||
- `tests/fixtures/openai_parity/<model-id>.json`, from
|
||||
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
|
||||
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
|
||||
output for it, and the response Python builds from that output. Launch the
|
||||
engine with the model's cookbook flags, including its parsers. Without the
|
||||
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
|
||||
fixture does.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
|
||||
`dynamo-renderer` or `dynamo-tokenizers`.
|
||||
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
|
||||
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
|
||||
sent, so such requests stay on the engine's route. Fixtures don't show this; a
|
||||
live engine does.
|
||||
- Python mutates the request before rendering (`_validate_request` rewrites tool
|
||||
schemas), so fixture generators drive the server's own entry points.
|
||||
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
|
||||
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
|
||||
generator pins them and `tests/parity.rs` clears them, so fixtures do not
|
||||
depend on the machine.
|
||||
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
|
||||
such as `task` and message-level `tools`. Parity is defined on `render_request`;
|
||||
report host-adapter gaps separately rather than bending the model code.
|
||||
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
|
||||
beyond the dispatch arm.
|
||||
@@ -1,665 +0,0 @@
|
||||
---
|
||||
name: sglang-runtime-context
|
||||
description: How SGLang's runtime configuration and process-global state are organized (RuntimeContext tiers, publish + namespace config bags, the pristine ServerArgs seed, override entry points, resource/stream/buffer leases, per-forward flags), the CI guardrails that enforce the design, and the idioms for developing and testing against it. Load this before touching server_args, model overrides, module-level state, or per-forward state in sglang.
|
||||
---
|
||||
|
||||
# SGLang runtime-context architecture
|
||||
|
||||
One container owns process-static runtime state: `sglang.srt.runtime_context.RuntimeContext`
|
||||
(a process singleton reached via `get_context()`). Everything below is a tier on it.
|
||||
|
||||
| Tier | Accessor | Holds | Lifecycle |
|
||||
|------|----------|-------|-----------|
|
||||
| raw config seed | `get_server_args()` | the published `ServerArgs` — the startup record, for debugging, dumps and provenance. **Business code does not read fields off it**: the read ratchet pins that at zero, and "Reading config: the seed is off limits" below says what to read instead, which forms the ratchet sees, and what is outside it by construction (a runtime-computed name; a whole-object hand-off) | published at process entry; re-publish is **last-publish-wins** (the tokenizer publish in the launcher process; sequential engine rebuild in one process, e.g. unit tests) and re-projects the bags; read-only |
|
||||
| resolved config | `get_exec()` `get_memory()` `get_schedule()` `get_model()` `get_spec()` `get_serving()` `get_observability()` `get_disagg()` `get_lora()` `get_mm()` `get_device()` | namespace **config bags** — the single source of truth for resolved config; leaves are real attributes (dynamo-traceable). Each is a **module function of no arguments**, and a module binds the name once: `manager.get_disagg()`, `self.get_disagg = get_disagg`, or a same-named import next to the bag one (`from model_loader import get_model`) all import fine and fail only when that path runs. `ruff --select F811` catches the import collision; `RuntimeContext` has no bag-named member and no `__getattr__`, so the member-call shapes are an `AttributeError` at call time — give it a delegating `__getattr__` and they go silent instead | projected at `publish` from the declarations over `server_args`' raw fields; mutated only via `get_context().override` |
|
||||
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
|
||||
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
|
||||
| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
|
||||
| parallel | `get_parallel()` | one spelling per name: scoped override, then permanent stamp, then published parallel config leaf | ranks: stamped at publish; group handles: stamped at distributed init; configured and derived widths: after publish |
|
||||
|
||||
`reset_context()` (unit-test teardown) drops the published config and installs fresh
|
||||
flags/resources/forward tiers.
|
||||
|
||||
## Config: publish + namespace bags
|
||||
|
||||
**`ServerArgs` holds the raw input and nothing else. Resolution writes no field:
|
||||
it declares, and the declarations are what the namespace bags are projected from.
|
||||
Business code never reads the record for a decision: a field read there answers
|
||||
with what the operator typed, not with what resolution decided.**
|
||||
|
||||
- Every publishing process entry calls `publish(server_args, role=...)`
|
||||
(`run_scheduler_process`, the Ray `SchedulerActor`, the DP controller, tokenizer,
|
||||
detokenizer, encoder, weight-cache daemon, the multi-tokenizer worker, the
|
||||
spawned encoder TP/DP workers, the benchmark work functions, ...); constructors
|
||||
do not publish — `ModelRunner`, `TokenizerManager` and `MMEncoder` call
|
||||
`assert_published` and fail loudly if an entry forgot. The roles are enumerated once,
|
||||
as the keys of `ROLE_NAMESPACE_SETS` — there is no `launcher` role, the launch
|
||||
path publishes as `tokenizer`. The remaining non-publisher is
|
||||
`run_multi_detokenizer_router_process`: it *is* handed a `ServerArgs`, and uses
|
||||
it only for `configure_logger(server_args)` today, so it has nothing to publish
|
||||
for — a bag read added under that entry needs a `publish` at the entry first.
|
||||
`publish` projects the config bags from the declarations over the record's raw
|
||||
fields; the accessors (`get_exec()` etc.) fail closed before it
|
||||
runs. `role` records which process type published, and keys per-role namespace
|
||||
enforcement: `SGLANG_ROLE_NAMESPACES=record` audits which namespaces each role's
|
||||
process actually reads (per-pair persisted via `SGLANG_ROLE_NAMESPACES_OUT`;
|
||||
reads inside torch.compile-traced code are NOT observed — audit with
|
||||
compilation disabled before restricting a role), and
|
||||
`=enforce` fails closed on bag reads outside the role's `ROLE_NAMESPACE_SETS` entry
|
||||
(`None` = full tree; only audited roles are restricted).
|
||||
- Bag membership is **where the field is declared**: one class per namespace under
|
||||
`arg_groups/fields/`, each carrying the `_NS_PATH` it stands for, and `ServerArgs`
|
||||
is assembled from them (`collect_input_fields`). The per-field `NS("path")` marker
|
||||
survives only for a class that cannot express this — an ad-hoc dataclass spanning
|
||||
namespaces, which is what the config-bag tests build. Coverage is linted two-way
|
||||
(`test_server_args_namespaces.py`, `test_runtime_context_config_bags.py`).
|
||||
- **Reading config**: `get_<ns>()[.sub].field` — e.g.
|
||||
`get_exec().moe.moe_a2a_backend`, `get_schedule().max_running_requests`. Bag leaves
|
||||
are plain instance attributes, safe inside `torch.compile`-traced code.
|
||||
- **Mutating config after publish**: the ONLY entry point is
|
||||
`get_context().override(source, **fields)`. It writes the bag leaves in place
|
||||
(namespace readers see the new value) and records provenance in the overrides log.
|
||||
There is **no write-through** to the `ServerArgs` instance — it stays pristine.
|
||||
There is no in-place mutation entry on the instance at all: it is read-only after
|
||||
resolution.
|
||||
- **Reading a leaf when the caller holds the field *name*** (a readback endpoint,
|
||||
a control-plane handler): `get_context().config_leaf(name)` — the read side of
|
||||
`override`. It resolves the flat name through the same `NS` map the write side
|
||||
uses and raises on a name that is not a config leaf. Code that knows its field
|
||||
when it is written reads the bag leaf directly; `config_leaf` is for
|
||||
name-driven code, not a way around the seed ratchet.
|
||||
- **Post-startup control-plane changes** — a weight update, a HiCache mirror
|
||||
attach, a parser resolved from the chat template — go through
|
||||
`TokenizerManager.record_config_updates(source, **fields)`, a named wrapper
|
||||
over `get_context().override`. One process keeps one log: the request dumps
|
||||
ship `get_context().overrides_log()`, and `config_value(name)` /
|
||||
`resolved_config_dict(base)` answer from the bags. Fields recorded through
|
||||
the wrapper follow the same ordering rules as a direct `override`: a later
|
||||
read must observe the updated bag, not the startup record.
|
||||
- **`model_path` and `served_model_name` are answered off the manager.** Both are
|
||||
`NS` leaves and `override` accepts them, but the tokenizer-side weight reload
|
||||
records only `load_format` and writes the two path fields as `TokenizerManager`
|
||||
attributes (`_MANAGER_OWNED_FIELDS`); `config_value` and `resolved_config_dict`
|
||||
overlay them on top of the bags. Bags do not cross a process boundary (above),
|
||||
so recording those two in the tokenizer process would leave every other
|
||||
process's bag on the old path while the log claimed a process-wide change. The
|
||||
scheduler rewrites its own copy where the reload happens —
|
||||
`ModelRunner.update_model_fields` overrides `model_path` / `load_format` for
|
||||
the target runner.
|
||||
- **Late launcher-stage resolution (pre-publish)**: a few rules cannot run inside
|
||||
`ServerArgs.resolve_once` — LoRA normalization, and the auto-parser detection that needs a
|
||||
tokenizer/chat-template load. They are resolution, not mutation, and they
|
||||
**declare** via `arg_groups.overrides.declare_resolution(server_args, source,
|
||||
**fields)`, the same call the rest of the pipeline makes; there is no
|
||||
`declare_late_resolution` any more. *When* a declaration is made is not
|
||||
something the code marks. The declaration lands
|
||||
in the stash on that very object, so every holder of it carries the decision —
|
||||
the HTTP server, the multi-tokenizer workers it is serialized for, the
|
||||
schedulers it forks — and each of them publishes bags projected from it. The
|
||||
fields stay the operator's input; `resolution_result(sa, field)` and the bags
|
||||
are what answer for the decision. Returning a variant here is a bug: the
|
||||
launcher rebinds its local and everyone else keeps the unresolved object.
|
||||
- **A value another runner / worker owns is a constructor argument, not a config
|
||||
copy.** The draft worker's `context_length`, load format and attention backend
|
||||
travel as arguments to `TpModelWorker` / `ModelRunner` and live on the runner
|
||||
(`ModelRunner.draft_attention_backend`, `kv_cache_dtype_str`, …); the encoder
|
||||
DP worker's device is `MMEncoder(gpu_id=...)`. There is no `ServerArgs.derive`
|
||||
any more — a config object is never copied-and-edited; test doubles that need a
|
||||
modified copy use `sglang.test.test_utils.server_args_variant`.
|
||||
|
||||
**Why a bag override cannot stand in for late resolution or per-runner
|
||||
construction.** The bags are projected at
|
||||
publish *from the declarations over the instance's raw fields*, so anything the
|
||||
runtime must read has to be declared before publish — an override afterwards puts instance and bags back out
|
||||
of agreement, and whole-object readers (`ModelConfig.from_server_args`,
|
||||
`build_load_config`, `MMEncoder`'s own `self.server_args.X`) never see it. And bags do
|
||||
not cross a process boundary: a child publishes from the object it receives and
|
||||
re-projects its own bags, so a parent-side override is lost. Values that feed
|
||||
construction before any bag exists (group init reads `server_args.tp_size`) have no
|
||||
bag to override at all.
|
||||
|
||||
|
||||
### Reads that legitimately stay on a `ServerArgs` instance
|
||||
|
||||
- **Per-runner values** — there is no per-runner `ServerArgs` any more. The
|
||||
draft-worker config copy is gone: every worker (`TpModelWorker`, the draft
|
||||
workers in `speculative/`) is handed the *same* instance the process published,
|
||||
so a bag leaf is the decision and `self.server_args.X` is the operator's
|
||||
input — a
|
||||
post-publish `override` moves only the bag, which is exactly why a field that
|
||||
is process-wide config (`attention_backend`, `skip_tokenizer_init`,
|
||||
`kv_cache_dtype`) reads from the bags like any other, and why a residual
|
||||
instance read on this path is stale the moment someone overrides that leaf.
|
||||
What is genuinely per-runner travels two ways, neither of them a config
|
||||
instance: **constructor arguments** (`ModelRunner(draft_attention_backend=...)`,
|
||||
`MMEncoder(gpu_id=...)`) and **runner attributes holding the resolved value**
|
||||
(`model_runner.kv_cache_dtype_str`, `prefill_attention_backend_str`,
|
||||
`linear_attn_backends`) — threaded to consumers as
|
||||
arguments, never backfilled onto a shared object. A per-runner choice also stays
|
||||
*out* of the bags: recording it there is how a second runner inherits the first
|
||||
one's answer, which is exactly the bug `linear_attn_backends` replaced. The two
|
||||
sanctioned bends in that rule are *scoped*. `ModelRunner._load_format_scope`
|
||||
exposes the draft's `--speculative-draft-load-format` through
|
||||
`get_model().override(load_format=...)` for exactly the duration of the draft
|
||||
build, because model construction reads that bag leaf — the override restores
|
||||
on exit, so nothing outlives the scope. The shared-expert layout is another scoped view:
|
||||
`get_model().shared_experts` selects the immutable `SharedExpertsConfig` owned
|
||||
by the loaded model (`model.shared_experts_config`). The loader binds it for
|
||||
construction, and runners bind it for execution, warmup and graph capture.
|
||||
Both the fused count and ordinary TopK slot count belong to this object;
|
||||
neither is a separately mutable flag. A scope selects a built layout and
|
||||
restores the previous reference on exit, including exceptions. It cannot
|
||||
resize weights, buffers or a captured graph. When the model is in hand, read
|
||||
its owned config directly (as the routed-experts capturer does).
|
||||
- **Per-instance boundaries** — the tokenizer-manager family, everything under
|
||||
`entrypoints/`, and the tokenizer-process multimodal processors read the bags.
|
||||
The old justification for keeping them on `self.server_args` ("several
|
||||
`Engine`s can share one process, bags are last-publish-wins across them") is
|
||||
**retracted** — owner ruling (2026-08-15): a process holds at most one live
|
||||
config at a time (concurrent multi-Engine is unsupported; sequential rebuild
|
||||
stays legal, unit tests rely on it). Nothing in those files reads the instance
|
||||
any more; review new instance reads against the raw-input contract. What
|
||||
genuinely stays per-instance is what differs per *worker* within one engine:
|
||||
`base_gpu_id` travels as a constructor argument (`MMEncoder(gpu_id=...)`;
|
||||
`BaseMultimodalProcessor._fast_image_processor_device` is the shape to copy).
|
||||
- **Whole-object passes** (`f(server_args)` handing the instance along) keep the
|
||||
supplied-instance contract; don't rewrite the parameter reads unless the
|
||||
field is runtime-mutated (see the elastic-EP `ep_size` case in
|
||||
`eplb/expert_location.py`) — **or the field is one that resolution fills in
|
||||
and the callee runs in a process that has published.** That second case is a
|
||||
decision, not a style question: the record carries the user's raw input, so a
|
||||
resolution-filled field read off it inside a runner-owned constructor answers
|
||||
with the pre-resolution value instead of the effective one. The answer is not
|
||||
automatically a bag read: pick where the value should come from — usually the
|
||||
`get_*()` bag, sometimes a runner stamp or a constructor argument (the per-mode
|
||||
attention pair and the encode-server `gpu_id` above are both this). The per-instance
|
||||
boundaries above are **not** exempt from this unless-clause (the multi-Engine
|
||||
exemption is retracted); each one gets its own disposition.
|
||||
Check direct attributes, `getattr`, and records stored on `self`; validate
|
||||
the resolved value and any later overrides in behavior tests.
|
||||
Two shapes stay parameter-form on purpose: a helper the
|
||||
*resolution pipeline* calls with a `resolved_view` (its parameter happens to be
|
||||
named `server_args`), and a factory whose contract is "build X from the record
|
||||
you are handed" (`create_kt_config_from_server_args`, `DllmConfig.from_server_args`).
|
||||
|
||||
### Four ways a config sweep breaks something no test runs
|
||||
|
||||
Each of these shipped in a review round and cost a real defect. Where a test
|
||||
guards one it is named; the next sweep checks all four by hand first.
|
||||
|
||||
1. **The other implementations of an interface.** Dropping a parameter means
|
||||
auditing implementers, not just callers: `CustomSpecAlgo` is the plugin
|
||||
base for speculative algorithms, and the dispatch calls it with the
|
||||
built-in's argument list. Nothing in the tree implements it, so only a
|
||||
plugin user hits the `TypeError`.
|
||||
Guard: `test_plugin_hook_signatures.py`.
|
||||
2. **Publish order inside a process entry, not per file.** A file containing a
|
||||
`publish` says nothing about whether a given read runs before it. Spawned
|
||||
workers (`MMEncoder` for encoder DP/TP, the Ray scheduler actor) start with
|
||||
an empty context, so a bag read above the publish raises only there.
|
||||
Guarded on the launch path only (`test_pre_publish_readers.py`); other
|
||||
entries, spawned workers included, are checked by review.
|
||||
3. **The role namespace a process publishes under.** `ROLE_NAMESPACE_SETS`
|
||||
narrows what each role may read; the DP controller is audited for `exec`
|
||||
alone. A helper that reaches for another namespace passes every default-mode
|
||||
test and aborts startup under `SGLANG_ROLE_NAMESPACES=enforce`. Prefer
|
||||
answering from the caller's own namespaces over widening the set.
|
||||
4. **Sibling surfaces of a readback.** Changing what one entry point reports
|
||||
means enumerating the others: HTTP, gRPC and in-process `Engine` each have
|
||||
their own server-info and model-info, and each passes its own tests while
|
||||
its users lose the field.
|
||||
Guarded for server info across HTTP, both gRPC roles and the Rust frontend
|
||||
(`TestServerInfoTransportParity` in `entrypoints/test_server_info.py`);
|
||||
`Engine.get_server_info` and model info are checked by review.
|
||||
|
||||
A fifth, from the same rounds: the accessor **name itself**. Called as an
|
||||
object member (`manager.get_disagg()`), or shadowed by a same-named import
|
||||
(`from model_loader import get_model` next to the model bag, where the later
|
||||
import silently wins and the loader call gets a zero-argument bag), it imports
|
||||
fine and fails only when that path runs. The invariant is one line: the name
|
||||
means the process-wide bag, takes no arguments, and is bound once per module.
|
||||
`ruff --select F811` catches the import collision; the member-call shapes are an
|
||||
`AttributeError` at call time only because `RuntimeContext` has no bag-named
|
||||
member and no `__getattr__` -- a delegating `__getattr__` would make them
|
||||
silent, and that is when this needs a guard again rather than a rule.
|
||||
|
||||
Write these guards over a **derived** set, never a hand-kept list: an entry
|
||||
naming a function that no longer exists, or a field list missing the one field
|
||||
nobody migrated, passes green forever. Both happened here -- a `_ENTRY_POINTS`
|
||||
row for a method the Ray actor does not have, and an effective-field set
|
||||
without `load_format` -- and both were invisible because the assertion had
|
||||
slack (`>= len(...) - 1`) or compared key names instead of value sources.
|
||||
|
||||
### `get_parallel()`: one spelling per name
|
||||
|
||||
**There is no `.config` hop.** `ParallelContext._read` answers every name in
|
||||
one order: a scoped override, then a permanent stamp, then the published
|
||||
`parallel` bag leaf. `publish` stamps ranks from the launcher; distributed init
|
||||
stamps group handles and validates their widths. Configured widths such as
|
||||
`tp_size`, `pp_size`, `attn_cp_size`, `dcp_size` and `moe_dp_size`, and config-only
|
||||
leaves such as `nccl_port`, `dp_size`, `ep_size` and `dwdp_size`, fall back to the
|
||||
published bag. Reading a config leaf before publish raises a `ValueError`
|
||||
naming the namespace; an unknown name is an `AttributeError`.
|
||||
|
||||
A size reads from the configuration because the groups are built at exactly the
|
||||
configured widths — checked at every assignment to `_TP` / `_PP` / `_ATTN_CP` /
|
||||
`_DCP` / `_MOE_DP` in `parallel_state.py`. Three things do not follow that rule:
|
||||
|
||||
- `initialize_model_parallel` aliases `_MOE_DP` to `_ATTN_CP` when `attn_cp_size >
|
||||
moe_dp_size`, so a reader that means **the MoE communicator's width** calls
|
||||
`get_moe_cp_size()`, not `get_parallel().moe_dp_size`.
|
||||
- `patch_tensor_parallel_group` runs a scope under a different TP group (draft
|
||||
workers), and declares it by overriding `tp_size`, `tp_rank` and `tp_group`
|
||||
for the scope's duration. Readers inside need no special spelling.
|
||||
- Elastic EP scales `ep_size` / `dp_size` on the published bag while the group
|
||||
coordinators keep their construction width. Those are different names, not two
|
||||
answers to one name.
|
||||
|
||||
DCP keeps its own pair: `get_parallel().attn_dcp_size` / `.dcp_enabled` are derived
|
||||
from the published configuration unless explicitly overridden; `dcp_size` is
|
||||
what the launch requested.
|
||||
|
||||
A process-global seed field-read of one of these sizes
|
||||
(`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure. A
|
||||
`server_args` the object was *handed* is a different thing and not a ratchet
|
||||
matter — see "Reads that legitimately stay on a ServerArgs instance".
|
||||
Fail-loud is narrower: an unstamped rank or group handle with no declared default
|
||||
raises; ranks are available after publish, before dist init. The six
|
||||
parallel quotients are not live reads at all — `attn_tp_size`, `attn_dp_size`,
|
||||
`attn_dcp_size`, `moe_ep_size`, `moe_tp_size`, `dcp_enabled` are a function of the
|
||||
configured leaves, computed once at publish into bag leaves, and answered
|
||||
override → stamp → published leaf. So `dcp_enabled` means "the launch configured
|
||||
DCP" (`dcp_size > 1`), not "a DCP group is installed here". Normal distributed
|
||||
initialization builds the groups to match the configured widths; it does not
|
||||
restamp these DCP leaves. `test_a_topology_is_stated_by_naming_the_width` and its
|
||||
neighbours in `test_runtime_context.py` pin this; they replaced
|
||||
`test_attn_dcp_defaults_when_group_is_uninitialized`. One consequence for tests:
|
||||
overriding a leaf no longer moves its quotient — state a topology by publishing a
|
||||
config, or by naming the width. After init,
|
||||
only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and
|
||||
moe-DP always install, as size-1 aliases if unused). Bare parallel leaf reads are
|
||||
dynamo-traceable; gate helpers like `enable_moe_dense_fully_dp()` run inside
|
||||
compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins
|
||||
this).
|
||||
|
||||
### Reading config: the seed is off limits
|
||||
|
||||
`get_server_args().field` in business code is a ratchet failure. Read:
|
||||
|
||||
- **a resolved leaf** → its namespace bag (`get_exec().moe.moe_runner_backend`,
|
||||
`get_schedule().chunked_prefill_size`, …). Bag-backed reads — a leaf directly, or
|
||||
a bag-derived accessor below — are what see post-publish overrides. Only the
|
||||
instance-derived accessors (the ones with no leaf to read) answer from the
|
||||
startup record and therefore do not.
|
||||
- **a leaf the caller names at runtime** (a readback reporting a list of fields)
|
||||
→ `get_context().config_leaf(name)`; it resolves the name through `NS` and
|
||||
raises on a non-leaf. A call site that knows its field reads the bag leaf.
|
||||
- **the live topology** → `get_parallel()` (bare names).
|
||||
- **a value derived from published leaves** → an accessor in `runtime_context` that
|
||||
derives it *from the bags*. The strongest form of this is a `Derived(fn=...)`
|
||||
declared beside the leaves it is computed from, in the namespace's own
|
||||
`arg_groups/fields/` class: `publish` computes it once and stores it as an
|
||||
ordinary bag leaf, so the read is a plain attribute load and it sees
|
||||
post-publish overrides. `enable_mamba_extra_buffer`, `is_ep_joiner`,
|
||||
`is_ep_scale_joiner` and `is_startup_weight_load_overlap` are declared that way
|
||||
now — read them where they are declared:
|
||||
`get_exec().mamba.enable_mamba_extra_buffer`, `get_exec().moe.is_ep_joiner`,
|
||||
`get_model().is_startup_weight_load_overlap`. (The namespace is the class that
|
||||
declares the field, not the namespaces its `fn` happens to read: the mamba one
|
||||
spans `exec.mamba` and `memory`, which is exactly why it could not be a method
|
||||
on either bag.) The
|
||||
old `mamba_extra_buffer_enabled()` / `is_ep_joiner()` functions and the
|
||||
same-named `ServerArgs` members are gone. The pre-publish helpers that remain
|
||||
exist for resolution, which has no bag to read yet. `attention_backends()` derives the
|
||||
`(prefill, decode)` pair from the three `exec.kernel` leaves, and
|
||||
`max_speculative_num_draft_tokens()` / `cutedsl_moe_max_num_tokens()` derive
|
||||
theirs from `spec` / `schedule` / `exec.graph`.
|
||||
- **a value only the instance can compute** → the named accessor in
|
||||
`runtime_context`, which is the one module allowed to read the slot:
|
||||
`mamba_cache_chunk_size()`, `mamba_state_chunk_size()`, `uses_mla_backend()`,
|
||||
`process_model_config()`.
|
||||
These have no leaf to read — they combine several fields, the HF config, or a
|
||||
property with no bag of its own. A new derived member gets an accessor here
|
||||
rather than call sites reaching for the record, and only when the bag-derived
|
||||
shape above cannot express it.
|
||||
- **a parallel size** → `get_parallel().{tp,pp,moe_dp,attn_cp,dcp}_size`, which is
|
||||
the parallel bag's own leaf: it answers with the resolved configuration and
|
||||
follows a post-publish override. Two questions are *not* that, and have their
|
||||
own spelling: the width of the MoE communicator you are about to collectively
|
||||
operate on is `get_moe_cp_size()` (the `_MOE_DP = _ATTN_CP` alias makes it
|
||||
differ), and DCP's derived width and enablement are
|
||||
`get_parallel().attn_dcp_size` / `.dcp_enabled` (published configuration
|
||||
unless explicitly overridden).
|
||||
- **this runner's resolved value** → the runner
|
||||
(`prefill_attention_backend_str`, `kv_cache_dtype_str`,
|
||||
`draft_attention_backend`); for what the model's layers bake in at
|
||||
construction, the layer (`num_fused_shared_experts` on a MoE block).
|
||||
|
||||
`self.server_args.field` is still right for handed per-instance config (see
|
||||
"Reads that legitimately stay on a ServerArgs instance" above for the full set —
|
||||
per-instance boundaries and whole-object passes; there are no per-runner config
|
||||
copies to read any more). The allow-list is `GrammarManager` and `MMEncoder`;
|
||||
what sits beside it is residue, not a family — and not for one single reason:
|
||||
|
||||
- the tokenizer-manager family and `entrypoints/` **read the bags**; review a new
|
||||
instance read there against the raw-input contract rather than assuming a
|
||||
directory is off-limits;
|
||||
- `GrammarManager` is a handed instance for its residual `self.server_args`
|
||||
reads, but backend selection is **not** on the instance any more:
|
||||
`create_grammar_backend` reads `get_exec().kernel.grammar_backend`, and
|
||||
`__init__` calls that factory whenever `skip_tokenizer_init` is false. In
|
||||
production the scheduler process has published; a test that constructs one
|
||||
without publishing has to keep patching the factory (or publish itself);
|
||||
- `MMEncoder` publishes the very instance it is handed (`publish(server_args,
|
||||
role="encoder")`) and takes its per-worker device as a separate `gpu_id`
|
||||
argument. Its `self.server_args` reads are on this list as a construction-path
|
||||
convention, and the residual is real: they answer with the raw input, so a leaf
|
||||
resolution decided and a post-publish `override` both pass them by.
|
||||
|
||||
Their tests are not one story: a `GrammarManager` built standalone turns the
|
||||
factory's bag read into "config namespace not published" unless the test patches
|
||||
it or publishes, while `MMEncoder` publishes in its own `__init__` and so needs
|
||||
no such arrangement.
|
||||
|
||||
**Test doubles publish, they do not inject.** A stand-in that carries
|
||||
`server_args=SimpleNamespace(field=...)` stops working the moment production reads
|
||||
the bag; seed the value with `override_server_args`, which publishes only once it is
|
||||
entered or installed — the bare call just builds the override:
|
||||
|
||||
```python
|
||||
override = get_context().override_server_args(field=...)
|
||||
override.install()
|
||||
self.addCleanup(override.restore) # or: with get_context().override_server_args(...):
|
||||
```
|
||||
|
||||
Five separate test files learned this the hard way during the sweep.
|
||||
|
||||
The rule is about a double standing in for **config**: a `SimpleNamespace` that
|
||||
pretends to be `server_args`. Prefer the context override even where a
|
||||
single-accessor stub would work — `override_server_args(...)` composed with the
|
||||
scoped bag / `get_parallel()` overrides expresses the *cause* (the configuration)
|
||||
rather than pinning one helper's answer, and it keeps working when a reader
|
||||
migrates between the accessor and the leaf. The sweep converted the last two
|
||||
accessor stubs to exactly that shape (`test_attention_patching.py` publishes the
|
||||
non-lazy strategy; `test_kimi_k3_vision.py` publishes `tp_size` and forces the
|
||||
live topology through `get_parallel().override`), so no test stubs an accessor
|
||||
today. Stubbing one *named accessor* remains a last resort for a case that
|
||||
isolates one branch of one helper where no published config can reach it —
|
||||
if you do it, say so in the test.
|
||||
|
||||
### Mid-resolution reads (inside the pipeline only)
|
||||
|
||||
Resolution runs in `ServerArgs.resolve_once` and **writes nothing onto the record**: a
|
||||
handler declares (`declare_resolution`), the declaration goes
|
||||
into the stash, and the fields keep what the caller passed. So a mid-resolution
|
||||
read of a field answers with the *raw input* — every reader in the pipeline goes
|
||||
through a view instead:
|
||||
|
||||
- `resolving_view(server_args)` — the live view (walks the
|
||||
stash per read). This is what handlers and hooks bind, conventionally as
|
||||
`cfg = resolving_view(server_args)` at the top of the handler.
|
||||
- `resolved_view(server_args)` — snapshots the overlay when built, which is what
|
||||
a post-process pass wants: it reads the state at *its* slot.
|
||||
|
||||
Resolution hooks and the helpers they call (`ModelConfig`, platform defaults,
|
||||
the spec-algo hook) must read through these views too. Keep coverage in
|
||||
`test_resolution_declarations.py`, `test_resolution_is_reproducible.py`, and
|
||||
`test_record_holds_the_raw_input.py` focused on the values callers observe.
|
||||
|
||||
One consequence worth knowing: because the fields are the raw input, resolving a
|
||||
bare `dataclasses.replace` copy lands in the same place as the parent — the
|
||||
pipeline reads only its own input. **So a resolved record is not copied at
|
||||
all.** A caller that needs one field different for the process it is about to
|
||||
hand the record to — the Ray paths and their `dist_init_addr` — declares it on
|
||||
the record it holds (`declare_resolution`) and hands that over: the declaration
|
||||
travels inside the object, the receiving process projects its bags from it, and
|
||||
nothing re-resolves. There is no `ServerArgs.replace_resolved` any more, and the
|
||||
`model_config`-memo bug that copying used to cause (a copy marked resolved but
|
||||
arriving without the memo cannot refill it, because the guard refuses the write)
|
||||
is gone by construction rather than guarded.
|
||||
|
||||
A bag `override` cannot stand in for this. It is *not* because overriding needs
|
||||
a publish — `set_server_args` is what projects the bags and `override` works as
|
||||
soon as the context holds a record — but because `override` writes bag leaves
|
||||
and by contract never touches the record, so its effect cannot travel inside an
|
||||
object to another process.
|
||||
|
||||
### The declaration stash has one writer
|
||||
|
||||
Everything that decides configuration goes through
|
||||
`declare_resolution(server_args, source, **fields)`. It validates the names,
|
||||
refuses the published config (the stash is projected at publish and never
|
||||
again, so a later declaration is a silent no-op), and appends. The other names
|
||||
around it are spellings, not mechanisms:
|
||||
|
||||
| name | what it adds |
|
||||
|---|---|
|
||||
| `run_post_process_pass` | runs a pass at its slot and validates its return; declares through `declare_resolution`. A pass returning an **empty** dict is a validation, not a declaration, and stays legal on the published instance — `Engine(server_args=sa)` after `Engine.shutdown()` re-runs `check_server_args` on the very instance the context holds |
|
||||
| `record_foreign_defaults` | for a resolver this tree does not own (an out-of-tree platform plugin, a registered speculative algorithm), whose interface is to *assign* fields. It gets a stand-in whose reads fall through to `resolving_view`; what it assigned is declared. The record is never written, so the write seal has no exception. In-tree code does not go through it — `handle_platform_defaults` wraps the platform hook, and the in-tree speculative dispatcher is called directly, because handed the stand-in its own `declare_resolution` calls would stash on that instead |
|
||||
|
||||
`resolution_projection` is gone; the whole-object readback is
|
||||
`ServerArgs.resolved_dict()`, which is what `/server_info` and its gRPC and
|
||||
in-process twins report.
|
||||
|
||||
### Adding a model-specific config adjustment
|
||||
|
||||
Never assign `server_args` fields from model code. Declare instead
|
||||
(`sglang/srt/arg_groups/overrides.py`):
|
||||
|
||||
- Constant per-arch values → `MODEL_OVERRIDES["MyArchForCausalLM"] = {...}`.
|
||||
- Derived values → `@register_model_override("MyArchForCausalLM")` returning a dict; the
|
||||
callable receives *pristine* `server_args` + `hf_config` and must not write.
|
||||
- Normalization that must see earlier declarations → a post-process pass invoked via
|
||||
`run_post_process_pass` at its slot (reads a view, returns a declaration dict).
|
||||
- Values only knowable at load time are **per-runner state**, not declarations:
|
||||
there is no `declare_load_time_override` any more. A model-family decision that
|
||||
its checkpoint drives (shared-experts fusion) is a question the *loader* asks
|
||||
the model class — `shared_experts_fusion_disable_reason(hf_config,
|
||||
quant_config)`, a classmethod answering without an instance — at the single
|
||||
model-instantiation point, together with how many shared experts it fuses
|
||||
(`num_fusable_shared_experts(hf_config)`; an entry class without both hooks
|
||||
fuses none). `resolve_shared_experts_config` returns one immutable config;
|
||||
the loader scopes it through `get_model().override(shared_experts=config)`
|
||||
while building and retains it on the resulting model. `num_shared_topk_slots`
|
||||
derives from the fused count and frozen Waterfill policy, so ordinary routing
|
||||
and Waterfill cannot both claim the shared slots. `TopK`/`HashTopK` retain the
|
||||
same config; `top_k` counts only routed experts at construction.
|
||||
`draft_model_build_scope` still selects draft boundary/MMA preferences; it
|
||||
does not save shared-expert flags. Target and draft each own their layout.
|
||||
These values have model lifetime, so they do not belong in `get_forward()`.
|
||||
A process-level load-time fact (the sm80 dtype fallback —
|
||||
device-driven, identical for every runner) records directly via
|
||||
`get_context().override`.
|
||||
|
||||
Declarable fields form a whitelist: `Arg(..., resolvable=True)` in the `ServerArgs`
|
||||
dataclass. A declaration against a non-whitelisted field fails at its slot.
|
||||
|
||||
### Load-time vs resolution-time (critical)
|
||||
|
||||
`ServerArgs.resolve_once` runs in the launcher process before any model/platform import. Logic that
|
||||
consults an **extensible registry** (e.g. out-of-tree platforms registering attention
|
||||
backends in `init_backend()`, which runs at `model_runner` import) must stay at load time
|
||||
(ModelRunner init), writing through `get_context().override()`. Before moving any
|
||||
load-time logic into resolution, verify everything it reads is already complete at
|
||||
resolution time.
|
||||
|
||||
## Runtime flags (`get_flags()`)
|
||||
|
||||
For state that init-time code *derives* and runtime code reads — parsed enums, platform
|
||||
probes, swappable ACTIVE values. Not for config mirrors (read the bag leaf instead).
|
||||
|
||||
- Groups are typed dataclasses on `Flags` (`capture` / `moe` / `dp`): typo-safe writes,
|
||||
transactional test-only `override(**kw)` context manager.
|
||||
- `flags.moe` is materialized by `initialize_moe_config()` at scheduler init (it
|
||||
reads `exec.moe` / `spec` / `model`, and takes no record);
|
||||
accessors (`get_moe_a2a_backend` etc.) are thin shims with lazy defaults. The speculative
|
||||
contexts (`speculative_moe_backend_context`) swap the ACTIVE leaves around draft forwards.
|
||||
- `flags.dp` is materialized by `initialize_dp_attention`; `is_dp_attention_enabled()` is a
|
||||
shim over `flags.dp.enabled`.
|
||||
- Adding a leaf: declare the dataclass field with a default equal to the pre-init behavior,
|
||||
materialize it at the owning subsystem's init, keep any public accessor as a shim.
|
||||
|
||||
## Resources (`get_resources()`)
|
||||
|
||||
Named slots + two keyed-lazy registries:
|
||||
|
||||
- `get_stream(name)` — get-or-create a named CUDA side stream; `set_stream(name, stream)`
|
||||
installs explicitly. **Name leases by subsystem ROLE**: all model alternate streams share
|
||||
`"alt"`; the offloader's copy stream is `"offload"`; DP-TBO comm is `"dp_tbo_comm"`; LoRA
|
||||
side stream is `"lora_side"`. Two call sites may share a name only if their work belongs
|
||||
on one stream — sharing across roles serializes intended overlap.
|
||||
- `get_buffer(name, factory)` — get-or-create a named persistent buffer. Grow-only or
|
||||
per-device semantics manage their `resources.buffers` entries directly (see SM120
|
||||
split / Marlin workspace). Buffer names are per-backend today; do not silently
|
||||
share.
|
||||
- Singletons with manager semantics (EP dispatcher buffers, EPLB recorder/metadata, graph
|
||||
memory pool) keep their owning accessors/classes as facades; only the *state* lives in a
|
||||
resources entry. Preserve exact semantics in the shim: lazy defaults (the EPLB recorder
|
||||
defaults to a Noop instance, not None), publish-once asserts, event-reuse contracts.
|
||||
- Stream/buffer creation is a driver call — it must happen outside cuda-graph capture;
|
||||
keep lease points at init/warmup time.
|
||||
|
||||
## Per-forward flags (`get_forward()`)
|
||||
|
||||
Contextvar-backed; a new thread sees the defaults; `scoped(**kw)` is the regular write path
|
||||
(transactional, restores on exit and on exception); `set(name, value)` exists for legacy
|
||||
sticky setters (`is_extend_in_batch` is intentionally sticky within a thread). Use this
|
||||
tier for anything set-per-forward and read-within-forward. Before adding cross-thread
|
||||
state here, prove the readers' thread affinity: contextvars do NOT propagate to already-
|
||||
running or newly spawned threads. Note TBO ("two-batch overlap") interleaves ubatches on
|
||||
ONE thread — do not design for TBO threads that don't exist.
|
||||
|
||||
## Testing idioms
|
||||
|
||||
- **Force a code path by overriding causes, not effects**: compose
|
||||
`get_context().override_server_args(**fields)` (publishes a fresh dummy-boundary
|
||||
`ServerArgs` carrying the overrides AND projects the bags — `with`-scoped, or
|
||||
`install()`/`restore()` + `addCleanup` for fixture-lifetime use) +
|
||||
`get_<ns>().override(...)` (scoped override of one bag's own leaves) +
|
||||
`get_parallel().override(...)` (live topology) + `get_flags().<group>.override(...)` +
|
||||
`get_forward().scoped(...)`. All are scoped and transactional. Tests control execution
|
||||
through the context — do not hand-build and publish config objects.
|
||||
- **Never monkeypatch import bindings** (`module.get_x = lambda: ...`) and never fake a
|
||||
config source with a `SimpleNamespace` stand-in: production reads the published bags,
|
||||
so a faked accessor silently stops intercepting after any reader migration. Publish
|
||||
for real (`override_server_args(...)`), then adjust bag leaves with the scoped bag
|
||||
`override` where the constructed `ServerArgs` cannot carry the value (e.g.
|
||||
`get_device().override(device="meta")`). The one carve-out is the deliberate
|
||||
single-accessor stub for isolating one predicate — the terms and the two
|
||||
sanctioned examples live under "Test doubles publish, they do not inject"
|
||||
above; anything wider than one named accessor is this rule.
|
||||
- Mocked runners/managers still need the **per-runner instance attributes** the code
|
||||
under test reads (`kv_cache_dtype_str`, `server_args` for whole-object passes) — set
|
||||
them explicitly on the mock; `MagicMock(spec=...)` raises on attributes that only
|
||||
exist post-`__init__`, which is the fastest way to find a missed stub.
|
||||
- `reset_context()` in teardown when a test publishes outside a scoped override.
|
||||
- `ServerArgs(model_path="dummy")` resolves through the pipeline's early return (few
|
||||
declarations) — fine for lightweight fixtures. The strict guard still applies once
|
||||
`resolve_once` has run, as it does at `publish`.
|
||||
- **Asserting what resolution decided reads `resolution_result(sa, "field")`**, not
|
||||
`sa.field`: the field is the raw input. Assert the field only when the point of
|
||||
the case *is* that the record stayed pristine (the FA4 page-size and waterfill
|
||||
cases do exactly that, and say so).
|
||||
- **Run changed test files per-file** (own process), the way CI does: a monolithic local
|
||||
pytest run lets a context published by an earlier file mask a missing-publish bug in a
|
||||
later one.
|
||||
|
||||
## Guardrails and review rules
|
||||
|
||||
1. **Strict mutation guard** (always on, and with no exception): bare
|
||||
`server_args.x = ...` after resolution raises unconditionally in
|
||||
`ServerArgs.__setattr__` — the named lift that out-of-tree plugins used to
|
||||
ask for is gone, they assign onto a stand-in instead — this *is* the guarantee that
|
||||
no writer can desync the bags, so there is no writer ratchet any more. Change
|
||||
resolved config with `get_context().override`; hand a per-runner value to its
|
||||
runner as a constructor argument. Projected bags are sealed the same way (leaf
|
||||
assignment raises).
|
||||
2. **No textual mutation ratchet**: the strict guard above is the only mutation
|
||||
check.
|
||||
3. **No-copy contract** (review only, no automated guard): neither
|
||||
`ServerArgs.override` nor `ServerArgs.derive` exists, and nothing in the package
|
||||
calls either form. Rerouting a writer to the bags means flipping **all its readers
|
||||
in the same commit** (no transitional dual-write).
|
||||
4. **The legacy accessor is retired** (`test_runtime_context.py`): every
|
||||
`get_global_server_args()` call now raises, because it answered with the record --
|
||||
a caller reading a field resolution had decided got a stale value and no error.
|
||||
The replacement for a *decision* is a bag leaf, a named accessor, or the owning
|
||||
runner's stamp — not `get_server_args().field`, which the read ratchet below pins
|
||||
at zero. `runtime_context.get_server_args()` is only for the
|
||||
whole-object shapes (dumps, provenance, a hand-off to a callee that takes a config).
|
||||
5. **Global config read check** (`test_global_config_read_ratchet.py`): no business code
|
||||
reads a field off `get_server_args()`, directly or through an alias (function-local
|
||||
— including local copies of an alias, `cfg = sa` — module-level, or parked on an
|
||||
instance attribute, plus the `getattr(..., "field")` spelling of each; a name
|
||||
computed at runtime or indirection deeper than a local name copy is outside the
|
||||
check, per the test's docstring). The scanner matches `get_server_args` by its
|
||||
literal name — bare or module-qualified (`ctx.get_server_args()`) — and
|
||||
`TestNoRenamedAccessorImports` in the same file *bans* `import ... as` renames of it,
|
||||
which is what makes literal-name matching sound. Exempt by owner
|
||||
module only (`runtime_context.py`, `server_args.py`, `arg_groups/`). Two classes,
|
||||
no more: `TestGlobalConfigReads` holds the zero check and
|
||||
`TestNoRenamedAccessorImports` holds the ban. There is no configured-size registry
|
||||
here any longer — `get_parallel()` has one spelling per name, so a size read is not a
|
||||
choice between two answers and nothing needs registering.
|
||||
6. **Module-state rule** (review only, no automated guard): a new module-level
|
||||
runtime global in the flag-owning layers belongs on a flags group / resources slot
|
||||
instead.
|
||||
7. **Namespace coverage** (`test_server_args_namespaces.py`,
|
||||
`test_runtime_context_config_bags.py`): every `ServerArgs` field resolves to a
|
||||
namespace — from the `arg_groups/fields/` class that declares it — and the
|
||||
projected bags must cover the fields exactly (two-way).
|
||||
|
||||
Never module-skip a test "until the migration settles" — seed the context instead
|
||||
(the deferral ratchet that once pinned this is retired; the rule stands).
|
||||
|
||||
## Hard-won pitfalls (check these before/while refactoring)
|
||||
|
||||
- **Moving code drops first-line guards**: early returns (`if self.is_draft_worker: return`)
|
||||
are the easiest thing to lose when relocating a method body. A draft is built from
|
||||
the target's published config — there is no draft config copy and no nested publish
|
||||
any more — so a body moved out of a draft-aware call site keeps reading the target's
|
||||
bags, and only that guard tells the two apart. What the draft build *does* scope is
|
||||
narrower and named: `draft_model_build_scope()` for the MoE fusion gates,
|
||||
`speculative_moe_backend_context()` for the runner backends.
|
||||
- **Registry-completeness timing**: a gate that consults an extensible list is only correct
|
||||
after the registrars ran (platform `init_backend()` at module import). See "load-time vs
|
||||
resolution-time".
|
||||
- **Late function-scope imports shadow module names** for the WHOLE function
|
||||
(UnboundLocalError at earlier lines). Audit moves with AST, not grep.
|
||||
- **Lease names are per-role**, not per-API-shape (the offloader-vs-"alt" lesson).
|
||||
- **Storage matrix for state read inside torch.compile-traced model code**
|
||||
(piecewise cuda graph compiles the whole model forward): contextvars are
|
||||
untraceable (hard error); dict-slot values are guarded per value — for a
|
||||
per-forward int that is one recompile per distinct size, straight into the
|
||||
recompile limit; **class/instance attributes are the only compile-friendly
|
||||
form** (attribute-source ints get automatic-dynamic after the first size
|
||||
change). Bools (≤2 values) are tolerable in any form — see
|
||||
`ForwardFlags._GRAPH_VISIBLE`. Config-bag leaves are real instance attributes for
|
||||
exactly this reason. Parallel leaves are the exception that was measured rather
|
||||
than assumed: they come through `ParallelContext.__getattr__`, which traces
|
||||
under `torch.compile(fullgraph=True)` (`object.__getattribute__` is the form
|
||||
that graph-breaks, and it is not on this path). Before moving such state,
|
||||
prove its readers sit outside compile coverage; a piecewise-prefill boot of a small
|
||||
model is the fast check (recompile storms show as `torch._dynamo hit
|
||||
config.recompile_limit` during the compile pass).
|
||||
- **Engine-booting e2e tests are the only coverage for launcher-path code**; a child crash
|
||||
kills the process tree and pytest dies silently — run with `PYTHONUNBUFFERED=1` and read
|
||||
child logs.
|
||||
- CI arms `SGLANG_ENABLE_ASYNC_ASSERT=1` (device-side `torch._assert_async` probes, e.g.
|
||||
KV-cache OOB): a fired device assert kills the tree with no Python traceback, and the
|
||||
same bug is *silent corruption* locally with the flag off. Arm it when reproducing CI
|
||||
crashes.
|
||||
- CI startup logs print the full `server_args=ServerArgs(...)`; diffing that dump between
|
||||
runs is the fastest config-divergence check.
|
||||
|
||||
## Where to read the code
|
||||
|
||||
Key source files: `python/sglang/srt/runtime_context.py` (the container, every tier,
|
||||
`publish`, `_ConfigBag`, `override_server_args`),
|
||||
`python/sglang/srt/arg_groups/overrides.py` (override registry, passes,
|
||||
`declare_resolution` and the spellings around it), `python/sglang/srt/server_args.py` (`NS` metadata,
|
||||
`Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under
|
||||
`test/registered/unit/` (`test_global_config_read_ratchet.py`,
|
||||
`test_server_args_namespaces.py`,
|
||||
`test_runtime_context.py` — the last one doubles
|
||||
as executable documentation of every tier's semantics).
|
||||
@@ -1,442 +0,0 @@
|
||||
---
|
||||
name: write-sglang-test
|
||||
description: Guide for writing SGLang CI/UT tests. Covers CustomTestCase, CI registration, server fixtures, model selection, mock testing, and test placement. Always read test/README.md for the full CI layout, how to run tests, and extra tips. Use when creating new tests, adding CI test cases, writing unit tests, or when the user asks to add tests for SGLang features.
|
||||
---
|
||||
|
||||
# Writing SGLang CI / UT Tests
|
||||
|
||||
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||
The implementation tutorial below does not replace that API contract.
|
||||
|
||||
This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fail-fast, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md). Whether a case is worth adding at all is decided by [`unit-test-admission`](../../rules/unit-test-admission.md) — read it before writing the case, not after.
|
||||
|
||||
## Core Rules
|
||||
|
||||
1. **Always use `CustomTestCase`** — never raw `unittest.TestCase`. It ensures `tearDownClass` runs even when `setUpClass` fails, preventing resource leaks in CI.
|
||||
2. **`tearDownClass` must shut the server down gracefully** — call `terminate_and_kill_process_tree(cls.process)`, never a bare `kill_process_tree`. SIGKILL alone skips the server's userspace cleanup and leaves its GPU memory charged to the dead process; the next class then OOMs while loading weights. Keep it defensive too: `hasattr`/null checks before accessing resources (e.g. `cls.process`) that `setUpClass` may not have finished allocating.
|
||||
3. **Place non-kernel tests in `test/registered/<kind>/<subsystem>/`** — `<kind>` is `unit`, `e2e`, `accuracy`, `perf`, or `stress`; kernel tests use `test/registered/kernels/{ops,benchmark}/<group>/`; hardware belongs in registrations, not directory names
|
||||
4. **Reuse server fixtures** — inherit from `DefaultServerBase` or write `setUpClass`/`tearDownClass` with `popen_launch_server`
|
||||
5. **Mock boundaries, not SGLang behavior** — mock slow or external dependencies only when the assertion still checks an observable result, state transition, or error. A test whose evidence is only `assert_called*` mirrors its mock and is not admissible. Launch a real server only when inference results or lifecycle behavior are the contract under test.
|
||||
|
||||
> **Existing files are not the reference.** About 290 test files still call the bare
|
||||
> `kill_process_tree(cls.process.pid)`, against 43 on the current helper. They predate
|
||||
> rule 2 and are being migrated, so grepping the repo for a teardown pattern finds the
|
||||
> wrong one roughly seven times out of eight. The same goes for `register_cuda_ci(suite=...)`:
|
||||
> four files still pass it, all of them under `test/registered/stress/`.
|
||||
|
||||
```python
|
||||
# Bad: kill_process_tree(cls.process.pid) # SIGKILL only; GPU memory lingers
|
||||
# Good: terminate_and_kill_process_tree(cls.process)
|
||||
|
||||
# Bad: register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small")
|
||||
# Good: register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small")
|
||||
```
|
||||
|
||||
JIT kernel notes:
|
||||
- If the task is adding or updating code under `python/sglang/kernels/jit/`, prefer the `add-jit-kernel` skill first.
|
||||
- JIT kernel correctness tests use `test/registered/kernels/ops/<group>/test_*.py`.
|
||||
- JIT kernel benchmarks use `test/registered/kernels/benchmark/<group>/bench_*.py`.
|
||||
- Those files are executed by `test/run_suite.py` through dedicated kernel suites (`base-b-kernel-*`); a `register_*_ci(...)` call placed under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook.
|
||||
|
||||
---
|
||||
|
||||
## Model & Backend Selection
|
||||
|
||||
| Scenario | Model | CI Registration | Suite |
|
||||
|----------|-------|-----------------|-------|
|
||||
| **Unit tests** (no server / engine launch) | None | `register_cpu_ci` | `base-a-test-cpu` |
|
||||
| **Common / backend-independent** (middleware, abort, routing, config, arg parsing) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` only | `base-b-test-1-gpu-small` |
|
||||
| **Model-agnostic functionality** (sampling, session, OpenAI API features) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` (+ AMD if relevant) | `base-b-test-1-gpu-small` |
|
||||
| **General performance** (single node, no spec/DP/parallelism) | `DEFAULT_MODEL_NAME_FOR_TEST` (8B) | `register_cuda_ci` | `base-b-test-1-gpu-large` |
|
||||
| **Bigger features** (spec, DP, TP, disaggregation) | Case by case | Case by case | See **Choosing a Suite** below |
|
||||
|
||||
**Key principle for E2E tests**: Do NOT add `register_amd_ci` unless the test specifically exercises AMD/ROCm code paths. Common E2E tests just need any GPU to run — duplicating across backends wastes CI time with no extra coverage.
|
||||
|
||||
### All model constants
|
||||
|
||||
Defined in `python/sglang/test/test_utils.py`:
|
||||
|
||||
| Constant | Model | When to use |
|
||||
|----------|-------|-------------|
|
||||
| `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` | Llama-3.2-1B-Instruct | Common features, model-agnostic tests |
|
||||
| `DEFAULT_SMALL_MODEL_NAME_FOR_TEST_BASE` | Llama-3.2-1B | Base (non-instruct) model tests |
|
||||
| `DEFAULT_MODEL_NAME_FOR_TEST` | Llama-3.1-8B-Instruct | General performance (single node) |
|
||||
| `DEFAULT_MOE_MODEL_NAME_FOR_TEST` | Mixtral-8x7B-Instruct | MoE-specific tests |
|
||||
| `DEFAULT_SMALL_EMBEDDING_MODEL_NAME_FOR_TEST` | — | Embedding tests |
|
||||
| `DEFAULT_SMALL_VLM_MODEL_NAME_FOR_TEST` | — | Vision-language tests |
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
A per-commit suite name is **generated** from registration metadata as `{stage}-test-{runner_config}` — you don't hand-write it:
|
||||
|
||||
- **`stage`** — the CI stage (e.g. `base-b`, `base-b-kernel-unit`, `base-c`).
|
||||
- **`runner_config`** — a runner-pool key from `scripts/ci/runner_configs.yml`, which maps it to the physical runner label (so `1-gpu-large` runs on `1-gpu-h100`). AMD/NPU use their own keys (e.g. `amd`).
|
||||
- **Suite** — `register_cuda_ci(stage="base-b", runner_config="1-gpu-small")` → `base-b-test-1-gpu-small`, the name you pass to `run_suite.py --suite`. The `-test-` is just the connector; never put it in `register_*_ci`.
|
||||
|
||||
> CUDA nightly uses the same shape with `stage="nightly"` (e.g. `stage="nightly", runner_config="1-gpu-large"` → `nightly-test-1-gpu-large`) and **no** `nightly=True` — the stage name carries the cadence, and setting the flag makes the test silently never run. Legacy single-string `suite=` is left only for `stress` and some AMD/CPU/NPU pools.
|
||||
|
||||
### All CI Suites
|
||||
|
||||
Do not work from a list copied into this file; it goes stale silently. Read the
|
||||
current one:
|
||||
|
||||
```bash
|
||||
grep -n "_SUITES = {" test/run_suite.py # PER_COMMIT_SUITES, NIGHTLY_SUITES, OTHER_SUITES
|
||||
cat scripts/ci/runner_configs.yml # runner_config -> physical runner label
|
||||
```
|
||||
|
||||
`scripts/ci/runner_configs.yml` calls itself the single source of truth for the
|
||||
`runner_config` field, and `run_suite.py` is what actually dispatches, so those two
|
||||
files settle any disagreement with prose anywhere else.
|
||||
|
||||
Nightly suites live in `NIGHTLY_SUITES` and run via `nightly-test-nvidia.yml`,
|
||||
`nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. CUDA nightly is
|
||||
named `nightly-test-{runner_config}` — one suite per machine type, holding everything
|
||||
that runs nightly on it, with `auto_partition` splitting the work. There is no
|
||||
per-purpose split; kernel, eval, perf, and precision all share their machine's suite.
|
||||
|
||||
> **Note**: Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
### Choosing a Suite
|
||||
|
||||
Use the lightest suite that meets your test's needs:
|
||||
|
||||
- **No GPU required** → `base-a-test-cpu`
|
||||
- **Most small GPU tests** → `base-b-test-1-gpu-small` (default choice)
|
||||
- **Need H100 memory or Hopper features** → `base-b-test-1-gpu-large`
|
||||
- **JIT kernel correctness** → `base-b-kernel-unit-test-1-gpu-large`
|
||||
- **JIT kernel correctness for B200 / SM100 paths** → `base-b-kernel-unit-test-4-gpu-b200`
|
||||
- **JIT kernel benchmarks** → `base-b-kernel-benchmark-test-1-gpu-large`
|
||||
- **Multi-GPU** → only when the test actually needs multiple GPUs
|
||||
|
||||
---
|
||||
|
||||
## Test File Templates
|
||||
|
||||
### Unit Tests (no server / engine launch)
|
||||
|
||||
See `test/registered/unit/README.md` for quick-start and rules. Unit tests live in `test/registered/unit/`, mirroring `python/sglang/srt/`:
|
||||
|
||||
```python
|
||||
"""Unit tests for srt/<module>"""
|
||||
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from sglang.srt.<module> import TargetClass
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
|
||||
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
|
||||
# Unit tests are CPU-only. GPU operator tests belong under `kernel/`.
|
||||
|
||||
class TestTargetClass(CustomTestCase):
|
||||
def test_basic_behavior(self):
|
||||
obj = TargetClass(...)
|
||||
self.assertEqual(obj.method(), expected)
|
||||
|
||||
@patch("sglang.srt.<module>.some_dependency")
|
||||
def test_with_mock(self, mock_dep):
|
||||
mock_dep.return_value = MagicMock()
|
||||
# test logic with dependency mocked
|
||||
...
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
```
|
||||
|
||||
Use `unittest.mock.patch` / `MagicMock` only at dependency boundaries. Assert the
|
||||
resulting value, state, protocol output, or error—not merely that the mock was
|
||||
called. If the module transitively imports GPU-only packages (e.g. `sgl_kernel`),
|
||||
they can be stubbed so the test runs on CPU CI. Do not modify `sys.modules` at
|
||||
module level—use `patch.dict` (as a class decorator or with `start`/`stop`) to
|
||||
ensure cleanup and avoid cross-test pollution. See
|
||||
`test/registered/unit/README.md` for details and examples.
|
||||
|
||||
**Quality bar** — test real logic (validation boundaries, state transitions, error paths, branching, etc.). Skip tests that just verify Python itself works (e.g., "does calling an abstract method raise `NotImplementedError`?", "does a dataclass store the field I assigned?"). Consolidate repetitive patterns into parameterized tests. No production code changes in test PRs.
|
||||
|
||||
### E2E test (small model, server needed)
|
||||
|
||||
```python
|
||||
import unittest
|
||||
|
||||
import requests
|
||||
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
from sglang.test.test_utils import (
|
||||
DEFAULT_SMALL_MODEL_NAME_FOR_TEST,
|
||||
DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
|
||||
DEFAULT_URL_FOR_TEST,
|
||||
CustomTestCase,
|
||||
popen_launch_server,
|
||||
terminate_and_kill_process_tree,
|
||||
)
|
||||
|
||||
register_cuda_ci(est_time=60, stage="base-b", runner_config="1-gpu-small")
|
||||
|
||||
|
||||
class TestMyFeature(CustomTestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
|
||||
cls.base_url = DEFAULT_URL_FOR_TEST
|
||||
cls.process = popen_launch_server(
|
||||
cls.model,
|
||||
cls.base_url,
|
||||
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
|
||||
other_args=["--arg1", "value1"], # feature-specific args
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
if hasattr(cls, "process") and cls.process:
|
||||
terminate_and_kill_process_tree(cls.process)
|
||||
|
||||
def test_basic_functionality(self):
|
||||
response = requests.post(
|
||||
self.base_url + "/generate",
|
||||
json={"text": "Hello", "sampling_params": {"max_new_tokens": 32}},
|
||||
)
|
||||
self.assertEqual(response.status_code, 200)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=3)
|
||||
```
|
||||
|
||||
Copy the `tearDownClass` above verbatim. Most existing E2E files still show the bare
|
||||
`kill_process_tree`; that form is being migrated out and must not be reproduced.
|
||||
|
||||
### E2E test (8B model, server needed, performance)
|
||||
|
||||
```python
|
||||
import time
|
||||
import unittest
|
||||
|
||||
import requests
|
||||
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
from sglang.test.test_utils import (
|
||||
DEFAULT_MODEL_NAME_FOR_TEST,
|
||||
DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
|
||||
DEFAULT_URL_FOR_TEST,
|
||||
CustomTestCase,
|
||||
popen_launch_server,
|
||||
terminate_and_kill_process_tree,
|
||||
)
|
||||
|
||||
register_cuda_ci(est_time=300, stage="base-b", runner_config="1-gpu-large")
|
||||
|
||||
|
||||
class TestMyFeaturePerf(CustomTestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.model = DEFAULT_MODEL_NAME_FOR_TEST
|
||||
cls.base_url = DEFAULT_URL_FOR_TEST
|
||||
cls.process = popen_launch_server(
|
||||
cls.model,
|
||||
cls.base_url,
|
||||
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
if hasattr(cls, "process") and cls.process:
|
||||
terminate_and_kill_process_tree(cls.process)
|
||||
|
||||
def test_latency(self):
|
||||
start = time.perf_counter()
|
||||
response = requests.post(
|
||||
self.base_url + "/generate",
|
||||
json={"text": "Hello", "sampling_params": {"max_new_tokens": 128}},
|
||||
)
|
||||
elapsed = time.perf_counter() - start
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertLess(elapsed, 5.0, "Latency exceeded threshold")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=3)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Server Fixture Reuse
|
||||
|
||||
For tests that only need a standard server, inherit from `DefaultServerBase` and override class attributes:
|
||||
|
||||
```python
|
||||
from sglang.test.server_fixtures.default_fixture import DefaultServerBase
|
||||
|
||||
class TestMyFeature(DefaultServerBase):
|
||||
model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
|
||||
other_args = ["--enable-my-feature"]
|
||||
|
||||
def test_something(self):
|
||||
...
|
||||
```
|
||||
|
||||
Available fixtures in `python/sglang/test/server_fixtures/`:
|
||||
|
||||
| Fixture | Use case |
|
||||
|---------|----------|
|
||||
| `DefaultServerBase` | Standard single-server tests |
|
||||
| `EagleServerBase` | EAGLE speculative decoding |
|
||||
| `PDDisaggregationServerBase` | Disaggregated prefill/decode |
|
||||
| `MMMUServerBase` | Multimodal VLM tests |
|
||||
|
||||
---
|
||||
|
||||
## CI Registration
|
||||
|
||||
Every CI-discovered test file must call a registration function at module level:
|
||||
|
||||
```python
|
||||
from sglang.test.ci.ci_register import (
|
||||
register_cuda_ci,
|
||||
register_amd_ci,
|
||||
register_cpu_ci,
|
||||
register_npu_ci,
|
||||
)
|
||||
|
||||
# Per-commit test (small 1-gpu, runs on 5090)
|
||||
register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small")
|
||||
|
||||
# Per-commit test (large 1-gpu, runs on H100)
|
||||
register_cuda_ci(est_time=120, stage="base-b", runner_config="1-gpu-large")
|
||||
|
||||
# Nightly-only test (same shape as per-commit, stage is just "nightly")
|
||||
register_cuda_ci(est_time=200, stage="nightly", runner_config="1-gpu-large")
|
||||
|
||||
# Multi-backend test (only when testing backend-specific code paths)
|
||||
register_cuda_ci(est_time=80, stage="base-a", runner_config="1-gpu-small")
|
||||
register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd")
|
||||
register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True)
|
||||
|
||||
# Temporarily disabled test
|
||||
register_cuda_ci(
|
||||
est_time=80, stage="base-b", runner_config="1-gpu-small", disabled="flaky - see #12345"
|
||||
)
|
||||
```
|
||||
|
||||
Parameters:
|
||||
- `est_time`: estimated runtime in seconds (used for CI partitioning)
|
||||
- `stage` + `runner_config`: the canonical pair for CUDA; the suite name is generated from them (see Naming Conventions)
|
||||
- `suite`: legacy single-string form. Only `stress` and some AMD/CPU/NPU pools still take it; `register_cpu_ci(suite="base-a-test-cpu")` is correct and is not being migrated
|
||||
- `nightly=True`: legacy cadence flag, for non-CUDA nightly suites only. CUDA nightly uses `stage="nightly"` and must leave this unset
|
||||
- `disabled="reason"`: temporarily disable with explanation
|
||||
|
||||
**Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time.
|
||||
|
||||
### JIT Kernel Registration
|
||||
|
||||
`run_suite.py` discovers every `test/registered/**/*.py`, JIT kernel files included.
|
||||
They are ordinary registered tests; only their stage differs:
|
||||
|
||||
```python
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
# Correctness tests in test/registered/kernels/ops/<group>/
|
||||
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
|
||||
register_cuda_ci(est_time=120, stage="base-b-kernel-unit", runner_config="8-gpu-h200")
|
||||
|
||||
# Benchmarks in test/registered/kernels/benchmark/<group>/
|
||||
register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
|
||||
|
||||
# Optional nightly registration — same form, stage is just "nightly"
|
||||
register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
|
||||
register_cuda_ci(est_time=120, stage="nightly", runner_config="8-gpu-h200")
|
||||
```
|
||||
|
||||
Every call generates a suite named `{stage}-test-{runner_config}`, e.g. `base-b-kernel-unit-test-1-gpu-large` and `nightly-test-1-gpu-large`. Keep `est_time`, `stage`, `runner_config`, and `suite` as **literal values** — `run_suite.py` collects them by AST parsing.
|
||||
|
||||
---
|
||||
|
||||
## Test Placement
|
||||
|
||||
```
|
||||
test/
|
||||
├── registered/ # CI tests (auto-discovered by run_suite.py)
|
||||
│ ├── unit/<subsystem>/ # CPU-only; no server or model weights
|
||||
│ ├── kernel/<group>/ # accelerator operator correctness/benchmarks
|
||||
│ ├── e2e/<subsystem>/ # engine/server integration
|
||||
│ ├── accuracy/<family>/ # scheduled eval floors
|
||||
│ ├── perf/<family>/ # scheduled latency/throughput contracts
|
||||
│ └── stress/<subsystem>/ # stress/weekly coverage
|
||||
├── manual/ # Non-CI: debugging, one-off, manual verification
|
||||
└── run_suite.py # CI runner (globs test/registered/**/*.py; nothing outside it)
|
||||
|
||||
python/sglang/kernels/jit/ # implementation + test-only helpers, never registered tests
|
||||
```
|
||||
|
||||
A `register_*_ci(...)` under `python/sglang/` is rejected by the
|
||||
`check-no-registered-tests-in-package` pre-commit hook.
|
||||
|
||||
**Decision rule** (see also `test/registered/README.md`):
|
||||
- CPU component logic, no server → `registered/unit/<subsystem>/`
|
||||
- JIT kernel correctness → `registered/kernels/ops/<group>/`
|
||||
- JIT kernel benchmarks → `registered/kernels/benchmark/<group>/`
|
||||
- Other accelerator operator correctness → `registered/kernels/ops/<group>/`
|
||||
- Server needed → `registered/e2e/<subsystem>/`
|
||||
- Eval floor / performance contract → `registered/{accuracy,perf}/<family>/`
|
||||
- Local debugging → `manual/`
|
||||
|
||||
---
|
||||
|
||||
## Eval Accuracy Mixins
|
||||
|
||||
**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (the eval call, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
|
||||
|
||||
Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults. `MMLUMixin` already scores through sgl-eval; `GSM8KMixin` still defaults to the deprecated `run_eval` backend for existing tests, so new tests set `gsm8k_backend = "sgl_eval"`. The exception is a base or completion-only model: sgl-eval only sends chat requests, so those stay on the 5-shot completion backend.
|
||||
|
||||
```python
|
||||
class TestMyFeature(CustomTestCase, MMLUMixin):
|
||||
mmlu_score_threshold = 0.65
|
||||
mmlu_num_examples = 64
|
||||
mmlu_num_threads = 32
|
||||
# test_mmlu is inherited — no code needed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Utilities
|
||||
|
||||
```python
|
||||
from sglang.test.test_utils import (
|
||||
CustomTestCase, # base class with retry logic
|
||||
popen_launch_server, # launch server subprocess
|
||||
terminate_and_kill_process_tree, # SIGTERM, then SIGKILL, then wait for
|
||||
# the GPU memory to come back
|
||||
DEFAULT_URL_FOR_TEST, # auto-configured base URL
|
||||
DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH, # 600s default
|
||||
run_bench_serving, # benchmark helper (launch + bench)
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
Before submitting a test:
|
||||
|
||||
- [ ] Inherits from `CustomTestCase` (not `unittest.TestCase`)
|
||||
- [ ] Has `register_*_ci(...)` call at module level
|
||||
- [ ] Placed in `test/registered/<kind>/<subsystem>/`
|
||||
- [ ] JIT kernel work: correctness tests live in `test/registered/kernels/ops/<group>/`, benchmarks live in `test/registered/kernels/benchmark/<group>/`, and only test-only helpers stay under `python/sglang/kernels/jit/`
|
||||
- [ ] Backend-independent tests: `register_cuda_ci` only + smallest model
|
||||
- [ ] Logic that doesn't need a server / engine launch → unit test in `registered/unit/` (see Unit Tests section)
|
||||
- [ ] `tearDownClass` is defensive — uses `hasattr`/null checks before accessing resources that may not have been allocated
|
||||
- [ ] Every case answers "what future diff turns this red?" — see [`unit-test-admission`](../../rules/unit-test-admission.md)
|
||||
- [ ] `est_time` is reasonable (measure locally)
|
||||
|
||||
Run these against the new file and paste the output rather than self-attesting:
|
||||
|
||||
```bash
|
||||
f=<your new test file>
|
||||
grep -n "kill_process_tree" $f # every hit must be terminate_and_kill_process_tree
|
||||
grep -n "register_.*_ci(" $f # CUDA: stage= + runner_config=, never suite=
|
||||
grep -n "CustomTestCase\|unittest.main" $f # both must appear
|
||||
```
|
||||
@@ -1,21 +0,0 @@
|
||||
# Serving a model? Read its cookbook page first
|
||||
|
||||
Before running, deploying, benchmarking, or reproducing a specific model, read
|
||||
its page under `docs/cookbook/`:
|
||||
|
||||
- `docs/cookbook/autoregressive/` — LLM and VLM serving
|
||||
- `docs/cookbook/diffusion/` — image, video, 3D, robotics
|
||||
- `docs/cookbook/omni/`, `docs/cookbook/vla/`, `docs/cookbook/specbundle/`,
|
||||
`docs/cookbook/base/`
|
||||
|
||||
That page carries the deployment we recommend to users — GPU count, parallelism
|
||||
degrees, quantization, and the flags that matter for that model — so it is the
|
||||
answer to "how should this be served", and the baseline any tuning starts from.
|
||||
|
||||
Configs under `test/` are a different thing: pinned, reproducible CI setups for
|
||||
correctness and regression checks, not deployment advice. Reach for them when
|
||||
you need an exactly reproducible run, and say which cookbook flags you diverged
|
||||
from — a measurement taken on a configuration nobody deploys describes nothing.
|
||||
|
||||
Keep it true in both directions: when a change alters a model's recommended
|
||||
deployment, update that model's cookbook page in the same PR.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
paths:
|
||||
- "**/bench_*.py"
|
||||
- "**/benchmark/**/*.py"
|
||||
- "benchmark/kernels/**"
|
||||
---
|
||||
|
||||
# Kernel benchmarks
|
||||
|
||||
How a benchmark is written is up to you. The requirement is only that the numbers are valid.
|
||||
|
||||
## Tools
|
||||
|
||||
In SGLang we recommend `marker.do_bench` (`python/sglang/kernels/jit/benchmark/marker.py`). It works for any callable, not only JIT kernels, and it measures with a cold L2 by default. It is not mandatory. CUPTI (e.g. CUDA 13 `cupti-python`), the torch profiler / `bench_kineto`, Triton's `do_bench` / proton, `flashinfer.testing`, Nsight Systems / Nsight Compute, or a hand-written loop are all fine. Each pitfall below must be handled, whichever tool you use.
|
||||
|
||||
## L2 cache reuse
|
||||
|
||||
Timing the same kernel on the same inputs in a loop measures an L2-hot kernel, not the one serving runs. `triton.testing.do_bench_cudagraph` with fixed arguments and bare `for` loops between two events both do this. Measure cold L2 by flushing between calls or by rotating enough input copies to exceed L2 several times over. Hot L2 is acceptable only when the working set far exceeds L2 or when hot L2 is the real serving pattern, and the reason must be stated.
|
||||
|
||||
Host-memory (zero-copy) loads do not go through device L2. Peer-GPU (NVLink) reads still see reuse (on remote GPU).
|
||||
@@ -6,5 +6,4 @@ Before modifying the following components, read the listed skill first.
|
||||
- **`Scheduler` / `TokenizerManager` / `ModelRunner` `__init__`** (`python/sglang/srt/managers/scheduler.py`, `python/sglang/srt/managers/tokenizer_manager.py`, `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Any edit to a frozen core file** (currently `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Environment variables** (adding, renaming, or reviewing any `SGLANG_*` env var, migrating a legacy `SGL_*` alias, or touching `python/sglang/srt/environ.py`) → [`env-var-conventions`](../skills/env-var-conventions/SKILL.md)
|
||||
- **Chat rendering in `rust/sglang-processor`** (adding or changing a model under `rust/sglang-processor/src/render/`, or its parity fixtures) → [`processor-model-parity`](../skills/processor-model-parity/SKILL.md)
|
||||
- **Scripted runtime** (anything related to the scripted runtime) → [`scripted-runtime-notes`](../skills/scripted-runtime-notes/SKILL.md)
|
||||
|
||||
@@ -36,4 +36,4 @@ a real `AttributeError` if the field is ever renamed:
|
||||
revision=getattr(server_args, "revision", None), # BAD
|
||||
revision=server_args.revision, # GOOD
|
||||
```
|
||||
(see `python/sglang/srt/parser/template_detection.py`)
|
||||
(see `python/sglang/srt/managers/template_detection.py`)
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Accuracy evals: use sgl-eval
|
||||
|
||||
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
|
||||
model's accuracy.
|
||||
|
||||
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
|
||||
server's env, at the version pinned in the `test` extra of
|
||||
`python/pyproject.toml`.
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
|
||||
```
|
||||
|
||||
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
|
||||
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
|
||||
|
||||
- **Sampling:** use the model's recommended settings (temperature, top_p,
|
||||
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
|
||||
them with its own defaults, so pass them explicitly, or use
|
||||
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
|
||||
sets the `thinking` chat-template key; if the model's template reads a
|
||||
different one (commonly `enable_thinking`), pass that key with
|
||||
`--chat-template-kwarg` instead. When comparing against a CI test, use that
|
||||
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
|
||||
|
||||
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
|
||||
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
|
||||
base models, since sgl-eval only sends chat requests.
|
||||
|
||||
Only compare scores from the same harness and settings, and say which ones you
|
||||
used when you report a number.
|
||||
@@ -27,8 +27,8 @@ A new unit test case must fall into one of these categories:
|
||||
3. **Critical-path bookkeeping.** Defends conventions that are easy to break by
|
||||
forgetting to sync -- registry completeness, field lifecycle, serialization
|
||||
compatibility. Enumerating assertions are fine here; the guarded failure
|
||||
mode is "someone extended X without updating Y". Example:
|
||||
`test/registered/unit/test_global_config_read_ratchet.py`.
|
||||
mode is "someone extended X without updating Y". Example: the ratchet tests
|
||||
(`test/registered/unit/test_module_state_ratchet.py`).
|
||||
|
||||
Not admissible:
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
../.agents/skills
|
||||
@@ -0,0 +1,742 @@
|
||||
---
|
||||
name: add-jit-kernel
|
||||
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
|
||||
---
|
||||
|
||||
# Tutorial: Adding a New JIT Kernel to SGLang
|
||||
|
||||
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||
|
||||
## Goal
|
||||
|
||||
Add a new operation that scales each element of a tensor by a scalar factor:
|
||||
|
||||
- Input: tensor `x` (CUDA) and scalar `factor` (float, passed at runtime)
|
||||
- Output: `x * factor` (element-wise), allocated internally
|
||||
- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
|
||||
|
||||
## When to use JIT vs AOT (`sgl-kernel`)
|
||||
|
||||
- **JIT (`jit_kernel`)**: prefer this first for kernels that do **not** depend on CUTLASS or another large C++ project. It is the default choice for lightweight kernels that benefit from rapid iteration and first-use compilation.
|
||||
- **AOT (`sgl-kernel`)**: prefer this when the kernel **does** depend on CUTLASS or another large C++ project, or when it should live in `python/sglang/kernels/aot/` and participate in the wheel build / torch op registration flow.
|
||||
- **Exception**: kernels that depend on `flashinfer`, or on CUTLASS that is already provided through `flashinfer`, can still be implemented as `jit_kernel`.
|
||||
|
||||
---
|
||||
|
||||
## Conventions
|
||||
|
||||
These hold for every step below.
|
||||
|
||||
- **`namespace sglang` is where JIT code lives.** Open it after the include block and close it at the end of the file, with the device kernels, traits and host wrapper inside. The shared `host::` / `device::` helpers are nested in it too, so they resolve unqualified. `load_jit` emits the `TVM_FFI_DLL_EXPORT_TYPED_FUNC` wrapper inside `namespace sglang` as well, so the `kernel_name` you pass from Python needs no `sglang::` prefix.
|
||||
- **Tuning policy belongs in Python; C++ gets the answer, not the decision.** For a hyperparameter — split factor, threads per item, block size, vector width, an algorithm variant — prefer (not required) making it a **template parameter** and letting the `@cache_once` module factory choose the value, over a runtime `if`/`switch` in the launcher that picks among pre-instantiated kernels. The kernel then compiles exactly one specialisation and asserts its own preconditions with `static_assert`, while the heuristic that produced the value sits in Python where it is readable, adjustable, and inspectable without recompiling CUDA. `store_cache`'s `num_threads` is the worked example: a `get_kernel(num_split)` ladder guarded by byte-alignment `if constexpr`s became one template argument plus a heuristic in `_jit_kvcache_module`. The exception is a knob that genuinely varies per call with the runtime shape — that has to stay a kernel argument.
|
||||
- **Check where the check is cheapest: `static_assert` > C++ host check > cached Python > per-call Python.** Anything fixed at compile time is a `static_assert`. Anything about the tensors is a `TensorMatcher` / `CHECK_HOST` in the C++ launcher, free next to a kernel launch. A check Python cannot delegate goes inside the `@cache_once` module factory, where it runs once per specialisation. What remains in the per-call entry point costs interpreter time on *every* forward, so it should be nothing but picking the module and allocating `out`.
|
||||
- **Fixed-width integer types.** Prefer `int32_t` / `int64_t` / `uint32_t` / `size_t` over `int`, `long`, or `long long`, so an index has the same width on both sides of the FFI boundary. Bare `int` is fine only where the width plainly cannot matter — an unrolled loop counter over a `constexpr` bound, a template `int` parameter. Shapes arrive as `int64_t` (`SymbolicSize::unwrap()`); narrowing to `uint32_t` for in-kernel indexing is a deliberate act, so write the `static_cast` explicitly and only where the range is known.
|
||||
- **Doxygen comments in C++.** Document exported entities with `///` or `/** ... */` blocks using `\brief`, `\param`, `\tparam`, `\return`, the way `include/sgl_kernel/` does. `python -m sglang.kernels.jit` writes `CommentFormat: Doxygen` into `.clangd` when clangd is 21 or newer, so these render on hover in the editor. Plain `//` remains fine for implementation notes inside a function body.
|
||||
- **ASCII only in C++ and CUDA sources.** Write `--`, `->`, `<=` instead of `—`, `→`, `≤`, including in comments. `grep -nP '[^\x00-\x7F]' <file>` before committing.
|
||||
- **`namespace details` is private.** Anything inside one is an implementation detail of its own header, free to change without notice. Do not name `details::` from another module, and never `using namespace details`. If you find yourself wanting something in there, that is the signal to promote it to a documented name instead.
|
||||
- **`const T* __restrict__` for read-only pointers.** This is what `csrc/` does throughout, and it lets the compiler emit non-coherent (`LDG`) loads.
|
||||
- **Watch the register budget.** For memory-bound kernels, keep to roughly 64 registers per thread so occupancy does not become the limit; prefer recomputing a value over letting it spill. Run `SGLANG_JIT_LOG_RESOURCE_USAGE=1 SGLANG_JIT_FORCE_RECOMPILE=1` to log per-kernel registers, spills and shared memory at INFO. Both halves are load-bearing: a cache hit has no compiler output to report, and passing `-Xptxas -v` by hand does nothing on its own because the build captures compiler output and replays it only on failure.
|
||||
- **Alignment is a contract you enforce, not a property you hope for.** A vectorized copy derives its access width from the row *size* (`load_bytes` picks 16B for a 1024B row), but the address it lands on is `base + index * row_stride` — and a stride is not constrained by the size. `TensorMatcher` admits any stride unless you say otherwise, so a padded cache row silently produces `misaligned address`. Every kernel that vectorizes is highly recommended to end its validation with `.ensure_alignment(w)`, where `w` is the width the kernel actually uses.
|
||||
|
||||
---
|
||||
|
||||
## Common Abstractions in `python/sglang/kernels/jit/include/sgl_kernel/`
|
||||
|
||||
**Always prefer these abstractions over raw CUDA primitives.** They provide safety, readability, and consistency with the rest of the codebase. The only reason to drop to raw primitives is performance the abstraction cannot reach — a trade you make deliberately, and justify in a comment.
|
||||
|
||||
### `utils.h` — Host-side utilities
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/utils.h>
|
||||
```
|
||||
|
||||
- **`CHECK_HOST(cond) << "msg " << value`** — **Preferred** runtime check: stream-style, throws `PanicError` with file/line info on failure. Zero overhead on the true path — the message expressions are only evaluated when the check fails.
|
||||
- **`host::RuntimeCheck(cond, args...)`** — Function-style alternative to `CHECK_HOST`. Note its message args are always evaluated (even when the check passes), so prefer `CHECK_HOST` — especially on hot paths.
|
||||
- **`host::Panic(args...)`** — Unconditionally throw a `PanicError` with a descriptive message.
|
||||
- **`host::div_ceil(a, b)`** — Integer ceiling division `(a + b - 1) / b`.
|
||||
- **`host::irange(n)`** / **`host::irange(start, end)`** — Range views for cleaner loops.
|
||||
- **`host::pointer::offset(ptr, offsets...)`** — Byte-safe pointer arithmetic on `void*`. Use this instead of raw casts.
|
||||
|
||||
### `utils.cuh` — Device-side utilities + `LaunchKernel`
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/utils.cuh>
|
||||
```
|
||||
|
||||
- **Type aliases**: `fp16_t`, `bf16_t`, `fp32_t`, `fp8_e4m3_t`, `fp8_e5m2_t` and their packed variants `fp16x2_t`, `bf16x2_t`, `fp32x2_t`, etc.
|
||||
- **`SGL_DEVICE`** — Expands to `__forceinline__ __device__`. Use on all device functions.
|
||||
- **`SGL_DEVICE_HOST`** — `__forceinline__ __device__ __host__`. Use it when a `constexpr` helper must be callable from both the kernel and its launcher, so the two cannot drift.
|
||||
- **`device::kWarpThreads`** — Constant `32` (the codebase's *logical* warp width). `kWarpSize` is an alias. Note this is a group size, not the hardware wave: on gfx950 `warpSize` is 64, and `warp::kFullWidth` reflects that.
|
||||
- **`device::get_lane_id<kNumThreads = kWarpThreads>()`** — this thread's index within its logical group. **Prefer it over `threadIdx.x % kNumThreads` whenever the value feeds an address.** On CUDA the lane register is one read that folds into the address computation, whereas the modulo is a derived value the compiler re-materialises at *every* address scale — 8 instructions and 2 registers on a two-tile warp copy, measured on sm_100a. It picks the cheaper form per platform, so callers never need to care. Only equals the true in-warp lane when `blockDim.x` is a multiple of `kNumThreads`.
|
||||
- **`device::load_as<T>(ptr, offset)`** / **`device::store_as<T>(ptr, val, offset)`** — Type-safe loads/stores from `void*`.
|
||||
- **`device::pointer::offset(ptr, offsets...)`** — Pointer arithmetic on device.
|
||||
- **`host::LaunchKernel(grid, block, device_or_stream [, smem])`** — RAII kernel launcher that:
|
||||
- Resolves the CUDA stream from a `DLDevice` via TVM-FFI automatically.
|
||||
- Checks the CUDA error with file/line info after launch via `operator()(kernel, args...)`.
|
||||
- Supports `.enable_pdl(bool)` for PDL (Programmatic Dependent Launch, SM90+).
|
||||
- **`device::PDLWaitPrimary<kUsePDL>()`** / **`device::PDLTriggerSecondary<kUsePDL>()`** — The two halves of PDL, on sm_90+ (no-ops on older archs and ROCm). Their guarantees are **not** symmetric:
|
||||
- `PDLTriggerSecondary` (`griddepcontrol.launch_dependents`) only lets the next kernel in the stream *start* early. It carries no memory ordering and publishes nothing — matching that, the header's asm has no `"memory"` clobber.
|
||||
- `PDLWaitPrimary` (`griddepcontrol.wait`) is the ordering point: it waits until the preceding kernel has fully finished and its writes are visible.
|
||||
|
||||
So every read of data the preceding kernel produced must come after `PDLWaitPrimary()`. What overlaps with the primary's tail is whatever you put *before* the wait — loading parameters, computing indices, touching buffers the primary never wrote — so a kernel that waits on its first line gains nothing. Neither call is a barrier: threads may reach or skip them independently. See "Programmatic Dependent Launch and Synchronization" in the CUDA C++ Programming Guide.
|
||||
- **`CHECK_CUDA(expr) << "context"`** — Stream-style CUDA error check; evaluates `expr` once and throws `PanicError` with `cudaGetErrorString` + file/line info if it is not `cudaSuccess`. Extra streamed context is optional.
|
||||
- **`host::RuntimeDeviceCheck(cudaError_t)`** — Function-style alternative to `CHECK_CUDA`. It takes no context message, so prefer `CHECK_CUDA`, which builds its error object only on the failure path.
|
||||
|
||||
### `tensor.h` — Tensor validation (`TensorMatcher`, Symbolic types)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/tensor.h>
|
||||
```
|
||||
|
||||
This is the **primary validation API** for all kernel launchers. Use it to validate every `tvm::ffi::TensorView` argument.
|
||||
|
||||
- **`host::SymbolicSize{"name"}`** — A named symbolic dimension. Call `.set_value(n)` to pin it, `.unwrap()` to extract after verification.
|
||||
- **`host::SymbolicDType`** — Symbolic dtype. Use `.set_options<Ts...>()` to restrict allowed types.
|
||||
- **`host::SymbolicDevice`** — Symbolic device. Use `.set_options<kDLCUDA>()` to restrict to CUDA.
|
||||
- **`host::TensorMatcher({dims...})`** — Fluent builder for tensor validation:
|
||||
- `.with_dtype<T>()` — require a specific C++ type (e.g. `fp16_t`)
|
||||
- `.with_dtype<T1, T2, ...>()` — allow a set of types
|
||||
- `.with_device<kDLCUDA>(device_sym)` — require CUDA and bind the checked device to a `SymbolicDevice`
|
||||
- `.with_strides({strides...})` — validate strides (omit to require contiguous)
|
||||
- `.ensure_alignment(bytes)` — require the data pointer **and every stride except the innermost** to be a multiple of `bytes` (a power of two). Size-1 dimensions are skipped, since their strides are arbitrary. This is how a vectorized kernel states its precondition; see the alignment convention above.
|
||||
- `.verify(tensor_view)` — execute the check; throws `PanicError` with full context on failure; **chainable** (`verify(a).verify(b)` to check multiple tensors with the same shape)
|
||||
- **`host::is_type<T>(dtype)`** — whether a `DLDataType` denotes the C++ type `T` (e.g. `fp16_t`).
|
||||
|
||||
**Typical pattern:**
|
||||
```cpp
|
||||
auto N = SymbolicSize{"num_elements"};
|
||||
auto device = SymbolicDevice{};
|
||||
device.set_options<kDLCUDA>();
|
||||
TensorMatcher({N}) //
|
||||
.with_dtype<fp16_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.ensure_alignment(16) // e.g. for 128-bit vectorized load/store
|
||||
.verify(dst)
|
||||
.verify(src); // same shape, dtype, device as dst
|
||||
const int64_t n = N.unwrap();
|
||||
const DLDevice dev = device.unwrap();
|
||||
const int64_t last_dim = 128;
|
||||
TensorMatcher({N, last_dim}) // a fixed dimension can be a plain integer
|
||||
.with_dtype<fp16_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(tensor_2d);
|
||||
```
|
||||
|
||||
### `ffi.h` — Tensor allocation and blob wrapping (`host::ffi::`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/ffi.h>
|
||||
```
|
||||
|
||||
The counterpart to `tensor.h`: that one validates what came in, this one produces new `tvm::ffi::Tensor` values. Allocation goes through the environment allocator (`TVMFFIEnvTensorAlloc`), so buffers come from PyTorch's caching allocator rather than a raw `cudaMalloc`.
|
||||
|
||||
- **`host::alloc_workspace_tensor(nbytes, device)`** (declared in `utils.cuh`) — **the way to get scratch memory**: a 1-D `uint8` tensor of `nbytes`, or an empty tensor when `nbytes == 0`. Hold the returned `Tensor` in a local across every launch that touches it — it frees on destruction.
|
||||
- **`host::ffi::empty(shape, dtype, device)`** — Uninitialized tensor; `shape` accepts a braced list, so `ffi::empty({rows, sizeof(Plan)}, dtype, device)` works for a typed scratch array.
|
||||
- **`host::ffi::empty_like(tensor_view)`** — Same shape, dtype, and device as an existing tensor.
|
||||
- **`host::ffi::from_blob(data, shape, dtype, device[, deleter, stride, byte_offset])`** / **`from_blob_like(data, tensor_view, ...)`** — View memory you already own as a `Tensor`, no copy. The default deleter does nothing, so ownership stays with the caller; pass one only when the `Tensor` should own the block. Strides default to contiguous.
|
||||
|
||||
### `type.cuh` — `DTypeTrait<T>`, `packed_t<T>`, and reduction traits
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/type.cuh>
|
||||
```
|
||||
|
||||
- **`DTypeTrait<T>`** — Static trait struct, specialized for integral types, `fp32_t`, `fp16_t`, `bf16_t`, `fp8_e4m3_t`, and their packed x2/x4 variants. Provides:
|
||||
- `DTypeTrait<T>::from(value)` — convert from another type via the right CUDA intrinsic (e.g. `fp32_t` → `fp16_t`)
|
||||
- `DTypeTrait<T>::abs/max/min` — type-dispatched math (fp32, fp16/bf16 scalar and x2, integrals)
|
||||
- `DTypeTrait<T>::sqrt/rsqrt/exp/sin/cos(x)` — `fp32_t` only
|
||||
- Metadata: `packed_t` / `unpacked_t` / `kVecSize` (packed layout), `kFloatMax` (dtype max as float, e.g. 448.0f for fp8-e4m3), `kZeroBits`
|
||||
- **`packed_t<T>`** — Two-element packed alias: `packed_t<fp16_t>` = `fp16x2_t`, `packed_t<bf16_t>` = `bf16x2_t`, `packed_t<fp32_t>` = `fp32x2_t`. Use for vectorized loads/stores.
|
||||
- **`device::cast<To, From>(value)`** — Type-safe cast using `DTypeTrait`, e.g. `cast<fp32x2_t, fp16x2_t>(v)`.
|
||||
- **`device::unpack(value)`** — View a packed value as an `unpacked_t[kVecSize]` array reference (e.g. `fp32x2_t` → `fp32_t[2]`); element writes propagate back to the packed value.
|
||||
- **`device::ReductionOp` (`SUM`/`MAX`/`MIN`) and `device::ReductionTrait<Op, T>::reduce(x, y)`** — One binary reduction step, dispatched through `DTypeTrait` (packed types reduce elementwise). This is the engine behind `warp::reduce`; use it directly when writing custom reductions.
|
||||
|
||||
### `vec.cuh` — Vectorized memory access (`AlignedVector`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/vec.cuh>
|
||||
```
|
||||
|
||||
- **`device::AlignedVector<T, N>`** — Aligned storage for N elements of type T. N must be a power of two, `sizeof(T)*N <= 32`. Enables vectorized loads/stores for bandwidth efficiency. In terms of API/codegen constraints, the upper bound is 256-bit; in practice, 128-bit is the portable default, while 256-bit vectorization is typically only viable on `SM100+` and should be gated by an architecture check when needed.
|
||||
- `.load(ptr, offset)` — vectorized load from `ptr[offset]`
|
||||
- `.store(ptr, offset)` — vectorized store to `ptr[offset]`
|
||||
- `.fill(value)` — fill all N elements with `value`
|
||||
- `operator[](i)` — element access
|
||||
- **`device::LoadStoreBytes`** — named widths to pass around instead of bare integers: `MAX_GMEM` (arch-dependent, 32 on Blackwell), `MAX_SMEM` (16), `MAX_PORTABLE` (16, safe on CUDA and HIP), `MIN_COALALESCED` (4), and `RAW_1B` .. `RAW_32B`.
|
||||
|
||||
### `tile.cuh` — `tile::Memory` (strided memory access pattern)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/tile.cuh>
|
||||
```
|
||||
|
||||
- `tile::Memory<T>` is fundamentally a **1D cooperative accessor** over a contiguous region.
|
||||
- **`device::tile::Memory<T>::cta(blockDim.x)`** — Creates a tile accessor where each thread handles `tid = threadIdx.x` with stride `tsize` (for `cta(blockDim.x)`, this is `blockDim.x`). Common for loops over a 1D array.
|
||||
- **`device::tile::Memory<T>::warp()`** — the 32-lane flavour, and what `warp::load_bytes` is built on. It takes its lane index from `get_lane_id()` rather than `threadIdx.x % 32`, for the addressing reason described under `utils.cuh`. Use `warp(int n)` for a narrower sub-group; that overload keeps the modulo, since a hardware lane index is the wrong grouping below 32.
|
||||
- **`device::tile::Memory<T>::thread()`** — single-thread accessor, no cooperation.
|
||||
- **`.load(ptr, offset)`** — loads `ptr[tid + offset * tsize]`
|
||||
- **`.store(ptr, val, offset)`** — stores to `ptr[tid + offset * tsize]`
|
||||
- **`.in_bound(n, offset)`** — boundary check
|
||||
|
||||
For a **2D tile**, either flatten `(row, col)` into a linear tile index first, or compute the address manually with `ptr[row * stride + col]` using your thread/block coordinates.
|
||||
|
||||
### `bits.h` — Compile-time bit helpers (`host::`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/bits.h>
|
||||
```
|
||||
|
||||
`constexpr` wrappers over `<bit>`, usable in `static_assert` and in template arguments: `host::is_pow2(x)`, `log2_floor(x)`, `log2_ceil(x)` (both `-1` for `x == 0`), `round_up_pow2(x)`, `round_down_pow2(x)`. They live in `host::` but are constant-expression-only, so device-side `static_assert` can call them.
|
||||
|
||||
### `math.cuh` — Device math (`device::math::`)
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/math.cuh>
|
||||
```
|
||||
|
||||
- `device::math::max/min<T>(a, b)` — type-dispatched binary math via `DTypeTrait`
|
||||
- `device::math::abs/sqrt/rsqrt/exp/sin/cos<T>(x)` — type-dispatched unary math via `DTypeTrait`
|
||||
|
||||
### `warp.cuh` — Warp-level primitives
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/warp.cuh>
|
||||
```
|
||||
|
||||
**Vectorized row copy — reach for this before hand-rolling.** A warp cooperatively moving one contiguous row is the single most common shape in this tree, and it used to be re-implemented per kernel.
|
||||
|
||||
- **`warp::load_bytes<kBytes, kPattern>(src)`** / **`warp::store_bytes<kBytes, kPattern>(dst, val)`** — the whole warp moves `kBytes` from/to one contiguous row. `kBytes` is arbitrary, down to 1; the helper picks the widest vector that divides it and handles the ragged tail. The returned value is opaque and encodes the width, so a load and its matching store **must use the same `<kBytes, kPattern>`**.
|
||||
- **`warp::LoadStorePattern`** — the `kPattern` values, and the sign carries the meaning. A **negative** one (`WARP_UNIFORM_16B` and friends: `_GMEM`, `_SMEM`, `_4B`/`_8B`/`_32B`) says *the whole warp splits this row*, so the width is derived from the per-lane share (`gcd(kBytes / 32, |kPattern|)`), falling back to `gcd(kBytes, 4)` when the row is too narrow to give every lane 4 bytes. A **positive** one (a bare `LoadStoreBytes` value) makes no warp-splitting assumption and just caps the per-thread vector at `gcd(kBytes, kPattern)`. `WARP_UNIFORM_16B` is the usual choice for a row copy.
|
||||
- **`LoadStorePattern::get_vec_bytes<kBytes, kPattern>()`** — the width actually chosen. `SGL_DEVICE_HOST`, so **the launcher must call this to feed `.ensure_alignment(...)`**. Pass it the same value the kernel passes `load_bytes` — for a split row that is the *per-warp share*, not the whole row; the full row can resolve to a different width and would under-constrain the strides.
|
||||
|
||||
**Reductions.**
|
||||
|
||||
- `warp::reduce<Op, kStart, kFinish>(value, active_mask)` — `Op` is a `device::ReductionOp` (`SUM`/`MAX`/`MIN`). `kStart` and `kFinish` bound a range of lane-index bits: `<N, 1>` reduces within contiguous groups of `N` lanes, `<kFullWidth, N>` reduces *across* groups at the same offset. The operation is **symmetric** — `<A, B>` and `<B, A>` reduce the same lane set.
|
||||
- `warp::reduce_sum/reduce_max/reduce_min<kStart, kFinish>(value)` — wrappers. Work for any type with a `ReductionTrait`: floats, integers, packed x2. This is more widely used in practice than generic `reduce`.
|
||||
- `warp::inclusive_reduce<Op, kWidth, kStart, kFinish>(val, lane_id, mask)` and `inclusive_sum` / `inclusive_max` / `inclusive_min` — segmented inclusive scan: every lane keeps its own running total, unlike `reduce`. Forward when `kStart < kFinish`, backward when `kStart > kFinish`. `lane_id` defaults to `get_lane_id<kWidth>()`, which is correct by construction — only pass it if you already have the **segment-relative** index (`threadIdx.x % kWidth`); a warp-relative one silently corrupts every segment past the first when `kWidth < 32`.
|
||||
- `warp::broadcast<kWidth>(value, src_lane)` — `src_lane` is **segment-relative**: each `kWidth` segment reads its own lane, not one warp-wide source.
|
||||
- `warp::get_lane_id`, `warp::elect_one_lane()` (one elected lane, for gating a single-thread TMA issue; CUDA-only), `warp::kFullMask` and `warp::kFullWidth` (32 on CUDA, 64 on HIP — note this differs from `kWarpThreads`).
|
||||
|
||||
### `cta.cuh` — CTA-level primitives
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/cta.cuh>
|
||||
```
|
||||
|
||||
- `device::cta::reduce_max<T>(value, smem, min_value)` — CTA-wide max using shared memory + warp reduction. Caller is responsible for a `__syncthreads()` after if the result in `smem[0]` is needed.
|
||||
|
||||
### `atomic.cuh` — Atomic operations
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/atomic.cuh>
|
||||
```
|
||||
|
||||
- `device::atomic::max(float* addr, float value)` — float atomic max (handles negative values correctly via bit tricks).
|
||||
- **`device::atomic::Event`** — a cross-CTA arrive/wait counter in one 32-bit word. Producers call `arrive()`; consumers call `wait(num_producers)` (exactly one consumer) or `wait_multi<kConsumerBits>(num_producers, num_consumers, n)` (several). The last released consumer resets the word, so one `Event` is reusable across launches with no host re-zero. Preconditions, all of them load-bearing:
|
||||
- the word must be **zeroed by the host** before first use, and must live in **global** memory — a shared or local `Event` is an illegal address;
|
||||
- `arrive()` is a release and the waits are acquires, so a producer's writes before `arrive()` are visible after the wait;
|
||||
- **generations must be explicitly ordered** — the word has no phase bit, so overlapping generation N+1 with a consumer still in generation N is undefined behaviour (a lagging consumer never exits);
|
||||
- `wait()` and `wait_multi()` lay the word out incompatibly; mixing them on one `Event` is undefined behaviour.
|
||||
|
||||
### `runtime.cuh` — Occupancy and device info
|
||||
|
||||
```cpp
|
||||
#include <sgl_kernel/runtime.cuh>
|
||||
```
|
||||
|
||||
- `host::runtime::get_blocks_per_sm(kernel, block_dim)` — max active blocks per SM (occupancy)
|
||||
- `host::runtime::get_sm_count(device_id [, use_cache])` — number of SMs on the device. Memoized per device ordinal, so it is cheap enough to call on a launch path; pass `use_cache=false` to force a driver query.
|
||||
- `host::runtime::get_cc_major` / `get_cc_minor` / `get_sm_version` — compute capability, same caching.
|
||||
|
||||
**Do not query the architecture at runtime.** The JIT compiles for the exact local GPU, so the arch is a *compile-time* fact: `SGL_CUDA_ARCH` is injected by `load_jit`, and `SGL_ARCH_HOPPER_OR_GREATER` / `SGL_ARCH_BLACKWELL_OR_GREATER` / `device::kMaxVecBytes` are derived from it. Reach for `get_cc_*` only for something the arch genuinely does not determine. SM *count* is the opposite case — it varies within an arch (B200 vs B300 vs a MIG slice), so it has to stay a runtime query.
|
||||
|
||||
**Persistent kernel pattern** (cap blocks to SM count × occupancy):
|
||||
```cpp
|
||||
static const uint32_t max_occ = runtime::get_blocks_per_sm(kernel, kBlockSize);
|
||||
static const uint32_t num_sm = runtime::get_sm_count(device.unwrap().device_id);
|
||||
const auto num_blocks = std::min(num_sm * max_occ, div_ceil(n, kBlockSize));
|
||||
LaunchKernel(num_blocks, kBlockSize, device.unwrap())(kernel, params);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 0 (optional): Generate a `.clangd` config for better IDE support
|
||||
|
||||
```bash
|
||||
python -m sglang.kernels.jit -h # for verbose help info about clangd configuration
|
||||
python -m sglang.kernels.jit
|
||||
python -m sglang.kernels.jit --dep cutlass flashinfer # with cutlass/flashinfer dependency
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Implement the CUDA kernel in `kernels/jit/csrc/`
|
||||
|
||||
Create `python/sglang/kernels/jit/csrc/elementwise/scale.cuh`.
|
||||
|
||||
The implementation fully uses the project abstractions described above:
|
||||
|
||||
```cpp
|
||||
// NOTE: Comments for headers are not common in practice.
|
||||
// It is only shown here for tutorial purposes to highlight the key abstractions.
|
||||
#include <sgl_kernel/tensor.h> // For TensorMatcher, SymbolicSize, SymbolicDevice
|
||||
#include <sgl_kernel/type.cuh> // For DTypeTrait, fp16_t, bf16_t, fp32_t
|
||||
#include <sgl_kernel/utils.h> // For CHECK_HOST, div_ceil
|
||||
#include <sgl_kernel/utils.cuh> // For LaunchKernel, SGL_DEVICE
|
||||
#include <sgl_kernel/vec.cuh> // For AlignedVector
|
||||
|
||||
#include <dlpack/dlpack.h>
|
||||
#include <tvm/ffi/container/tensor.h>
|
||||
|
||||
namespace sglang {
|
||||
|
||||
/**
|
||||
* \brief Element-wise scale using vectorized 128-bit loads/stores.
|
||||
*
|
||||
* \tparam T Element type: fp16_t | bf16_t | fp32_t
|
||||
* \tparam kVecN Elements per vector load (e.g. 8 for fp16)
|
||||
* \tparam kUsePDL Whether to emit the PDL wait/trigger pair
|
||||
* \param dst Output buffer, `n_total` elements
|
||||
* \param src Input buffer, `n_total` elements
|
||||
* \param factor Runtime scale factor
|
||||
* \param n_total Number of elements to scale
|
||||
*/
|
||||
template <typename T, int kVecN, bool kUsePDL>
|
||||
__global__ void scale_kernel(T* __restrict__ dst,
|
||||
const T* __restrict__ src,
|
||||
float factor,
|
||||
uint32_t n_total) {
|
||||
using vec_t = device::AlignedVector<T, kVecN>;
|
||||
const uint32_t n_vecs = n_total / kVecN;
|
||||
|
||||
// If using PDL, wait for primary kernel before any global memory load.
|
||||
// This is NOT a synchronization point, which means some threads can early exit before this.
|
||||
device::PDLWaitPrimary<kUsePDL>();
|
||||
|
||||
// --- vectorised body ---
|
||||
const uint32_t vec_stride = blockDim.x * gridDim.x;
|
||||
for (uint32_t vi = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
vi < n_vecs;
|
||||
vi += vec_stride) {
|
||||
vec_t v;
|
||||
v.load(src, vi);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kVecN; ++i) {
|
||||
v[i] = static_cast<T>(static_cast<float>(v[i]) * factor);
|
||||
}
|
||||
v.store(dst, vi);
|
||||
}
|
||||
|
||||
// --- scalar tail ---
|
||||
const uint32_t base = n_vecs * kVecN;
|
||||
const uint32_t scalar_stride = blockDim.x * gridDim.x;
|
||||
for (uint32_t i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
base + i < n_total;
|
||||
i += scalar_stride) {
|
||||
dst[base + i] = static_cast<T>(static_cast<float>(src[base + i]) * factor);
|
||||
}
|
||||
|
||||
// If using PDL, signal for the secondary kernel to start after all threads have finished
|
||||
// This is NOT a synchronization point, which means some threads can early exit before this.
|
||||
device::PDLTriggerSecondary<kUsePDL>();
|
||||
}
|
||||
|
||||
/**
|
||||
* \brief Validate the tensors, select the vector width, launch `scale_kernel`.
|
||||
*
|
||||
* \tparam T Element type: fp16_t | bf16_t | fp32_t
|
||||
* \tparam kUsePDL Whether to launch with PDL enabled
|
||||
* \param dst Output tensor; same shape / dtype / device as `src`
|
||||
* \param src Input tensor on CUDA
|
||||
* \param factor Runtime scale factor
|
||||
*/
|
||||
template <typename T, bool kUsePDL>
|
||||
void scale(tvm::ffi::TensorView dst, tvm::ffi::TensorView src, float factor) {
|
||||
using namespace host;
|
||||
|
||||
// 1. Validate input tensors with TensorMatcher
|
||||
SymbolicSize N = {"num_elements"};
|
||||
SymbolicDevice device_;
|
||||
device_.set_options<kDLCUDA>();
|
||||
|
||||
TensorMatcher({N}) //
|
||||
.with_dtype<T>()
|
||||
.with_device<kDLCUDA>(device_)
|
||||
.verify(dst)
|
||||
.verify(src); // same shape / dtype / device as dst
|
||||
|
||||
const uint32_t n = static_cast<uint32_t>(N.unwrap());
|
||||
const DLDevice device = device_.unwrap();
|
||||
|
||||
CHECK_HOST(n > 0) << "scale: num_elements must be > 0, got " << n;
|
||||
|
||||
// 2. Choose vector width for 128-bit loads (16 bytes)
|
||||
// fp16/bf16: 8 elements x 2 bytes = 16 bytes
|
||||
// fp32: 4 elements x 4 bytes = 16 bytes
|
||||
// We encourage using `device::kMaxVecBytes`, which will change according to
|
||||
// the target architecture and can enable 256-bit vectorization on SM100+ if desired.
|
||||
// But 128-bit is more commonly adapted for better compatibility,
|
||||
// so it's still ok to hardcode 16 here just for simplicity.
|
||||
constexpr int kVecN = 16 / sizeof(T);
|
||||
const uint32_t n_work_items = div_ceil(n, static_cast<uint32_t>(kVecN));
|
||||
|
||||
// 3. Launch
|
||||
constexpr uint32_t kBlockSize = 256;
|
||||
const uint32_t grid = div_ceil(n_work_items, kBlockSize);
|
||||
|
||||
// PDL feature is 100% optional. Without `enable_pdl`, the code should still be correct.
|
||||
// Try to enable it if profiling shows that it can benefit the performance of this kernel.
|
||||
LaunchKernel(grid, kBlockSize, device).enable_pdl(kUsePDL)(
|
||||
scale_kernel<T, kVecN, kUsePDL>,
|
||||
static_cast<T*>(dst.data_ptr()),
|
||||
static_cast<const T*>(src.data_ptr()),
|
||||
factor,
|
||||
n);
|
||||
}
|
||||
|
||||
} // namespace sglang
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Include headers from `sgl_kernel/` — **not** raw CUDA headers for anything already covered
|
||||
- Use `TensorMatcher` for all tensor validation; never manually check shape/dtype/device
|
||||
- Use `AlignedVector` for vectorised 128-bit loads/stores — significant bandwidth win
|
||||
- Use `LaunchKernel` — it resolves the stream and checks errors automatically
|
||||
- Use `CHECK_HOST(cond) << ...` for runtime assertions with useful error messages (zero overhead when the check passes)
|
||||
- Prefer passing runtime scalars like `factor` directly unless compile-time specialisation is genuinely required
|
||||
- `fp16_t` / `bf16_t` / `fp32_t` are the project's type aliases (from `utils.cuh`)
|
||||
- `device::cast<To, From>` or `DTypeTrait<T>::from(val)` for cross-type conversions
|
||||
- `device::math::` functions for device math instead of bare `__` intrinsics if possible.
|
||||
- Consider PDL — it can help when the kernel has prologue work to overlap. Place `PDLWaitPrimary()` right before the first read of upstream data, not at the top of the kernel
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Add the Python wrapper in `kernels/ops/`
|
||||
|
||||
The wrapper lives next to its functional group under `python/sglang/kernels/ops/`, not beside the CUDA source — `kernels/jit/` holds only the JIT infrastructure (`csrc/`, `include/`, `utils/`, `benchmark/`). Create `python/sglang/kernels/ops/elementwise/scale.py`:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.utils import (
|
||||
cache_once,
|
||||
is_arch_support_pdl,
|
||||
load_jit,
|
||||
make_cpp_args,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tvm_ffi.module import Module
|
||||
|
||||
|
||||
@cache_once
|
||||
def _jit_scale_module(dtype: torch.dtype) -> Module:
|
||||
"""Compile and cache the JIT scale module for a given dtype."""
|
||||
# Checks on the compile key live here, not in `scale`: `cache_once` keys on
|
||||
# `dtype`, so this runs once per specialisation instead of once per call.
|
||||
if dtype not in (torch.float16, torch.bfloat16, torch.float32):
|
||||
raise RuntimeError(
|
||||
f"Unsupported dtype {dtype}. Supported: float16, bfloat16, float32"
|
||||
)
|
||||
args = make_cpp_args(dtype, is_arch_support_pdl())
|
||||
return load_jit(
|
||||
"scale",
|
||||
*args,
|
||||
cuda_files=["elementwise/scale.cuh"],
|
||||
cuda_wrappers=[("scale", f"scale<{args}>")],
|
||||
)
|
||||
|
||||
|
||||
def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) -> torch.Tensor:
|
||||
"""
|
||||
Element-wise scale: dst = src * factor.
|
||||
|
||||
Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
src : CUDA tensor (FP16 / BF16 / FP32)
|
||||
factor : scale factor
|
||||
out : optional pre-allocated output tensor (same shape/dtype as src)
|
||||
|
||||
Returns
|
||||
-------
|
||||
Scaled tensor (dst = src * factor).
|
||||
"""
|
||||
# DO NOT add proactive validation here: every check costs interpreter time
|
||||
# on a per-forward path. Tensor invariants belong in the C++ launcher, and
|
||||
# anything about the compile key belongs in `_jit_scale_module`.
|
||||
if out is None:
|
||||
out = torch.empty_like(src)
|
||||
|
||||
module = _jit_scale_module(src.dtype)
|
||||
module.scale(out, src, factor)
|
||||
return out
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Use `cache_once` — **not** `functools.lru_cache` (incompatible with `torch.compile`)
|
||||
- `load_jit` first arg(s) form the unique build marker; same marker = same cached binary
|
||||
- Only include compile-time specialisation knobs in the build marker; runtime values like `factor` should stay runtime unless the kernel truly needs templating
|
||||
- `cuda_wrappers`: `(export_name, kernel_symbol)` — `export_name` is called from Python
|
||||
- `make_cpp_args(dtype, ...)` converts `torch.dtype` to C++ type alias:
|
||||
- `is_arch_support_pdl()` checks if the current architecture supports PDL, which is typically passed as a template argument to the kernel.
|
||||
- Keep the entry point thin (see Conventions). What Python must still check goes in the `@cache_once` module factory, not in the entry point: `cache_once` keys on its arguments, so a check there costs one evaluation per specialisation instead of one per call — that is where the supported-dtype guard lives. Tensor invariants belong in the C++ launcher; if something here never reaches a `.verify(...)`, close that gap on the C++ side rather than in Python
|
||||
|
||||
| `torch.dtype` | C++ type |
|
||||
|--------------------|------------|
|
||||
| `torch.float16` | `fp16_t` |
|
||||
| `torch.bfloat16` | `bf16_t` |
|
||||
| `torch.float32` | `fp32_t` |
|
||||
|
||||
---
|
||||
|
||||
## Step 3 (optional): Tune JIT build flags
|
||||
|
||||
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
|
||||
|
||||
```python
|
||||
return load_jit(
|
||||
"scale",
|
||||
*args,
|
||||
cuda_files=["elementwise/scale.cuh"],
|
||||
cuda_wrappers=[("scale", f"scale<{args}>")],
|
||||
extra_cuda_cflags=["-O3", "--use_fast_math"],
|
||||
)
|
||||
```
|
||||
|
||||
If your kernel requires SM90+, raise a clear Python error before calling `load_jit`. Arch gating is one of the checks that has to live in Python — it decides whether to compile at all, so the C++ launcher never gets to run:
|
||||
|
||||
```python
|
||||
if torch.cuda.get_device_capability()[0] < 9:
|
||||
raise RuntimeError("This kernel requires SM90 (Hopper) or later")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Write tests (required)
|
||||
|
||||
JIT kernel correctness tests and benchmarks live under `test/registered/kernels/ops/<group>/` and `test/registered/kernels/benchmark/<group>/`, mirroring the wrapper's group under `python/sglang/kernels/ops/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/kernels/jit/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
|
||||
|
||||
- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`.
|
||||
- **Nightly kernel suite**: register with `stage="nightly"` plus the `runner_config` of the machine it needs (e.g. `1-gpu-large`), giving the `nightly-test-1-gpu-large` suite. `.github/workflows/nightly-test-nvidia.yml` sets `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` for the whole nightly run, so the expanded parameter grids apply automatically (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). There is no separate kernel-only nightly job: every nightly test on one machine type shares that machine's suite.
|
||||
|
||||
Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing):
|
||||
|
||||
```python
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
# Optional B200/SM100 registration for tests that cover Blackwell-specific code paths
|
||||
# register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
|
||||
# Optional second registration: same file also runs nightly, same form,
|
||||
# stage is just "nightly" there (and no `nightly=True`)
|
||||
# register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
|
||||
```
|
||||
|
||||
CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. Nightly uses the same shape with `stage="nightly"`; the single-string `suite=` form is left only for `stress` and non-CUDA pools.
|
||||
|
||||
Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery.
|
||||
|
||||
Use `register_cuda_ci(..., disabled="reason")` if the file must stay in-tree but should be skipped in CI (e.g. multi-GPU only).
|
||||
|
||||
**Run like CI** (from repo root):
|
||||
|
||||
```bash
|
||||
(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-1-gpu-large)
|
||||
# For B200/SM100-specific coverage:
|
||||
(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-4-gpu-b200)
|
||||
```
|
||||
|
||||
For fast iteration you can still run `pytest` on a single file locally; CI coverage is via `run_suite.py`.
|
||||
|
||||
Create `test/registered/kernels/ops/elementwise/test_scale.py`:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
import torch
|
||||
from sglang.kernels.ops.elementwise.scale import scale
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||
@pytest.mark.parametrize("size", [1, 127, 128, 1024, 4097]) # cover tail remainder
|
||||
@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0, 3.0])
|
||||
def test_scale_correctness(dtype, size, factor):
|
||||
src = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = scale(src, factor)
|
||||
expected = src * factor
|
||||
|
||||
rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
|
||||
torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||
def test_scale_out_param(dtype):
|
||||
src = torch.randn(1024, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(src)
|
||||
result = scale(src, 2.0, out=out)
|
||||
assert result is out
|
||||
torch.testing.assert_close(out, src * 2.0, rtol=1e-2, atol=1e-2)
|
||||
|
||||
|
||||
def test_scale_cpu_error():
|
||||
src = torch.randn(128, dtype=torch.float16) # CPU tensor
|
||||
with pytest.raises(RuntimeError, match="CUDA"):
|
||||
scale(src, 2.0)
|
||||
|
||||
|
||||
def test_scale_unsupported_dtype():
|
||||
src = torch.randint(0, 10, (128,), dtype=torch.int32, device="cuda")
|
||||
with pytest.raises(RuntimeError, match="dtype"):
|
||||
scale(src, 2.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(pytest.main([__file__, "-v", "-s"]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Add a benchmark (required)
|
||||
|
||||
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
|
||||
|
||||
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
|
||||
|
||||
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
|
||||
- **`@marker.parametrize(names, vals, ci_vals=None)`** — stackable decorator that adds a row axis (pytest-style). Each `@parametrize` adds one (or more, correlated) parameter the benchmark is swept over (Cartesian product across all `parametrize` decorators). `names` may be a single name (`"size"`) or a comma-separated correlated tuple axis (`"h,d"`, with `vals` then a list of tuples like `[(1, 64), (2, 128)]`). Pass the optional third `ci_vals` for a smaller sweep that is auto-selected under `is_in_ci()` — this is the built-in CI-shrinking mechanism, so you usually don't need `get_benchmark_range` for swept axes.
|
||||
- **`marker.do_bench(fn, *, input_args=(), input_kwargs={}, ...)`** — runs `fn` under CUDA graph (default) or a naive loop, returns a `BenchResult`. Key knobs:
|
||||
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
|
||||
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
|
||||
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
|
||||
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning).
|
||||
- `use_cuda_graph=False` for kernels that can't be captured.
|
||||
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
|
||||
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
|
||||
- **`utils.create_random(*shape)` / `utils.create_empty(*shape)`** — shorthand for `torch.randn` / `torch.empty` with `DEFAULT_DTYPE` (`bfloat16`) and `DEFAULT_DEVICE` (`"cuda"`). Override via the `dtype=` / `device=` kwargs.
|
||||
- **`utils.get_benchmark_range(full_range, ci_range)`** — returns the smaller `ci_range` under CI (`is_in_ci()`), the `full_range` locally. Still available for the `benchmark(...)` column axis (which has no `ci_vals`); for `parametrize` row axes prefer the built-in `ci_vals` argument.
|
||||
|
||||
Create `test/registered/kernels/benchmark/elementwise/bench_scale.py`:
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.benchmark import marker
|
||||
from sglang.kernels.jit.benchmark.utils import create_random
|
||||
from sglang.kernels.ops.elementwise.scale import scale as jit_scale
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
|
||||
|
||||
|
||||
@torch.compile()
|
||||
def torch_impl_scale(src: torch.Tensor, factor: float) -> torch.Tensor:
|
||||
return src * factor
|
||||
|
||||
|
||||
FN_MAP = {
|
||||
"jit": jit_scale,
|
||||
"torch": torch_impl_scale,
|
||||
}
|
||||
|
||||
|
||||
# `parametrize(name, full_vals, ci_vals)`: the 3rd arg is the smaller sweep
|
||||
# auto-selected under CI; the full range runs locally.
|
||||
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096, 65536]) # 1K .. 512K
|
||||
@marker.benchmark("impl", ["jit", "torch"])
|
||||
def benchmark(size: int, impl: str):
|
||||
src = create_random(size)
|
||||
factor = 2.0
|
||||
return marker.do_bench(
|
||||
FN_MAP[impl],
|
||||
input_args=(src, factor),
|
||||
# `src` is read -> clone it per iter to avoid L2 reuse; factor is a scalar.
|
||||
graph_clone_args=(0,),
|
||||
# Defaults already report bandwidth: memory_args="all" counts src,
|
||||
# memory_output="out" counts the returned tensor -> bytes(src)+bytes(out).
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
benchmark.run()
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- The `line_arg` name passed to `benchmark` (`"impl"` here) must match a parameter on `benchmark(...)`; same for every `parametrize` name (`"size"`).
|
||||
- Stack `@parametrize` once per swept axis. The required `@marker.benchmark` is the **innermost** decorator (bottom of the stack, directly above the function) — `@parametrize` rows go above it.
|
||||
- Prefer `create_random` / `create_empty` from `utils.py` over open-coding `torch.randn(..., dtype=..., device=...)`.
|
||||
- The GB/s column appears by default (`memory_args="all"` + `memory_output="out"`). For memory-bound kernels it's the most informative number; scope `memory_args` / `memory_output` to the tensors actually touched if the defaults over- or under-count. For compute-bound kernels where bandwidth is misleading, set `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1` (or `disable_log_bandwidth=True`).
|
||||
- For in-place kernels (which return `None`), pass the written tensors via `memory_output=(...)` since the `"out"` default would capture nothing.
|
||||
- Tune `graph_clone_args` / `graph_clone_kwargs` to all the arguments that might be read by the kernel. We can only skip cloning for write-only args. For in-place modified args, we still need to clone them to get accurate timing (reusing the same buffer keeps it L2-hot and skews results).
|
||||
- Call `benchmark.run()` (no `print_data=` kwarg — the marker framework prints directly).
|
||||
|
||||
Run locally:
|
||||
|
||||
```bash
|
||||
python test/registered/kernels/benchmark/elementwise/bench_scale.py
|
||||
```
|
||||
|
||||
Run the benchmark suite the way CI does:
|
||||
|
||||
```bash
|
||||
cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config`; starred args and non-literal values break AST collection
|
||||
- **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations
|
||||
- **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...`
|
||||
- **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way.
|
||||
- **Missing GB/s column**: the column is on by default; check that `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH` is not `1` and `disable_log_bandwidth` is not `True`. For in-place kernels (return `None`) the `memory_output="out"` default counts nothing — pass the written tensors via `memory_output=(...)`
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `docs/docs/developer_guide/development_jit_kernel_guide.mdx`
|
||||
- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
|
||||
- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
|
||||
- `python/sglang/kernels/jit/utils/compile.py` — `load_jit`, `make_cpp_args`
|
||||
- `python/sglang/kernels/jit/utils/common.py` — `cache_once`, `should_run_full_tests`, `get_ci_test_range`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`, `is_type`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/ffi.h` — `ffi::empty`, `ffi::empty_like`, `ffi::from_blob`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/utils.cuh` — type aliases, `LaunchKernel`, `SGL_DEVICE`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/vec.cuh` — `AlignedVector`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/tile.cuh` — `tile::Memory`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/bits.h` — compile-time bit helpers
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/type.cuh` — `DTypeTrait`, `packed_t`, `device::cast`, `device::unpack`, `ReductionTrait`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/math.cuh` — `device::math::`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/warp.cuh` — `warp::reduce<Op>` and `reduce_sum/max/min` wrappers
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
|
||||
- `python/sglang/kernels/jit/csrc/add_constant.cuh` — minimal runnable reference
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
|
||||
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
|
||||
- `python/sglang/kernels/jit/benchmark/utils.py` — `create_random` / `create_empty` / `get_benchmark_range` helpers and `DEFAULT_DTYPE` / `DEFAULT_DEVICE`
|
||||
- `test/registered/kernels/benchmark/layernorm/bench_qknorm.py` — real example: multi-axis `parametrize` (with `ci_vals`) + in-place `memory_output`
|
||||
- `test/registered/kernels/benchmark/kvcache/bench_store_cache.py` — real example: scoped `memory_args` / `memory_output` + selective `graph_clone_args`
|
||||
|
||||
## Summary of Files Created
|
||||
|
||||
```
|
||||
python/sglang/kernels/jit/csrc/elementwise/scale.cuh # NEW: CUDA kernel
|
||||
python/sglang/kernels/ops/elementwise/scale.py # NEW: Python wrapper
|
||||
test/registered/kernels/ops/elementwise/test_scale.py # NEW: Tests
|
||||
test/registered/kernels/benchmark/elementwise/bench_scale.py # NEW: Benchmark
|
||||
```
|
||||
@@ -0,0 +1,367 @@
|
||||
---
|
||||
name: add-sgl-kernel
|
||||
description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel to sgl-kernel (including tests & benchmarks)
|
||||
---
|
||||
|
||||
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
|
||||
|
||||
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||
|
||||
## Goal
|
||||
|
||||
Add a new operation that scales each element of a tensor by a scalar factor:
|
||||
|
||||
- Input: tensor `x` (CUDA) and scalar `factor` (float)
|
||||
- Output: `x * factor` (element-wise, in-place or into pre-allocated `out`)
|
||||
- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
|
||||
- Dispatched via `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro (defined in `python/sglang/kernels/aot/include/utils.h`)
|
||||
|
||||
## Two rules of thumb (must follow)
|
||||
|
||||
1. **Prefer `python/sglang/kernels/jit` first** when the kernel does **not** depend on CUTLASS or another large C++ project. This is the default path for lightweight kernels that benefit from rapid iteration.
|
||||
2. **Prefer `sgl-kernel`** when the kernel **does** depend on CUTLASS or another large C++ project, or when it should be part of the AOT wheel / torch op registration flow.
|
||||
3. **Exception**: if the dependency is `flashinfer`, or CUTLASS that is already provided through `flashinfer`, the kernel can still be implemented as `jit_kernel`.
|
||||
|
||||
In addition, every new kernel must ship with:
|
||||
|
||||
- **Tests** (pytest)
|
||||
- **A benchmark script** (triton.testing)
|
||||
|
||||
---
|
||||
|
||||
## Repository integration map
|
||||
|
||||
You will typically touch these files/areas:
|
||||
|
||||
- Implementation: `python/sglang/kernels/aot/csrc/elementwise/scale.cu` (pick the right subdirectory)
|
||||
- Public declarations: `python/sglang/kernels/aot/include/sgl_kernel_ops.h`
|
||||
- Torch extension registration: `python/sglang/kernels/aot/csrc/common_extension.cc`
|
||||
- Build: `python/sglang/kernels/aot/CMakeLists.txt` (`set(SOURCES ...)`)
|
||||
- Python API: `python/sglang/kernels/aot/python/sgl_kernel/` and `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||
- Tests: `python/sglang/kernels/aot/tests/test_scale.py`
|
||||
- Benchmarks: `python/sglang/kernels/aot/benchmark/bench_scale.py`
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Implement the kernel in `csrc/`
|
||||
|
||||
Pick the right subdirectory:
|
||||
|
||||
- `csrc/elementwise/` — for element-wise ops (our example)
|
||||
- `csrc/gemm/`, `csrc/attention/`, `csrc/moe/` — for other categories
|
||||
|
||||
Create `python/sglang/kernels/aot/csrc/elementwise/scale.cu`:
|
||||
|
||||
```cpp
|
||||
#include <ATen/cuda/CUDAContext.h>
|
||||
#include <c10/cuda/CUDAGuard.h>
|
||||
#include <torch/all.h>
|
||||
|
||||
#include "utils.h" // DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16
|
||||
|
||||
// scale_kernel: out[i] = input[i] * factor
|
||||
// Supports float, half (__half), __nv_bfloat16 via template T
|
||||
template <typename T>
|
||||
__global__ void scale_kernel(T* __restrict__ out,
|
||||
const T* __restrict__ input,
|
||||
float factor,
|
||||
int64_t n) {
|
||||
int64_t idx = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
|
||||
if (idx < n) {
|
||||
out[idx] = static_cast<T>(static_cast<float>(input[idx]) * factor);
|
||||
}
|
||||
}
|
||||
|
||||
void scale(at::Tensor& out, const at::Tensor& input, double factor) {
|
||||
TORCH_CHECK(input.is_cuda(), "input must be a CUDA tensor");
|
||||
TORCH_CHECK(input.is_contiguous(), "input must be contiguous");
|
||||
TORCH_CHECK(out.is_cuda(), "out must be a CUDA tensor");
|
||||
TORCH_CHECK(out.is_contiguous(), "out must be contiguous");
|
||||
TORCH_CHECK(out.sizes() == input.sizes(), "out and input must have the same shape");
|
||||
TORCH_CHECK(out.scalar_type() == input.scalar_type(),
|
||||
"out and input must have the same dtype");
|
||||
|
||||
const int64_t n = input.numel();
|
||||
const int threads = 256;
|
||||
const int blocks = (n + threads - 1) / threads;
|
||||
|
||||
const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
|
||||
const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
|
||||
|
||||
// Dispatches over float, float16, bfloat16
|
||||
DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16(input.scalar_type(), c_type, [&] {
|
||||
scale_kernel<c_type><<<blocks, threads, 0, stream>>>(
|
||||
static_cast<c_type*>(out.data_ptr()),
|
||||
static_cast<const c_type*>(input.data_ptr()),
|
||||
static_cast<float>(factor),
|
||||
n);
|
||||
cudaError_t status = cudaGetLastError();
|
||||
TORCH_CHECK(status == cudaSuccess,
|
||||
"scale_kernel launch failed: ", cudaGetErrorString(status));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Use `at::Tensor` (PyTorch tensors), `TORCH_CHECK` for validation, `at::cuda::getCurrentCUDAStream()` for stream
|
||||
- Keep Python wrappers thin; do shape/dtype/device validation in C++ right around the launch path
|
||||
- `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` covers `float`, `half` (FP16), `__nv_bfloat16` (BF16)
|
||||
- Add device error checking after every kernel launch
|
||||
- If a kernel only works on certain architectures, enforce that with `TORCH_CHECK` and skip logic in tests
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Add a C++ declaration in `include/sgl_kernel_ops.h`
|
||||
|
||||
Edit `python/sglang/kernels/aot/include/sgl_kernel_ops.h`, add to the elementwise section:
|
||||
|
||||
```cpp
|
||||
void scale(at::Tensor& out, const at::Tensor& input, double factor);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Register the op in `csrc/common_extension.cc`
|
||||
|
||||
Edit `python/sglang/kernels/aot/csrc/common_extension.cc`, inside `TORCH_LIBRARY_FRAGMENT(sgl_kernel, m)`:
|
||||
|
||||
```cpp
|
||||
// From csrc/elementwise
|
||||
m.def("scale(Tensor! out, Tensor input, float factor) -> ()");
|
||||
m.impl("scale", torch::kCUDA, &scale);
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- `Tensor!` means in-place / mutable output argument
|
||||
- The schema is important for `torch.compile` and for consistent call signatures
|
||||
- Keep the torch schema in PyTorch scalar types (`float` here), but note that the C++ launcher signature still needs `double` for scalar arguments accepted by `torch::Library`
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Add the new source file to `CMakeLists.txt`
|
||||
|
||||
Edit `python/sglang/kernels/aot/CMakeLists.txt`, add to `set(SOURCES ...)`:
|
||||
|
||||
```cmake
|
||||
csrc/elementwise/scale.cu
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Keep the list **alphabetically sorted** (the file explicitly requires this)
|
||||
- If the kernel has arch constraints, reflect that in tests/benchmarks via skip logic
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Expose a Python API under `python/sglang/kernels/aot/python/sgl_kernel/`
|
||||
|
||||
Prefer following the existing module organization first. For elementwise kernels, the usual pattern is:
|
||||
|
||||
- implement the Python wrapper in `python/sglang/kernels/aot/python/sgl_kernel/elementwise.py`
|
||||
- then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||
|
||||
For example, in `python/sglang/kernels/aot/python/sgl_kernel/elementwise.py`, add:
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
def scale(
|
||||
input: torch.Tensor,
|
||||
factor: float,
|
||||
out: torch.Tensor | None = None,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Element-wise scale: out = input * factor.
|
||||
|
||||
Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input : CUDA input tensor
|
||||
factor : scale factor (float)
|
||||
out : optional pre-allocated CUDA output tensor (same shape/dtype as input)
|
||||
"""
|
||||
if out is None:
|
||||
out = torch.empty_like(input)
|
||||
torch.ops.sgl_kernel.scale.default(out, input, factor)
|
||||
return out
|
||||
```
|
||||
|
||||
Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py` following the existing import style used by other kernels.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Write tests (required)
|
||||
|
||||
Create `python/sglang/kernels/aot/tests/test_scale.py`:
|
||||
```python
|
||||
import pytest
|
||||
|
||||
import torch
|
||||
import sgl_kernel
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||
@pytest.mark.parametrize("size", [128, 1024, 4096, 65536])
|
||||
@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0])
|
||||
def test_scale_correctness(dtype, size, factor):
|
||||
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(input)
|
||||
|
||||
result = sgl_kernel.scale(input, factor, out=out)
|
||||
assert result is out
|
||||
|
||||
expected = input * factor
|
||||
rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
|
||||
torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
|
||||
|
||||
|
||||
def test_scale_shape_mismatch():
|
||||
input = torch.randn(128, dtype=torch.float16, device="cuda")
|
||||
out = torch.empty(256, dtype=torch.float16, device="cuda")
|
||||
with pytest.raises(RuntimeError, match="same shape"):
|
||||
sgl_kernel.scale(input, 2.0, out=out)
|
||||
|
||||
|
||||
def test_scale_cpu_input():
|
||||
input = torch.randn(128, dtype=torch.float16) # CPU
|
||||
out = torch.empty_like(input)
|
||||
with pytest.raises(RuntimeError, match="CUDA"):
|
||||
sgl_kernel.scale(input, 2.0, out=out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(pytest.main([__file__, "-q"]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Add a benchmark (required)
|
||||
|
||||
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
|
||||
|
||||
```python
|
||||
import itertools
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.testing
|
||||
|
||||
import sgl_kernel
|
||||
from sglang.utils import is_in_ci
|
||||
|
||||
IS_CI = is_in_ci()
|
||||
|
||||
dtypes = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
|
||||
sizes = [4096] if IS_CI else [2**n for n in range(10, 20)] # 1K … 512K
|
||||
factors = [2.0]
|
||||
|
||||
configs = list(itertools.product(dtypes, sizes))
|
||||
|
||||
|
||||
def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
|
||||
return input * factor
|
||||
|
||||
|
||||
@triton.testing.perf_report(
|
||||
triton.testing.Benchmark(
|
||||
x_names=["dtype", "size"],
|
||||
x_vals=configs,
|
||||
line_arg="provider",
|
||||
line_vals=["sglang", "torch"],
|
||||
line_names=["SGL Kernel", "PyTorch"],
|
||||
styles=[("green", "-"), ("red", "--")],
|
||||
ylabel="µs (median)",
|
||||
plot_name="scale-performance",
|
||||
args={},
|
||||
)
|
||||
)
|
||||
def benchmark(dtype, size, provider):
|
||||
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(input)
|
||||
factor = 2.0
|
||||
|
||||
if provider == "sglang":
|
||||
fn = lambda: sgl_kernel.scale(input, factor, out=out)
|
||||
else:
|
||||
fn = lambda: torch_scale(input, factor)
|
||||
|
||||
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
|
||||
fn, quantiles=[0.5, 0.2, 0.8]
|
||||
)
|
||||
return 1000 * ms, 1000 * max_ms, 1000 * min_ms
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
benchmark.run(print_data=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Build
|
||||
|
||||
Build:
|
||||
|
||||
```bash
|
||||
cd python/sglang/kernels/aot
|
||||
make build -j16
|
||||
```
|
||||
|
||||
If you need to limit host resource usage:
|
||||
|
||||
```bash
|
||||
cd python/sglang/kernels/aot
|
||||
make build -j1 MAX_JOBS=2 CMAKE_ARGS="-DSGL_KERNEL_COMPILE_THREADS=1"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Validate
|
||||
|
||||
After building successfully, run the test and benchmark:
|
||||
|
||||
```bash
|
||||
pytest python/sglang/kernels/aot/tests/test_scale.py -q
|
||||
python python/sglang/kernels/aot/benchmark/bench_scale.py
|
||||
```
|
||||
|
||||
PR CI also runs `pr-test-sgl-kernel.yml`, including the B200 job
|
||||
`sgl-kernel-b200-test` when kernel changes are detected. Use that job as the
|
||||
Blackwell coverage signal for AOT `sgl-kernel` changes.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Async CUDA errors**: `CUDA_LAUNCH_BLOCKING=1`
|
||||
- **Memory errors**: `compute-sanitizer --tool memcheck python ...`
|
||||
- **Build is too slow / OOM**: reduce `MAX_JOBS` and `SGL_KERNEL_COMPILE_THREADS`
|
||||
- **Binary bloat**: use `python/sglang/kernels/aot/analyze_whl_kernel_sizes.py`
|
||||
- **CMake sources list**: if your `.cu` file is missing from `SOURCES`, the symbol will be undefined at link time
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `python/sglang/kernels/aot/README.md`
|
||||
- `python/sglang/kernels/aot/include/sgl_kernel_ops.h`
|
||||
- `python/sglang/kernels/aot/csrc/common_extension.cc`
|
||||
- `python/sglang/kernels/aot/CMakeLists.txt`
|
||||
- `python/sglang/kernels/aot/include/utils.h` — `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro and friends
|
||||
- `python/sglang/kernels/aot/csrc/elementwise/activation.cu` — reference for the FP16/BF16/FP32 dispatch pattern
|
||||
|
||||
## Summary of Files Created/Modified
|
||||
|
||||
```
|
||||
python/sglang/kernels/aot/csrc/elementwise/scale.cu # NEW: CUDA kernel + launcher
|
||||
python/sglang/kernels/aot/include/sgl_kernel_ops.h # MODIFIED: C++ declaration
|
||||
python/sglang/kernels/aot/csrc/common_extension.cc # MODIFIED: schema + dispatch registration
|
||||
python/sglang/kernels/aot/CMakeLists.txt # MODIFIED: add source file (alphabetical)
|
||||
python/sglang/kernels/aot/python/sgl_kernel/elementwise.py # MODIFIED: Python wrapper
|
||||
python/sglang/kernels/aot/python/sgl_kernel/__init__.py # MODIFIED: re-export Python API
|
||||
python/sglang/kernels/aot/tests/test_scale.py # NEW: tests
|
||||
python/sglang/kernels/aot/benchmark/bench_scale.py # NEW: benchmark
|
||||
```
|
||||
@@ -0,0 +1,413 @@
|
||||
---
|
||||
name: ci-workflow-guide
|
||||
description: Guide to SGLang CI workflow orchestration — stage ordering, fail-fast, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages.
|
||||
---
|
||||
|
||||
# SGLang CI Workflow Orchestration Guide
|
||||
|
||||
This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and aborted on failure across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md).
|
||||
|
||||
---
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- **Suite**: `base-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `base-b-test-1-gpu-small`)
|
||||
- **Test group**: Directory-level registered test group under `test/registered/` (e.g., `hicache` maps to `test/registered/hicache/test_*.py`)
|
||||
- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions |
|
||||
| `.github/workflows/pr-test-extra.yml` | Extra workflow — gated by BOTH `run-ci` and `run-ci-extra` labels |
|
||||
| `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting |
|
||||
| `.github/actions/check-pr-test-health/action.yml` | Cross-job fail-fast: queries API for any failed job |
|
||||
| `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete |
|
||||
| `.github/actions/check-maintenance/action.yml` | Maintenance mode check |
|
||||
| `.github/scripts/ci-labels.cjs` | Resolves the four CI control labels into dispatch axes |
|
||||
| `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests |
|
||||
| `python/sglang/test/ci/ci_register.py` | Test registration (AST-parsed markers), LPT auto-partition |
|
||||
| `python/sglang/test/ci/ci_utils.py` | `run_unittest_files()`: execution, retry, continue-on-error |
|
||||
| `scripts/ci/utils/slash_command_handler.py` | Handles slash commands from PR comments |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ build kernel │
|
||||
└──────┬───────┘
|
||||
│
|
||||
├─ check-changes ──── detects which packages changed
|
||||
│ (main_package, sgl_kernel, jit_kernel, multimodal_gen)
|
||||
│
|
||||
├─ call-gate ──────── pr-gate.yml (draft? label? rate limit?)
|
||||
│
|
||||
├─────────────────────────────────────────────────────┐
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────────────────────────┐ │
|
||||
│ Base A (~3 min) │ │
|
||||
│ pre-flight check │ │
|
||||
│ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ base-a-test-1-gpu-small │ │ │
|
||||
│ │ (small GPUs) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ base-a-test-cpu │ │ │
|
||||
│ │ (CPU) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
└──────┬──────────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────┐ ┌──────────────────────────┐
|
||||
│ Base B (~30 min) │ │ kernel test │
|
||||
│ base tests │ └──────────────────────────┘
|
||||
│ │ ┌──────────────────────────┐
|
||||
│ ┌─────────────────────────────┐ │ │ multimodal gen test │
|
||||
│ │ base-b-test-1-gpu-small │ │ └──────────────────────────┘
|
||||
│ │ (small GPUs, e.g. 5090) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-b-test-1-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-b-test-2-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Base C (~30 min) │
|
||||
│ advanced tests │
|
||||
│ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-c-test-4-gpu-h100 │ │
|
||||
│ │ (H100 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-c-test-8-gpu-h200 │ │
|
||||
│ │ (8 x H200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ base-c-test-4-gpu-b200 │ │
|
||||
│ │ (4 x B200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ Other advanced tests │ │
|
||||
│ │ (DeepEP, PD Disagg, GB300) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ pr-test-finish │
|
||||
│ aggregates all results, fails if │
|
||||
│ any job failed/cancelled │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fails fast (red X) with a root cause annotation.
|
||||
|
||||
**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fail-fast is also disabled.
|
||||
|
||||
---
|
||||
|
||||
## Fail-Fast Layers
|
||||
|
||||
4 layers of fail-fast, from fine to coarse:
|
||||
|
||||
| Layer | Mechanism | Granularity | Disabled on schedule? |
|
||||
|-------|-----------|-------------|----------------------|
|
||||
| **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes |
|
||||
| **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes |
|
||||
| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fail-fast (red X) | Yes |
|
||||
| **4. Stage → stage (cross-stage)** | `wait-for-base-*` + `needs` | Base A fails → base B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
|
||||
|
||||
- **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py`
|
||||
- **Layer 2**: `--continue-on-error` flag in `run_suite.py` — off for PRs, on for scheduled runs
|
||||
- **Layer 3**: `check-pr-test-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
|
||||
- **Layer 4**: `wait-for-base-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
|
||||
|
||||
---
|
||||
|
||||
## Execution Modes
|
||||
|
||||
| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | Manual dispatch (`workflow_dispatch`) |
|
||||
|--------|---------------------|------------------------------|--------------------------------------|
|
||||
| **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only |
|
||||
| **Cross-job fail-fast** | Yes (`check-pr-test-health`) | Yes | Yes |
|
||||
| **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No |
|
||||
| **Retry** | Enabled | Enabled | Enabled |
|
||||
| **max_parallel** | 3 (default), 14 if `max-concurrency` label | 14 | 3 (default), 14 if `max-concurrency` |
|
||||
| **PR gate** | Yes (draft, label, rate limit) | Skipped | Skipped |
|
||||
| **Concurrency** | `cancel-in-progress: true` per PR | Queue (no cancel) | Isolated per stage+SHA |
|
||||
|
||||
Four labels relax these limits for one PR: `bypass-fail-fast`, `parallel-stages`,
|
||||
`max-concurrency`, and `highest-priority` (all three). `.github/scripts/ci-labels.cjs`
|
||||
resolves them; the [contribution guide](https://docs.sglang.io/developer_guide/contribution_guide.html#ci-control-labels)
|
||||
describes what each one does.
|
||||
|
||||
---
|
||||
|
||||
## Stage Gating (`wait-for-jobs` action)
|
||||
|
||||
`wait-for-base-a` and `wait-for-base-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
|
||||
|
||||
**How it works:**
|
||||
1. Calls `listJobsForWorkflowRun` to list all jobs in the current run
|
||||
2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `base-b-test-1-gpu-small (3)`)
|
||||
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
|
||||
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
|
||||
|
||||
**Job specs example** (base-b):
|
||||
```json
|
||||
[
|
||||
{"prefix": "base-b-test-1-gpu-small", "expected_count": 8},
|
||||
{"prefix": "base-b-test-1-gpu-large", "expected_count": 14},
|
||||
{"prefix": "base-b-test-2-gpu-large", "expected_count": 4},
|
||||
{"prefix": "base-b-test-4-gpu-b200", "expected_count": 1}
|
||||
]
|
||||
```
|
||||
|
||||
> **Critical**: `expected_count` must match the matrix size. If you add/remove matrix entries, update the wait job's spec accordingly.
|
||||
|
||||
**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and manual dispatches skip these entirely, allowing parallel execution.
|
||||
|
||||
---
|
||||
|
||||
## Cross-Job Fail-Fast (`check-pr-test-health` action)
|
||||
|
||||
Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`).
|
||||
|
||||
**How it works:**
|
||||
1. Queries `listJobsForWorkflowRun` for the current workflow run
|
||||
2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-pr-test-health` (excludes cascade failures)
|
||||
3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names
|
||||
4. If none → does nothing (step succeeds)
|
||||
|
||||
**Cascade filtering**: When job A fails fast due to the health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fail-fast. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list.
|
||||
|
||||
**Usage pattern:**
|
||||
```yaml
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
...
|
||||
|
||||
- uses: ./.github/actions/check-pr-test-health
|
||||
id: pr-test-health
|
||||
|
||||
- name: Install dependencies # skipped automatically if health check failed
|
||||
... # (default if: success() is false)
|
||||
|
||||
- name: Run test # also skipped
|
||||
...
|
||||
```
|
||||
|
||||
**Visual effect**: Job shows **red X** (failure) with error annotation showing root cause job names. Subsequent steps are naturally skipped (default `if: success()` is false after a failed step). No per-step `if` guards needed.
|
||||
|
||||
**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fail-fast.
|
||||
|
||||
**Error message example:**
|
||||
```
|
||||
Fail-fast: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Within-Suite Failure Handling
|
||||
|
||||
Controlled by `run_unittest_files()` in `python/sglang/test/ci/ci_utils.py`.
|
||||
|
||||
### Flags
|
||||
|
||||
| Flag | PR default | Scheduled default | Effect |
|
||||
|------|------------|-------------------|--------|
|
||||
| `--continue-on-error` | Off | On | Off: stop at first failure. On: run all files, report all failures at end |
|
||||
| `--enable-retry` | On | On | Retry retriable failures (accuracy/perf assertions) |
|
||||
| `--max-attempts` | 2 | 2 | Max attempts per file including initial run |
|
||||
|
||||
### Retry Classification
|
||||
|
||||
When a test fails and retry is enabled, the output is classified:
|
||||
|
||||
**Non-retriable** (checked first — real code errors):
|
||||
`SyntaxError`, `ImportError`, `ModuleNotFoundError`, `NameError`, `TypeError`, `AttributeError`, `RuntimeError`, `CUDA out of memory`, `OOM`, `Segmentation fault`, `core dumped`, `ConnectionRefusedError`, `FileNotFoundError`
|
||||
|
||||
**Retriable** (accuracy/performance):
|
||||
`AssertionError` with comparison patterns (`not greater than`, `not less than`, `not equal to`), `accuracy`, `score`, `latency`, `throughput`, `timeout`
|
||||
|
||||
**Default**: Unknown `AssertionError` → retriable. Other unknown failures → not retriable.
|
||||
|
||||
### How `continue_on_error` is set
|
||||
|
||||
In `pr-test.yml`'s `check-changes` job:
|
||||
- `schedule` runs or `run_all_tests` flag → `continue_on_error = 'true'`
|
||||
- PR runs → `continue_on_error = 'false'`
|
||||
|
||||
Each test job propagates via:
|
||||
```yaml
|
||||
env:
|
||||
CONTINUE_ON_ERROR_FLAG: ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||
run: |
|
||||
python3 run_suite.py --hw cuda --suite <name> $CONTINUE_ON_ERROR_FLAG
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Partitioning
|
||||
|
||||
Large suites are split across matrix jobs using the **LPT (Longest Processing Time) heuristic** in `ci_register.py:auto_partition()`:
|
||||
|
||||
1. Sort tests by `est_time` descending, filename as tie-breaker (deterministic)
|
||||
2. Greedily assign each test to the partition with smallest cumulative time
|
||||
3. Result: roughly equal total time per partition
|
||||
|
||||
**Partition table** (CUDA per-commit suites):
|
||||
|
||||
| Suite | Partitions | Runner | max_parallel |
|
||||
|-------|-----------|--------|-------------|
|
||||
| `base-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
|
||||
| `base-a-test-cpu` | 4 | `ubuntu-latest` | — |
|
||||
| `base-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
|
||||
| `base-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
|
||||
| `base-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
|
||||
| `base-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `base-b-kernel-unit-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `base-b-kernel-unit-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `base-b-kernel-unit-test-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `base-b-kernel-benchmark-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `base-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
|
||||
| `base-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
|
||||
| `base-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
|
||||
| `base-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
|
||||
| `base-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
|
||||
| `base-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
|
||||
| `base-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
|
||||
| `base-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
|
||||
|
||||
> **Suite names are generated**, not hand-written: each comes from a test's `register_*_ci(stage=..., runner_config=...)` as `{stage}-test-{runner_config}`, and `runner_config` maps to the `Runner` column via `scripts/ci/runner_configs.yml`.
|
||||
>
|
||||
> **Note**: Kernel suites (`base-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `base-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
**Workflow usage:**
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
partition: [0, 1, 2, 3, 4, 5, 6, 7]
|
||||
steps:
|
||||
- run: python3 run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
|
||||
--auto-partition-id ${{ matrix.partition }} --auto-partition-size 8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## check-changes Job
|
||||
|
||||
Determines which test suites to run based on file changes.
|
||||
|
||||
### Detection Methods
|
||||
|
||||
| Trigger | Method | Details |
|
||||
|---------|--------|---------|
|
||||
| `pull_request` | `dorny/paths-filter` | Detects changes via GitHub diff |
|
||||
| `workflow_dispatch` (with `pr_head_sha`) | GitHub API | `repos/{repo}/compare/main...{sha}` |
|
||||
| `schedule` / `run_all_tests` | Force all true | Runs everything |
|
||||
|
||||
### Output Flags
|
||||
|
||||
| Output | Triggers |
|
||||
|--------|----------|
|
||||
| `main_package` | Base A/B/C test suites |
|
||||
| `sgl_kernel` | Kernel wheel builds + kernel test suites; also switches B200 jobs to kernel-build runner labels outside `target_stage` mode |
|
||||
| `jit_kernel` | JIT kernel test workflow |
|
||||
| `multimodal_gen` | Multimodal-gen test workflow |
|
||||
|
||||
> **Note**: In `target_stage` mode, `sgl_kernel` is only active when `include_wheel_build=true`. Without that opt-in, kernel-change reruns fail validation instead of running a target stage without freshly built wheels. Outside `target_stage`, `sgl_kernel=true` switches B200 jobs from `4-gpu-b200` / `4-gpu-b200-low-disk` to `4-gpu-b200-kernel` / `4-gpu-b200-kernel-low-disk`.
|
||||
|
||||
---
|
||||
|
||||
## Concurrency Control
|
||||
|
||||
```
|
||||
group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
```
|
||||
|
||||
| Segment | Source | Purpose |
|
||||
|---------|--------|---------|
|
||||
| `event_name` | `github.event_name` | Prevents scheduled runs colliding with fork PRs named `main` |
|
||||
| `branch` | `github.head_ref \|\| github.ref_name` | Per-branch isolation |
|
||||
| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates manual dispatches from main runs |
|
||||
| `stage` | `inputs.target_stage \|\| 'all'` | Allows parallel stage dispatches |
|
||||
|
||||
`cancel-in-progress: true` for `pull_request` events (new push cancels old run), `false` for `workflow_call`.
|
||||
|
||||
---
|
||||
|
||||
## How To: Add a New Stage Job
|
||||
|
||||
1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-base-X, ...]`
|
||||
2. Copy the `if:` condition pattern from an existing same-stage job (handles `target_stage`, `schedule`, `main_package`)
|
||||
3. Add `checkout` step
|
||||
4. Add `check-pr-test-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
|
||||
5. Add `check-maintenance` step
|
||||
6. Add `download-artifact` step if `sgl_kernel` changed
|
||||
7. Add `install dependencies` step
|
||||
8. Add `run test` step with `$CONTINUE_ON_ERROR_FLAG`
|
||||
9. Add `upload-cuda-coredumps` step with `if: always()`
|
||||
10. Register the suite name in `PER_COMMIT_SUITES` in `test/run_suite.py`
|
||||
11. If using matrix, add `--auto-partition-id` and `--auto-partition-size` to the run command
|
||||
12. **Update `wait-for-base-X`** job spec with the new job name and `expected_count` (if matrix)
|
||||
13. **Add the job to `pr-test-finish.needs`** list
|
||||
|
||||
---
|
||||
|
||||
## How To: Debug CI Failures
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---------|-------------|---------------|
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
|
||||
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
|
||||
|
||||
---
|
||||
|
||||
## Slash Commands
|
||||
|
||||
| Command | Effect |
|
||||
|---------|--------|
|
||||
| `/tag-run-ci-label` | Adds `run-ci` label to PR |
|
||||
| `/tag-run-ci-label extra` | Adds both `run-ci` and `run-ci-extra` labels |
|
||||
| `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run |
|
||||
| `/tag-and-rerun-ci` | Adds `run-ci` label + reruns failed |
|
||||
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
|
||||
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
|
||||
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||
|
||||
### Label-gated workflow dispatch (pr-test, pr-test-extra)
|
||||
|
||||
`pr-test.yml` and `pr-test-extra.yml` both listen for `pull_request.labeled` (in addition to `opened`/`synchronize`/`reopened`). The `check-changes.if` gate has two clauses:
|
||||
|
||||
1. **For `labeled` events**: the just-added label must be one of the gating labels (`run-ci` for pr-test, `run-ci` or `run-ci-extra` for pr-test-extra) — otherwise every unrelated label addition would dispatch a full CI run.
|
||||
2. **All events**: the PR must currently carry the required labels.
|
||||
|
||||
This is what lets `/tag-run-ci-label` (and the `extra` variant) trigger a fresh CI run without an extra push.
|
||||
|
||||
**Caveat — skipped runs cannot be un-skipped by `run.rerun()`:** GitHub's rerun API reuses the original event payload, so rerunning a `pull_request`-event run that was skipped because of missing labels will skip again (label set in the frozen payload doesn't update). The only way to recover a label-skipped run is to add the missing label, which fires a fresh `labeled` event with the current label set. `handle_rerun_failed_ci` in the slash handler is for rerunning failed/non-label-skipped runs; it cannot revive label-skipped ones.
|
||||
+1
-1
@@ -16,7 +16,7 @@ These are investigation hints, not a list of changes to apply.
|
||||
| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. |
|
||||
| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. |
|
||||
| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. |
|
||||
| Repeated template detection messages | `parser/template_detection.py` and `parser/template_manager.py` | Check whether a summary already contains the same information. |
|
||||
| Repeated template detection messages | `managers/template_detection.py` and `managers/template_manager.py` | Check whether a summary already contains the same information. |
|
||||
| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. |
|
||||
| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. |
|
||||
| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. |
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
name: cookbook-add-model
|
||||
description: Add a new model to the SGLang Cookbook (docs/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the homepage vendor card. Interactive, multi-phase. Run with /cookbook-add-model.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Add a model to the SGLang Cookbook
|
||||
|
||||
> Migrating an **existing legacy-template page** (one that imports a monolithic
|
||||
> `…/autoregressive/<slug>-deployment.jsx` generator)? Use the
|
||||
> `cookbook-migrate-model` skill instead — same target format, but the legacy
|
||||
> page (not the user) is the source of truth.
|
||||
|
||||
The cookbook is **config-driven**: two shared engines contain NO model-specific code —
|
||||
`docs/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
|
||||
`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
|
||||
a per-model `config` (+ optional `benchmarks`) consumed by both engines, plus an MDX page
|
||||
that imports them. No engine edits.
|
||||
|
||||
**Instantiate the model-agnostic template** (NOT a clone of any live cookbook — the
|
||||
template is decoupled and covers all hardware + all axes):
|
||||
- `templates/config.jsx.tmpl` → `docs/src/snippets/configs/<hf-org>/<model-slug>.jsx`
|
||||
- `templates/benchmarks.jsx.tmpl` → `…/<model-slug>-benchmarks.jsx` (skip if no numbers)
|
||||
- `templates/page.mdx.tmpl` → `docs/cookbook/<category>/<Vendor>/<ModelName>.mdx`
|
||||
|
||||
The template uses explicit `__TOKEN__` placeholders; you fill them, prune what the model
|
||||
lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a populated
|
||||
*instance* you can consult, but is not the template.
|
||||
|
||||
**Deep references (read on demand, don't inline):**
|
||||
- [references/authoring-reference.md](references/authoring-reference.md) — field-by-field config / cells / playground / MDX contract.
|
||||
- [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose.
|
||||
- [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work).
|
||||
- [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`).
|
||||
- [references/diffusion-authoring.md](references/diffusion-authoring.md) — required opening, tag, command-picker, and feature-overlay contract for diffusion model pages. Read before editing any `docs/cookbook/diffusion/<Vendor>/<Model>.mdx` page.
|
||||
|
||||
## Architecture at a glance
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ cookbook/<category>/<Vendor>/<Model>.mdx │
|
||||
│ import { Deployment } from "/src/snippets/_deployment.jsx"; │
|
||||
│ import { Playground } from "/src/snippets/_playground.jsx"; │
|
||||
│ import { config } from "/src/snippets/configs/.../X.jsx"; │
|
||||
│ <Deployment config={config} /> <Playground config={config} />│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (config passed as React prop)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ src/snippets/configs/<vendor>/<model>.jsx │
|
||||
│ export const config = { │
|
||||
│ supportedHardware, variants, quantizations, strategies, ... │
|
||||
│ cells: [ { match:{hw,variant,quant,strategy,nodes}, │
|
||||
│ env:[...], flags:[...] }, ... ], // 5-dim matrix │
|
||||
│ playgroundFeatures: { attention, moe, parsers, ... }, │
|
||||
│ }; │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (consumed by BOTH engines — no model code in engines)
|
||||
▼
|
||||
┌──────────────────────────────────┬──────────────────────────────┐
|
||||
│ _deployment.jsx │ _playground.jsx │
|
||||
│ Renders the verified matrix; │ Renders override chips + │
|
||||
│ one cell → its env/flags. │ diff against the cell. │
|
||||
└──────────────────────────────────┴──────────────────────────────┘
|
||||
```
|
||||
|
||||
The two widgets stay in sync via: the **URL hash** (deploy mirrors its selection;
|
||||
playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatches on
|
||||
every change; playground listens — `replaceState` doesn't fire `hashchange`), and the
|
||||
shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders).
|
||||
|
||||
> The main template is **autoregressive**. Diffusion pages use
|
||||
> `templates/diffusion-page.mdx.tmpl` plus
|
||||
> [references/diffusion-authoring.md](references/diffusion-authoring.md); do not force the
|
||||
> autoregressive deployment matrix on them. Omni pages follow their own category structure.
|
||||
> All categories still obey the Mintlify / NEW-tag / docs.json / category-card / validation
|
||||
> rules below.
|
||||
|
||||
---
|
||||
|
||||
**Interactive, multi-step workflow. Collect inputs incrementally — don't ask for
|
||||
everything upfront.** The real work is the verified `cells[]` recipes + measured
|
||||
benchmarks (Phases 2 + 4); everything else is filling the template.
|
||||
|
||||
## Phase 1 — Collect inputs
|
||||
|
||||
1. **Model card** — HuggingFace repo/URL. **Fetch the page** and extract description,
|
||||
param count, architecture, context length, **license**. (Fetching guards factual bugs
|
||||
like an off-by-a-few-B param count.) If the model isn't public, ask the user.
|
||||
2. **Variants / quantizations** — keep separate: variants are size/mode (e.g. Flash/Pro,
|
||||
Instruct/Thinking); quantizations come from the HF card / linked repos (BF16/FP8/FP4/…).
|
||||
Default to BF16 when a full-precision repo exists.
|
||||
3. **Tested hardware + parallelism** — which platforms are actually tested, and TP/EP/DP
|
||||
for each. List only tested hw (unlisted greys out).
|
||||
4. **Verified launch recipes** — the full `sglang serve` flags per
|
||||
(hw × variant × quant × strategy × nodes) combo → these become `cells[]`. Rewrite any
|
||||
`python -m sglang.launch_server` to `sglang serve` form.
|
||||
5. **sglang version / image tag** — ask which sglang build the recipes + benchmarks ran on
|
||||
(a release like `0.5.x`, or `main`/nightly). **Never guess or hallucinate it.** This one
|
||||
tag fills `dockerImages` and the benchmarks' `sglang_version`; when the user is unsure,
|
||||
default the image to `lmsysorg/sglang:dev` (nightly) rather than inventing a release.
|
||||
6. **Pre-flight**: `gh pr list --repo sgl-project/sglang --search "<model>"` (dup check).
|
||||
|
||||
**Hardware reference** (the shared `HARDWARE_CATALOG` in `_deployment.jsx`). A GPU **not** in
|
||||
this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.hardware`
|
||||
(`{id,label,vram,vendor}`) — the engine merges it in; don't edit the engine catalog:
|
||||
|
||||
| Platform | Vendor | VRAM | Docker image |
|
||||
|---|---|---|---|
|
||||
| H100 | NVIDIA | 80GB | `lmsysorg/sglang:<ver>` |
|
||||
| H200 | NVIDIA | 141GB | `lmsysorg/sglang:<ver>` |
|
||||
| B200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` |
|
||||
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
|
||||
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:<ver>` — needs a **CUDA 13** build |
|
||||
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
| MI355X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
|
||||
- **Image tag (`<ver>`)**: don't guess — ask the user for the tag the recipes ran on, or
|
||||
default to `dev` (nightly). The same tag goes in `dockerImages` and benchmarks'
|
||||
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
|
||||
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
|
||||
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
|
||||
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300
|
||||
single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so
|
||||
its only multi-GPU topology is TP=2 across 2 nodes.
|
||||
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
|
||||
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
|
||||
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
|
||||
- **EP** (MoE): 8-GPU NVIDIA `--tp 8 --ep 8`; AMD `EP = TP`; small NVIDIA (TP≤4) omit
|
||||
`--ep` unless benchmarked. (The template's AMD example cell shows these.)
|
||||
|
||||
## Phase 2 — Instantiate the template
|
||||
|
||||
For a diffusion model, instantiate `templates/diffusion-page.mdx.tmpl` and keep the
|
||||
shared `DiffusionModelTags` component, plus `templates/diffusion-config.jsx.tmpl` for the
|
||||
opt-in scoped command builder. Put the compact install command and builder in §1 Quick
|
||||
start. The first two paragraphs in §2 Model capabilities are not generic filler: they must
|
||||
state the model's capability range, strongest differentiator, when to choose it, and at
|
||||
least one real deployment or capability boundary. Put orthogonal runtime features in
|
||||
`scope: "serve"` or `scope: "request"`, not in the base recipe; use the schema from
|
||||
`references/diffusion-authoring.md`.
|
||||
|
||||
1. **Copy** the three template files to their target paths (above). Note the two
|
||||
vendor-folder conventions: under `configs/` the folder is the **HuggingFace org**
|
||||
(`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`).
|
||||
2. **Replace every `__TOKEN__`**: `__MODEL_DISPLAY__`, `__MODEL_SLUG__`, `__HF_ORG__`,
|
||||
`__HF_REPO__`, `__REASONING_PARSER__`, `__TOOLCALL_PARSER__`, `__ONE_LINER__`. Verify
|
||||
none remain: `grep -rn '__[A-Z_]*__' <new files>`.
|
||||
3. **Prune** to what the model supports (delete, don't stub) — using
|
||||
[references/authoring-reference.md](references/authoring-reference.md):
|
||||
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
|
||||
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
|
||||
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
|
||||
- `playgroundFeatures` axes: remove the `megamoe` backend option + the
|
||||
`megamoeQuant` block from the `moe` axis (non-Blackwell-MoE), delete `hisparse`
|
||||
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
|
||||
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
|
||||
to single `default` if there's no variant axis (then drop the `variant` half of
|
||||
`modelNames`/`defaultAccuracy` keys).
|
||||
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||
a guessed release; key by `hw`, or a more specific key — `hw|variant|quant`, `variant|quant`,
|
||||
`hw|quant|strategy`, `hw|quant`, resolved in that order before `hw` — when one
|
||||
variant/quant/strategy needs its own image), `multiNodeHints` only for fabric-specific hw
|
||||
(e.g. gb200).
|
||||
|
||||
5. **Diffusion pages: add the ComfyUI section.** Every diffusion cookbook page ends with
|
||||
a `## <n>. Run in ComfyUI` section so a reader never has to guess whether the model is
|
||||
reachable from ComfyUI. It is one component; the per-model facts live in the component,
|
||||
not the page:
|
||||
|
||||
```mdx
|
||||
## <n>. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="<key>" />
|
||||
```
|
||||
|
||||
Pick `model` from the table in
|
||||
`docs/src/snippets/diffusion/comfyui-support.jsx`. Use the model's own key when it has
|
||||
an entry (its executor or dedicated node differs); otherwise use the generic `image` or
|
||||
`video`. A model gets its own key only when the plugin actually treats it specially —
|
||||
an entry in `executor_class_dict`
|
||||
(`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a
|
||||
dedicated node. Adding a key without the matching plugin support makes the page lie.
|
||||
|
||||
### Site-wiring (do all three)
|
||||
|
||||
- **`docs/docs.json`** — add the page under Cookbook → `<category>` → `<Vendor>`, at
|
||||
the **top** of that vendor's `pages` (root-relative, no `.mdx`:
|
||||
`cookbook/<category>/<Vendor>/<Model>`). New vendor group → insert in the section's
|
||||
local ordering.
|
||||
- **NEW-tag hygiene** — the new page keeps `tag: NEW` (from the template). Scan the
|
||||
vendor dir for existing NEW and strip it from siblings; verify ≤1:
|
||||
`grep -rn 'tag: NEW' docs/cookbook/<category>/<Vendor>/` → at most one result. (Scan
|
||||
files; don't assume the first `docs.json` entry holds NEW.)
|
||||
- **Homepage card** — `docs/cookbook/<category>/intro.mdx`: if the org already has a
|
||||
`<Card>`, update only its `href` (keep `img`). If the org is **new**, add a `<Card>`
|
||||
(title = nav-group name; keep card order aligned with `docs.json`) **and create its logo**:
|
||||
ask the user for the brand logo, then generate the conforming **icon-only 940×525 RGBA
|
||||
transparent** PNG → `docs/cards/logos/<org-slug>.png` per
|
||||
[references/vendor-logo.md](references/vendor-logo.md) (track with `git add -f` — `*.png`
|
||||
is gitignored repo-wide). Never invent or copy a logo.
|
||||
|
||||
## Phase 3 — Validate
|
||||
|
||||
```bash
|
||||
cd docs
|
||||
mint validate # frontmatter, missing nav entries, MDX/JSX errors
|
||||
mint broken-links
|
||||
mint dev # visual smoke test at http://localhost:3000/cookbook/<category>/<Vendor>/<Model>
|
||||
```
|
||||
|
||||
Spot-check: cells render sensible commands; URL-hash nav persists across reload; the
|
||||
Playground inherits the Deploy selection live; each axis toggle produces the expected
|
||||
diff; Docker mode wraps in `docker run` with the right image; multi-node cells emit the
|
||||
hints + `--nnodes N`; cURL resolves the model name; the NEW badge shows on the new page
|
||||
and is gone from same-vendor siblings; the homepage card points to the new model.
|
||||
|
||||
## Phase 4 — Interactive testing
|
||||
|
||||
The user deploys each cell, runs the benches, and pastes results; you fill the data:
|
||||
- mark each tested `cells[]` entry `verified: true` (absent = yellow/unverified badge);
|
||||
- fill the `<model>-benchmarks.jsx` entries (one per cell `match`) with measured
|
||||
speed/accuracy + the `sglang_version` the user reports (don't invent one — the template's
|
||||
`0.0.0` is a deliberate TODO); set model-level `defaultAccuracy` per variant. Leave a
|
||||
cell's entry as a bare `match` stub if it has no numbers yet (the card shows "pending").
|
||||
|
||||
## Phase 5 — Prose & config tips
|
||||
|
||||
**Read [references/mintlify-authoring.md](references/mintlify-authoring.md) first** (it
|
||||
carries the parser-output-shape / thinking-mode / Output-Example / no-hardcoded-sampling
|
||||
rules + the Mintlify forbidden-syntax list). Then rewrite the MDX prose from the HF card +
|
||||
user notes: §1 Model Introduction (description, links, params, license, variants table),
|
||||
§2 Configuration Tips (hw-specific tuning, caveats), §3 Advanced Usage (Reasoning /
|
||||
Tool-Calling / HiCache — keep only what applies; match the reasoning example to the
|
||||
parser's output shape; each runnable block gets an `**Output Example:**`).
|
||||
|
||||
For diffusion pages, follow the category-specific Quick start and capability contract in
|
||||
`references/diffusion-authoring.md`. Run `node docs/scripts/check_cookbook_configs.mjs` to
|
||||
verify the tag widget and introduction structure before rendering the page.
|
||||
|
||||
## Phase 6 — Review
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Git workflow
|
||||
|
||||
Always branch — never commit to main directly.
|
||||
|
||||
```bash
|
||||
git checkout -b add-<model>-cookbook
|
||||
git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs/src/snippets/configs/<hf-org>/<slug>-benchmarks.jsx \
|
||||
docs/cookbook/<category>/<Vendor>/<Model>.mdx \
|
||||
docs/docs.json docs/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
+3
-3
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||
then parallelism (`--tp`, `--attn-dp-size`), then MoE
|
||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
knobs/values that are disabled under the live facts, so stale picks never
|
||||
emit a blocked combination. Interleave prefill-CP + DP-Attention is
|
||||
deliberately NOT grayed (combined support is planned upstream even though
|
||||
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
|
||||
current releases assert `dp_size == 1` for interleave) — the engine shows
|
||||
a warning hint below the command box instead.
|
||||
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
|
||||
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
|
||||
+5
-3
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
|
||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
@@ -357,7 +357,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--attn-dp-size 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
@@ -402,7 +403,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 16",
|
||||
"--attn-dp-size 16",
|
||||
"--dp 16",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
+2
-2
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
|
||||
|
||||
## 1. Quick start
|
||||
|
||||
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
|
||||
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
|
||||
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
|
||||
a verified recipe below.
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
---
|
||||
name: cookbook-migrate-model
|
||||
description: Migrate a legacy-template SGLang cookbook page (monolithic per-model generator under docs/src/snippets/autoregressive/) onto the config-driven template (shared _deployment.jsx / _playground.jsx engines + per-model config). Use when asked to migrate, convert, or port an existing cookbook page — NOT for brand-new models (use cookbook-add-model for those). Run with /cookbook-migrate-model <Model page name, e.g. GLM-5.1>.
|
||||
---
|
||||
|
||||
# Cookbook Migrate Model
|
||||
|
||||
Convert one legacy cookbook page to the config-driven format, faithfully. The
|
||||
legacy page — its generator widget and its measured benchmark blocks — is the
|
||||
**single source of truth**. You are transcribing it into the new data model,
|
||||
not improving it.
|
||||
|
||||
Reuses the `cookbook-add-model` skill's assets (read them on demand):
|
||||
- `../cookbook-add-model/templates/config.jsx.tmpl`, `page.mdx.tmpl`, `benchmarks.jsx.tmpl`
|
||||
- `../cookbook-add-model/references/authoring-reference.md` (config/cells/playground contract)
|
||||
- `../cookbook-add-model/references/mintlify-authoring.md` (MDX rules)
|
||||
|
||||
Migration-specific references in this skill:
|
||||
- [references/dimension-mapping.md](references/dimension-mapping.md) — legacy-control → new-dimension mapping rules, command rewrite table, per-family strategy sets, and the Qwen3.5 pilot as a worked example (PR #27848).
|
||||
|
||||
The round's per-model inventory (scope, batch order, quirks, measured-data
|
||||
survey) is tracked by the migration maintainer outside the repo — expect it in
|
||||
your dispatch prompt, or ask for it.
|
||||
|
||||
## Hard rules (non-negotiable)
|
||||
|
||||
1. **Never modernize.** Env vars, flags, TP values, docker tags, version strings
|
||||
are copied verbatim from the legacy page — even when today's defaults differ
|
||||
(e.g. a flag whose behavior has since become the default; keep it anyway). The recipe
|
||||
that was verified is the recipe as written. Allowed normalizations are ONLY
|
||||
the five alias rewrites in dimension-mapping.md §2 (`launch_server`→`sglang
|
||||
serve`, `--model`→`--model-path`, `--tp-size`→`--tp`, abbreviated
|
||||
`--speculative-algo`→`--speculative-algorithm`,
|
||||
`--expert-parallel-size`→`--ep`). **Accuracy-degrading flags**
|
||||
(`--kv-cache-dtype fp8_e4m3`, W4A4-style runtime quant) follow a
|
||||
deterministic rule — enforced in migration, no asking: offered as a
|
||||
legacy **selectable option** → never select it (cells mirror the
|
||||
accuracy-safe side), and the option **survives as a Playground axis** —
|
||||
an existing one where it fits, else add one via a separate prior engine
|
||||
PR (rule 4); a legacy choice never degrades to a tips mention. Baked
|
||||
into the recipe's **default/unconditional command** → keep it verbatim
|
||||
(the recipe was measured with it, and fp8 KV halves KV memory —
|
||||
stripping could OOM it). See dimension-mapping.md §2 caveats.
|
||||
2. **Never invent versions or numbers.** Benchmark numbers only from the
|
||||
legacy page's measured blocks, and a result migrates ONLY when its
|
||||
`sglang_version` is a **reproducible anchor** — the bar is reproducibility,
|
||||
not "must be a release":
|
||||
- ✅ release tag/version (`v0.5.9` / `0.5.9`), commit hash, OR — for
|
||||
**Day-0 support** (the enabling PR isn't merged and no release is cut
|
||||
yet) — a specific **PR (`PR #27944`) or commit** you can `gh pr checkout`
|
||||
/ `git checkout <sha>`. Commit is most precise; a PR pin is fine for day-0.
|
||||
- ❌ a moving ref — `"main branch"`, `"main (2026-06-11)"`, open-ended
|
||||
`"0.5.8+"` — is NOT reproducible: **drop the WHOLE result (speed AND
|
||||
accuracy)**, not just speed. Keep `benchmarkCommands` so ⚡Reproduce still
|
||||
guides re-measurement against a pinned build.
|
||||
**Never inherit cross-model numbers** — measurements the legacy page
|
||||
attributes to a *different model* (e.g. a K2.6 page carrying K2.5-measured
|
||||
speed) are dropped regardless of version. When kept, `sglang_version` is the
|
||||
legacy page's string verbatim. Docker tags only the ones the legacy page
|
||||
pinned (unmapped hw falls back to `:dev`).
|
||||
3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the
|
||||
legacy page has concrete measured data for that exact 5-dim combo AND
|
||||
(b) the cell's flags equal the deployment command used for that measurement
|
||||
— modulo `{{HOST_IP}}`/`{{PORT}}`, the five alias rewrites, and **parser
|
||||
flags**: `--reasoning-parser`/`--tool-call-parser` are stripped from every
|
||||
cell (Playground-only feature; when the measured run had them on, say so in
|
||||
the benchmarks file header). When the measured command diverges from the
|
||||
generator default, **the verified cell follows the measured command**; the
|
||||
generator default stays as the sibling strategy/cell or a tips note. Everything else is unverified (yellow) —
|
||||
including combos that look memory-infeasible; keep them verbatim and list
|
||||
them in the PR body for the re-verification track.
|
||||
4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not
|
||||
change in a migration PR. Model-specific features are config DATA consumed
|
||||
by generic axis handlers (MegaMoE precedent), so they need NO engine change.
|
||||
A **titled single-select that strips a flag family** — KV Cache DType
|
||||
(`--kv-cache-dtype`), mamba (`--mamba-radix-cache-strategy`), … — is already
|
||||
covered by the merged generic **`flagSelects`** axis: declare it in the
|
||||
config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5
|
||||
mamba example), **no engine PR**. Only a genuinely new control *shape* that
|
||||
`flagSelects` can't express would need a one-time generic primitive (never a
|
||||
model-named handler) on a separate prior PR (engine-axis.md).
|
||||
5. **`github.cookbookModel` must be set** (`<hf-org>/<page-slug>`, e.g.
|
||||
`qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels
|
||||
as deepseek-v4. The issue template itself needs NO edits (free-form input).
|
||||
6. **Strategy tiers are signal-driven.** A cell goes under `low-latency` /
|
||||
`high-throughput` ONLY on a signal present in the legacy source (an
|
||||
explicit performance toggle, a named recipe, or prose stating the
|
||||
operating point); **no signal → `balanced`**. Never derive a slant from
|
||||
your own hardware intuition — re-tiering on measured evidence is the
|
||||
hardware owner's follow-up PR, not part of a migration
|
||||
(dimension-mapping.md §4).
|
||||
|
||||
## Workflow (one model = one PR)
|
||||
|
||||
### 1. Inventory the legacy assets
|
||||
- Read the legacy generator (`docs/src/snippets/autoregressive/<slug>-deployment.jsx`)
|
||||
end-to-end: every option dimension (radio vs checkbox vs dynamic), every
|
||||
gate/SUPPORT matrix, the full emitted command per reachable combo (env
|
||||
prefixes, `# Error` pseudo-commands included).
|
||||
- Read the legacy MDX: §2 install docker tags → `dockerImages` (pinned, not
|
||||
upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
|
||||
Output Examples verbatim); §5 benchmark blocks → transcribe each measured
|
||||
block: deploy command used, bench command (dataset/isl/osl/num-prompts/
|
||||
concurrency), P50 (median) TTFT/TPOT, output tok/s, hardware, version string.
|
||||
- Inbound-anchor sweep: `grep -rn "<PageName>" docs/ --include='*.mdx'` —
|
||||
find links/`#fragments` into this page (`mint broken-links` does NOT check
|
||||
fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
|
||||
- Check the maintainer-provided inventory notes for this model's known quirks —
|
||||
but treat them (and the §4 family table) as a **survey snapshot**: re-verify
|
||||
every dimension against the live legacy files before mapping. Pages keep
|
||||
receiving updates (precedent: Kimi-K2.6's live generator has a speculative
|
||||
toggle the 2026-06-10 survey notes lack).
|
||||
|
||||
### 2. Design the 5-dim mapping
|
||||
Apply [references/dimension-mapping.md](references/dimension-mapping.md). Key
|
||||
decision — which legacy toggle becomes the `strategies` dimension: a toggle
|
||||
that **changes other parts of the command** (TP, mem) must (the Playground
|
||||
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
|
||||
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||
Playground axis with the flags baked into cells when the legacy default was
|
||||
ON — EXCEPT parsers:
|
||||
`--reasoning-parser`/`--tool-call-parser` are NEVER baked into cells, they are
|
||||
Playground-only (DSv4 convention). **Every legacy control survives as an
|
||||
interactive control** — a dimension or a Playground axis, never a tips-only
|
||||
mention — and a model-specific control is **config data, not engine code**
|
||||
(MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure
|
||||
config whenever it fits an existing axis's data schema. A **titled
|
||||
single-select that strips a flag family** (Nemotron3's "KV Cache DType",
|
||||
mamba `--mamba-radix-cache-strategy`, …) fits the merged generic **`flagSelects`**
|
||||
axis — so it too is config-only (declare a `flagSelects` list). Only a control
|
||||
whose *shape* `flagSelects` still can't express would need a ONE-TIME generic
|
||||
primitive (never a model-named handler) on a separate PRIOR engine PR, keeping
|
||||
the migration PR data-only (hard rule 4, engine-axis.md). The strategy count follows the page's
|
||||
operating points: **one recipe → a single `balanced`; two → `low-latency` +
|
||||
`high-throughput`; three → the full trio (the ideal)**. The tiers apply per
|
||||
(hw × variant × quant) combination — a single-recipe combination on a
|
||||
multi-strategy page parks under its semantically honest tier (no
|
||||
latency/throughput slant → `balanced`; the page's list is the union). When the
|
||||
legacy toggle is MTP / speculative decoding, the direction is a deterministic
|
||||
default — apply without asking: **MTP on → `low-latency`, MTP off →
|
||||
`high-throughput`** (reversed only with maintainer confirmation). Tier
|
||||
placement is signal-driven (hard rule 6). Never invent a recipe just to fill
|
||||
strategy chips (see dimension-mapping.md §4). Record the outcome as a
|
||||
**strategy mapping table** for the PR body — one row per group of
|
||||
combinations sharing the same legacy signal (e.g. "all GPU combos: MTP
|
||||
toggle → low-latency / high-throughput"; "xeon: (none) → balanced"), with a
|
||||
one-line rationale each; don't enumerate 60 identical rows. The table is what
|
||||
hardware owners sign off on at review.
|
||||
|
||||
### 3. Generate the config (codegen, then audit)
|
||||
- For >~30 cells, port the legacy `generateCommand()` into a throwaway Node
|
||||
script that enumerates combos and emits the `cells:[...]` literal (output
|
||||
must stay a pure literal — Mintlify forbids runtime spreads/calls). Apply the
|
||||
verified-cell override in the script. See the pilot scripts embedded in the
|
||||
worked example of dimension-mapping.md §5.
|
||||
- **Independent equivalence audit (required):** extract the ORIGINAL generator
|
||||
from git (`git show main:<path>` — NOT `HEAD:`, the migration branch deletes
|
||||
the file, see dimension-mapping.md §5 item 7), stub React hooks, run it for
|
||||
every combo, and diff token-by-token against the new cells. Expected deltas
|
||||
only: the appended `--host {{HOST_IP}}`/`--port {{PORT}}`, the
|
||||
engine-injected multi-node trio, the §2 alias rewrites (the entrypoint
|
||||
rewrite doesn't appear in cell tokens — cells hold flags only; the audit
|
||||
script normalizes it on the legacy side), and the intentional verified-cell
|
||||
override. Paste the PASS count + the audit script in the PR body
|
||||
(collapsed `<details>`).
|
||||
- Hand-author the non-cells fields per authoring-reference.md. Structural
|
||||
self-checks: every cell resolves a `modelNames` key; no `--nnodes/--node-rank/
|
||||
--dist-init-addr/--host/--port` literals; every `{{KEY}}` declared; every
|
||||
`supportedHardware` id has ≥1 cell.
|
||||
|
||||
### 4. Benchmarks file
|
||||
One entry per measured block only (cells without entries already render
|
||||
"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
|
||||
total (in+out) tok/s/GPU = `output tok/s ÷ (tp × nnodes) ×
|
||||
(isl+osl)/osl` — stored directly (the card shows it as-is). TTFT/TPOT
|
||||
take the P50 (median) rows; set `config.latencyPercentile` (default `"P50"`; use
|
||||
`"Mean"` only for legacy Mean-recorded data — temporary, being migrated to P50; an
|
||||
entry-level `latencyPercentile` overrides the page value per cell).
|
||||
Put the workload's
|
||||
`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
|
||||
the benchmarks carry accuracy data** — the engine ships no default eval set
|
||||
(#27842), so missing labels means the accuracy rows silently don't render;
|
||||
extra context (sample counts, suites that don't fit) goes in the entry's
|
||||
`notes`. Zero-measured-data pages: skip the file and the `benchmarks` prop
|
||||
entirely, but keep `benchmarkCommands` so ⚡Reproduce still guides users.
|
||||
|
||||
### 5. Rewrite the MDX
|
||||
From `page.mdx.tmpl`: keep the original `title` (nav identity), write a fresh
|
||||
SEO `description` (top-level — delete any legacy `metatags.description`), **no
|
||||
`tag: NEW`** (a migration is not a launch), **no `mode:`**. Install accordion
|
||||
carries the legacy install content + pinned images. Keep the template's
|
||||
DSv4-style strategy bullets — serving semantics first (single-user chat /
|
||||
typical multi-user / batch throughput), trimmed to the strategies the page
|
||||
ships, plus at most a one-line note on what each strategy changes on this
|
||||
model; do NOT rewrite them as toggle-/migration-centric explanations.
|
||||
Legacy §5 benchmark prose is deleted (numbers → benchmark card, commands →
|
||||
⚡Reproduce); legacy prose deploy commands are deleted (doc↔config parity —
|
||||
fold their unique flags into §2 tips). Invocation examples + real outputs carry
|
||||
over verbatim, but **wrapped in Accordions** — §3 commands and outputs are
|
||||
collapsible (required, DeepSeek-V4 pattern): code in an
|
||||
`<Accordion title="… (Python)">`, output in a following
|
||||
`<Accordion title="Example Output">`; legacy pages kept them inline.
|
||||
|
||||
### 6. Delete the legacy generator
|
||||
Remove `docs/src/snippets/autoregressive/<slug>-deployment.jsx` and its
|
||||
import. `grep -rn "<slug>-deployment" docs/` must return nothing (config
|
||||
provenance comments must not name the deleted path). Site wiring needs **no
|
||||
changes**: docs.json path/title unchanged, vendor card + logo already exist.
|
||||
|
||||
### 7. Validate
|
||||
- `grep -rn '__[A-Z_]*__'` on the new files (no template tokens).
|
||||
- `cd docs && mint validate && mint broken-links` (pre-existing breaks on
|
||||
main are not yours — say so in the PR).
|
||||
- `mint dev` browser smoke: initial selection = the verified cell (first in
|
||||
`cells[]`) with green badge; multi-node cells show the injected trio +
|
||||
header; AMD cells show env prefixes; Docker mode wraps with the pinned image
|
||||
and passes cell env as `--env`; condition-hidden combos grey out; benchmark
|
||||
card values; NO parser flags in any Deploy command; Playground parser
|
||||
toggles ADD the parser flags (green additions) while spec toggles strike
|
||||
the baked spec flags (red); Submit ↗ prefills this model. **Probe pitfall:** drive at most
|
||||
ONE programmatic click per evaluation and wait for React to settle —
|
||||
multiple clicks in one synchronous script batch and read stale DOM.
|
||||
- Token-level audit from step 3 passes.
|
||||
|
||||
### 8. PR + review
|
||||
One PR per model. PR body: migration framing, verified policy applied, the
|
||||
strategy mapping table (step 2), the audit PASS count + script, any
|
||||
inherited-infeasible combos flagged for re-verification. Then run `/cookbook-review-pr <N>` and fix findings.
|
||||
FYI: docs previews only build for in-repo (`sgl-project/sglang`) branches —
|
||||
a fork-headed PR is perfectly fine but renders no preview; a maintainer can
|
||||
re-push the branch in-repo if a preview is wanted for review.
|
||||
|
||||
### 9. Keep this skill current
|
||||
Any new convention, engine behavior, or pitfall you discover while migrating
|
||||
(naming decisions, audit-script gotchas, review-rule conflicts, …) MUST be fed
|
||||
back into this skill — same PR if it's skill-file-only, or an immediate
|
||||
follow-up commit on the skill's branch/PR. The next agent runs on what's
|
||||
written here, not on your session's context.
|
||||
+2
-2
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
|
||||
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
||||
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
|
||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
|
||||
Caveats discovered in the pilot:
|
||||
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
name: cookbook-review-pr
|
||||
description: Review a pull request against the SGLang Cookbook (docs/, Mintlify) contribution checklist — the config-driven format (per-model config + benchmarks JSX consumed by the shared _deployment.jsx / _playground.jsx engines). Run with /cookbook-review-pr <PR number>.
|
||||
---
|
||||
|
||||
# Cookbook Review PR
|
||||
|
||||
Fetch the diff, run the checklist, report what you find. The cookbook is **config-driven**:
|
||||
shared engines (`_deployment.jsx`, `_playground.jsx`) with NO model-specific code; each
|
||||
model is a data `config` (+ optional `benchmarks`) under `src/snippets/configs/<vendor>/`
|
||||
plus an MDX page. This checklist targets that layout. Field-schema detail lives in
|
||||
`.claude/skills/cookbook-add-model/references/authoring-reference.md` — defer to it rather
|
||||
than restating.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. `gh pr view <N> --repo sgl-project/sglang --json title,body,files,author,baseRefName,headRefName,commits,reviews`
|
||||
2. `gh pr diff <N> --repo sgl-project/sglang`
|
||||
3. `gh pr list --repo sgl-project/sglang --state open --search "<model name>"` (duplicate check)
|
||||
4. Run every checklist item against the diff.
|
||||
5. Output per-file verdicts + overall recommendation.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. File hygiene
|
||||
- A cookbook PR should only touch: `docs/src/snippets/configs/<vendor>/*.jsx`
|
||||
(config + benchmarks), `docs/cookbook/**/*.mdx`, `docs/docs.json`,
|
||||
`docs/cookbook/<category>/intro.mdx` (vendor card), `docs/cards/logos/<vendor>.png`
|
||||
(new vendor only). Flag stray files (`settings.local.json`, lockfiles, IDE configs).
|
||||
- Pages must be `.mdx`, not `.md`. Files end with a trailing newline. Check commit history
|
||||
for unrelated commits accidentally included.
|
||||
- **Engines untouched**: `_deployment.jsx` / `_playground.jsx` should NOT change in a
|
||||
model-add PR (adding a model is data-only). Engine edits = a separate axis/feature PR
|
||||
(see `cookbook-add-model/references/engine-axis.md`); review them against that checklist.
|
||||
|
||||
### 2. Config quality (the per-model `config`)
|
||||
- Single `export const config = { ... }` literal — **no** function calls, spreads,
|
||||
fragment refs, or IIFE (Mintlify re-evals at hydration → `ReferenceError`).
|
||||
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
|
||||
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
|
||||
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog — unless the platform is
|
||||
genuinely shared, which is an engine change the PR body has to argue (and the
|
||||
`cookbook-add-model` hardware table moves with it). A new catalog entry is inert for
|
||||
other pages: each filters the catalog by its own `supportedHardware`.
|
||||
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||
- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
|
||||
2 → `low-latency` + `high-throughput`; 3 → the full trio. Tiers apply per
|
||||
(hw × variant × quant) combination: a single-recipe combination must park under its
|
||||
semantically honest tier (clear slant → that tier, e.g. a workstation card under
|
||||
`low-latency`; no slant → `balanced`, e.g. a CPU platform) — **flag a no-slant recipe
|
||||
parked under low-latency/high-throughput**. Mixed unions
|
||||
like [low-latency, balanced, high-throughput] with per-selection greying are fine. Also
|
||||
flag model-specific ids (e.g. `mtp`), and flag an INVERTED speculative mapping — the
|
||||
deterministic default is MTP/spec-decoding ON → `low-latency`, OFF → `high-throughput`
|
||||
(at saturation the draft+verify overhead outweighs the speedup); the reverse needs an
|
||||
explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics
|
||||
in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal
|
||||
toggles.
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback); a
|
||||
`hw|quant` key (resolved before the plain `hw`) is valid when one quant on a shared GPU needs
|
||||
a different image (e.g. an FP4 dev build) — don't flag those.
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
Hints render above BOTH run modes, so flag one that reads as docker-only ("add these
|
||||
docker run flags") — `docker run` flags belong in the hardware entry's
|
||||
`multiNodeDockerFlags`, which the engine emits into the Docker command itself.
|
||||
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
|
||||
template's `model` field is a free-form input prefilled from this value; if the config
|
||||
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
|
||||
page's submissions get mislabeled.
|
||||
- `playgroundFeatures` is opt-OUT: the **general axes ship on every cookbook by default**
|
||||
(`attention` TP/CP/DP-Attn, `moe` backend+EP for MoE models, `parsers`, `speculative`,
|
||||
`pdDisagg`, `hicache`) — flag a missing general axis unless the model genuinely cannot
|
||||
use it. Model-specific axes only where applicable (MegaMoE backend + `megamoeQuant`
|
||||
only on Blackwell MoE, gated by `requiresHw`; `hisparse` only DSA-style). Knobs that are
|
||||
meaningless for a subset of variants/hw are `disable`d with a reason, not silently live
|
||||
(e.g. MoE knobs greyed on dense variants). No empty/stub axes.
|
||||
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
|
||||
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
|
||||
nothing).
|
||||
- **All-hardware considered**: every `supportedHardware` id (from the catalog or `config.hardware`) has ≥1 cell OR is a deliberate
|
||||
greyed "coming soon"; AMD was pruned or kept on purpose (not a leftover template family).
|
||||
|
||||
### 3. Cells / 5-dim matrix
|
||||
- Every cell `match` has EXACTLY the 5 keys (`hw`, `variant`, `quant`, `strategy`, `nodes`).
|
||||
- `env` / `flags` are flat literals (only `{{PLACEHOLDER}}` subst) — no shared
|
||||
`commonFlags` reference (Mintlify won't inline it).
|
||||
- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
|
||||
(the renderer injects them from `match.nodes`).
|
||||
- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
|
||||
- NO `--reasoning-parser` / `--tool-call-parser` in any cell — parsers are a
|
||||
Playground-only feature added on top of the base command (DSv4 convention);
|
||||
flag any cell that bakes them in.
|
||||
- Accuracy-degrading flags in cells — runtime quant below the checkpoint
|
||||
(e.g. MegaMoE W4A4 — DSv4 gates it behind the Playground's `megamoeQuant`)
|
||||
and lossy `--kv-cache-dtype` (e.g. `fp8_e4m3` over a higher-precision-KV
|
||||
checkpoint): **flag for explicit maintainer confirmation**. Output quality
|
||||
should be exactly what the quant chip declares, so absent a recorded
|
||||
sign-off in the PR (e.g. carried verbatim from a measured legacy recipe's
|
||||
default command), request the flag move to Playground/tips.
|
||||
- Flag order: `--model-path` first (an optional `--trust-remote-code` may precede it —
|
||||
the DSv4 cells do), then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||
last (the playground's insert anchors assume this).
|
||||
- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
|
||||
(BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
|
||||
|
||||
### 4. Benchmarks
|
||||
- Each `benchmarks[]` entry's `match` tuple corresponds to a real cell.
|
||||
- `accuracyLabels` is present whenever the benchmarks carry accuracy data — the engine
|
||||
ships NO default eval set; without it the accuracy rows silently don't render.
|
||||
`defaultAccuracy` / per-cell `accuracy` / `benchmarkCommands.accuracy` keys all
|
||||
∈ `config.accuracyLabels`.
|
||||
- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
|
||||
that only released FP8/FP4 is a factual bug.
|
||||
- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
|
||||
from the `sglang serve` deploy command, and should carry `--flush-cache`: bench_serving's
|
||||
`random` prompts are deterministic, so a warm rerun hits the radix cache and inflates
|
||||
throughput — speed numbers are measured cache-cold.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
- **Latency percentile**: `config.latencyPercentile` (default `"P50"`, or `"Mean"`) matches the
|
||||
percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. A benchmarks
|
||||
entry may carry its own `latencyPercentile` to override the page value per cell
|
||||
(entry → config → `"P50"`): on a P50 page, kept legacy Mean cells must set it — a
|
||||
`sglang_version` tag alone doesn't convey the percentile. (`"Mean"` is temporary — legacy
|
||||
data is being re-measured to P50.)
|
||||
- **Throughput convention**: `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU**
|
||||
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
|
||||
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||
column must be produced by the SAME harness — flag a page that, say, measures one
|
||||
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
|
||||
`run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
|
||||
(the scores aren't comparable). Either standardize on one harness (matching
|
||||
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
|
||||
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
|
||||
|
||||
### 5. Doc ↔ config parity (the #1 finding)
|
||||
- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
|
||||
equal what the engine emits from the corresponding cell — same flags, same order. Drift
|
||||
here is the most common review miss.
|
||||
|
||||
### 5b. ComfyUI section (diffusion pages)
|
||||
- A diffusion page ends with `## <n>. Run in ComfyUI` rendering `<ComfyUISupport />`. A
|
||||
reader must not have to guess whether the model is reachable from ComfyUI.
|
||||
- The `model` prop is a key that exists in `docs/src/snippets/diffusion/comfyui-support.jsx`.
|
||||
A model-specific key is only correct when the plugin really treats it specially — an entry
|
||||
in `executor_class_dict`
|
||||
(`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a dedicated
|
||||
node. Otherwise the generic `image` / `video` key is the honest one; a model-specific key
|
||||
without matching plugin support makes the page claim support that does not exist.
|
||||
- Prose describing ComfyUI support inline instead of using the component is a finding: the
|
||||
facts drift from the plugin.
|
||||
|
||||
### 6. Commands / port
|
||||
- Launch uses `sglang serve` — flag any `python -m sglang.launch_server` /
|
||||
`python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`;
|
||||
guard against prose/cells reintroducing the old launcher.
|
||||
- Port `30000` everywhere (launch, curl, client `base_url`, bench) — flag `8000`.
|
||||
Launch port must match client/curl port on the same page.
|
||||
|
||||
### 7. Frontmatter
|
||||
- Every new MDX page has `title:` and a **top-level** `description:` (a real one-line value
|
||||
prop, not copied from another vendor) — NOT `metatags.description` (non-canonical; the
|
||||
top-level field is what renders as the subtitle and SEO meta — see mintlify-authoring).
|
||||
- **No `mode: wide` on a model page** — it hides the right-hand "On this page" ToC that every
|
||||
other model page has. Leave `mode` unset (the Deploy/Playground panels self-cap at 900px, so
|
||||
the default column holds them fine). `mode: wide` belongs only on category `intro.mdx` grids.
|
||||
- `tag: NEW` only for genuine new launches; when one is added, stale `tag: NEW` on older
|
||||
pages should be dropped in the same PR (`grep -RlE "^tag: NEW" docs/cookbook/`).
|
||||
- MDX imports BOTH `Deployment` and `Playground` from `/src/snippets/...` (absolute).
|
||||
- Deploy heading slugs to `deployment` (or `deploy`), Playground to `playground` — so
|
||||
"↑ Switch base" and "Open the Playground →" scroll. No numbered headings for these two.
|
||||
|
||||
### 8. Navigation & homepage
|
||||
- New page → `docs/docs.json` updated: under the right vendor group inside
|
||||
`navigation` → Cookbook → Autoregressive Models, root-relative, **no `.mdx`**:
|
||||
`cookbook/<category>/<Vendor>/<Model>`.
|
||||
- Homepage `<Card href>` in `docs/cookbook/<category>/intro.mdx` points to the vendor's
|
||||
flagship; new vendors get a new `<Card>` + a logo at `docs/cards/logos/<vendor>.png` —
|
||||
**940×525 RGBA transparent, icon-only (no wordmark)**, lowercase filename, tracked via
|
||||
`git add -f` (`*.png` is gitignored repo-wide). Card order matches the `docs.json` nav order.
|
||||
- Don't change `docs/cookbook/intro.mdx` for individual model adds (top-level only).
|
||||
|
||||
### 9. Links & factual
|
||||
- HuggingFace URLs resolve to a real model. License section matches the actual HF license
|
||||
(don't copy from another model). Docker images from `lmsysorg/sglang`; no `sgl-project-dev`.
|
||||
The image **tag** is a real build (a release the author ran, or `:dev`/nightly) — not a
|
||||
guessed version.
|
||||
- Internal links root-relative, no extension (`/cookbook/.../<Model>`); flag `.md`/`.mdx`
|
||||
or `../`-relative links. `docs.sglang.io` is canonical.
|
||||
- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
|
||||
not `${VAR}` (a bash no-op).
|
||||
- **Parser ids must exist in the code registries** on the PR's target branch: every
|
||||
`--reasoning-parser X` / `--tool-call-parser Y` named in prose or in
|
||||
`playgroundFeatures.parsers` flags is a registered key in
|
||||
`python/sglang/srt/parser/reasoning_parser.py` (DetectorMap) /
|
||||
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
|
||||
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
|
||||
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
|
||||
(`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's
|
||||
chat template to the right parser — no rule match means auto silently disables the
|
||||
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
|
||||
|
||||
### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
|
||||
- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
|
||||
blocks (`> [!NOTE]`), markdown **pipe tables** (use JSX `<table>`), inline `<details>`,
|
||||
or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
|
||||
- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
|
||||
the opening fence); a fenced block nested inside another uses four backticks outside.
|
||||
- §3 commands and outputs are **collapsible** (DeepSeek-V4 pattern): every runnable
|
||||
example wrapped in an `<Accordion>`, its real output in a following
|
||||
`<Accordion title="Example Output">` (`Pending update...` only with user
|
||||
acknowledgement). Flag bare/inline example blocks and `**Output Example:**` headings.
|
||||
- Reasoning-parser example matches the parser's **output shape**: separate-field
|
||||
(`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
|
||||
- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
|
||||
`generation_config.json` defaults); listing them in §1 informationally is fine.
|
||||
|
||||
### 10. Quantization rules
|
||||
- **NVFP4** checkpoints are Blackwell-only (B200/B300/GB300) — never AMD. An AMD FP4 cell
|
||||
is legitimate ONLY when the vendor published an **MXFP4** checkpoint for it (e.g.
|
||||
`amd/Qwen3.5-397B-A17B-MXFP4` on MI355X) — verify the HF repo resolves; otherwise the
|
||||
AMD FP4 chip must be absent/`disabled`.
|
||||
- BF16 / FP8 work on NVIDIA and AMD. `--kv-cache-dtype fp8_e4m3` in a cell is an
|
||||
accuracy-degrading flag — see §3 (needs explicit maintainer sign-off; default
|
||||
home is Playground/tips).
|
||||
|
||||
### 11. Scope
|
||||
- Changes match the PR title. Flag global changes hiding behind a platform-specific title
|
||||
(e.g. an "H200 FP8" PR that adds a flag to ALL cells). Unmentioned side-fixes belong in
|
||||
the PR body.
|
||||
|
||||
### 12. Duplicate PRs
|
||||
- Another open PR for the same model? Flag it; compare completeness; note merge-conflict
|
||||
risk on `docs.json` + the vendor card; flag a superseded older PR by the same author.
|
||||
|
||||
### 13. Build / validate
|
||||
```bash
|
||||
cd docs
|
||||
mint validate
|
||||
mint broken-links
|
||||
```
|
||||
Optional: `mint dev` for a visual smoke test.
|
||||
|
||||
### 14. Reviewer feedback
|
||||
- `gh api repos/sgl-project/sglang/pulls/<N>/comments` — have prior reviewer requests been
|
||||
addressed? Unresolved requested-changes should be flagged.
|
||||
|
||||
### 15. Grammar & spelling
|
||||
- Check added/changed prose for typos and grammar (e.g. "recommend" vs "recommended").
|
||||
Flag each with the exact wrong text + correction.
|
||||
|
||||
## Output
|
||||
|
||||
Per file:
|
||||
- ✅ PASS
|
||||
- ⚠️ ISSUE: \<what\>
|
||||
- 🔴 BLOCK: \<what\>
|
||||
|
||||
Overall: **APPROVE** / **REQUEST CHANGES** / **BLOCKED**
|
||||
@@ -0,0 +1,203 @@
|
||||
---
|
||||
name: env-var-conventions
|
||||
description: Conventions for SGLang environment variables — where to define, how to access, how to name, and how to deprecate. Use when adding, renaming, or reviewing any `SGLANG_*` environment variable (or migrating a legacy `SGL_*` alias), or when touching `python/sglang/srt/environ.py`.
|
||||
---
|
||||
|
||||
# Environment Variables — Conventions
|
||||
|
||||
Apply this skill when adding, renaming, or reviewing any sglang-owned environment variable (`SGLANG_*`, or a legacy `SGL_*` alias being phased out), or when touching `python/sglang/srt/environ.py`.
|
||||
|
||||
## Rule 1 — Define in the `Envs` class in `python/sglang/srt/environ.py`
|
||||
|
||||
All sglang-owned env vars live as `EnvField` descriptors on the `Envs` class. Never add a new `os.getenv("SGLANG_...")`, `get_bool_env_var("SGLANG_...")`, or `get_int_env_var("SGLANG_...")` call site — the helpers in `python/sglang/srt/utils/common.py` carry an explicit `FIXME: move your environment variable to sglang.srt.environ` and exist only for pre-existing call sites.
|
||||
|
||||
Group the new entry under an existing section comment (e.g. `# Logging Options`, `# Scheduler: recv interval`, `# Flashinfer`). Add a new section comment only when none fits — never drop a new entry at the bottom of an unrelated block.
|
||||
|
||||
### Decision table: register in `Envs` or use `os.getenv`?
|
||||
|
||||
| Variable | Owner | Goes through `Envs`? |
|
||||
|---|---|---|
|
||||
| `SGLANG_*` | sglang | **Always.** The canonical prefix for all new entries. |
|
||||
| `MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` | Upstream/vendor alias that sglang wants to centralize | **Yes** — register in `Envs` so `.get()` / `.override()` work uniformly. Keep the upstream prefix. |
|
||||
| `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `HF_HUB_*` (raw upstream) | External tooling | **No.** Read with `os.getenv` — they're set by the launcher / driver, not by sglang. |
|
||||
| `RANK`, `LOCAL_RANK`, `WORLD_SIZE`, `MASTER_ADDR`, `MASTER_PORT`, `HOME`, `PATH` | Distributed launcher / OS | **No.** `os.getenv` only. |
|
||||
| Test runner internals (`PYTEST_CURRENT_TEST`, etc.) | Test framework | **No.** `os.getenv` only. |
|
||||
|
||||
`SGL_*` is **not** a parallel valid prefix — it's a deprecated legacy alias. `_convert_SGL_to_SGLANG` rewrites `SGL_*` to `SGLANG_*` at import time with a `DeprecationWarning`. Never define a new `SGL_*` descriptor or `os.getenv("SGL_...")` call site; if you see one in code, it's tech debt to migrate.
|
||||
|
||||
The rule of thumb: if the value's lifecycle is owned by sglang code (we read it, we may want to override it in tests, we may want to rename it), put it in `Envs`. If the value is set by something outside sglang and we only consume it as-is, use `os.getenv`.
|
||||
|
||||
## Rule 2 — Pick the typed descriptor
|
||||
|
||||
| Type | Use for |
|
||||
|---|---|
|
||||
| `EnvBool(default)` | boolean flag |
|
||||
| `EnvInt(default)` | integer |
|
||||
| `EnvFloat(default)` | float |
|
||||
| `EnvStr(default)` | string |
|
||||
| `EnvTuple(())` | comma-separated list, parsed via `s.split(",")` and stripped |
|
||||
|
||||
Default value:
|
||||
- For a knob whose "unset" state must be distinguishable from any concrete value, use `None` as the default (e.g. `EnvStr(None)`, `EnvInt(None)`). The descriptor handles set-to-None correctly via `_set_to_none`.
|
||||
- For a feature flag, the default encodes the production behavior. `ENABLE_FOO = EnvBool(False)` means foo is off in prod; `DISABLE_FOO = EnvBool(False)` means foo is on in prod. See Rule 4 on picking the verb.
|
||||
|
||||
### IntEnum for multi-state knobs
|
||||
|
||||
When a knob has more than two discrete states (e.g. an off / soft / strict ladder), define an `IntEnum` next to `Envs` and pass the enum member as the `EnvInt` default. Callers compare against the enum, not raw integers:
|
||||
|
||||
```python
|
||||
class ToolStrictLevel(IntEnum):
|
||||
OFF = 0
|
||||
FUNCTION = 1
|
||||
PARAMETER = 2
|
||||
|
||||
SGLANG_TOOL_STRICT_LEVEL = EnvInt(ToolStrictLevel.OFF)
|
||||
|
||||
# At the call site:
|
||||
if envs.SGLANG_TOOL_STRICT_LEVEL.get() >= ToolStrictLevel.PARAMETER:
|
||||
...
|
||||
```
|
||||
|
||||
Don't pile `SGLANG_ENABLE_FOO_STRICT` / `SGLANG_ENABLE_FOO_OFF` boolean knobs that fight each other — one ordered integer is cleaner.
|
||||
|
||||
## Rule 3 — Access via the `EnvField` API, never raw `os.environ`
|
||||
|
||||
```python
|
||||
from sglang.srt.environ import envs
|
||||
|
||||
if envs.SGLANG_FOO.get():
|
||||
...
|
||||
```
|
||||
|
||||
`envs.SGLANG_FOO` is the descriptor itself; its `__bool__` and `__len__` raise on purpose so that `if envs.SGLANG_FOO:` fails loudly instead of silently reading as truthy. The `.get()` is mandatory at every read site.
|
||||
|
||||
### Full API surface
|
||||
|
||||
| Method | Use |
|
||||
|---|---|
|
||||
| `.get()` | Read value (parsed). Returns `default` if unset, or `None` if explicitly set to None. |
|
||||
| `.set(value)` | Set value. `set(None)` flips the internal `_set_to_none` flag so the next `.get()` returns `None`, not `default`. |
|
||||
| `.clear()` | Unset entirely. Next `.get()` returns `default`. |
|
||||
| `.is_set()` | True iff the key is present in `os.environ` (regardless of value, including the explicit-None case). |
|
||||
| `.override(value)` | Context manager — set on enter, restore exactly what was there on exit. Use this in tests. |
|
||||
|
||||
The `_set_to_none` distinction matters: `clear()` → `is_set()=False, get()=default`; `set(None)` → `is_set()=True, get()=None`. A few descriptors (e.g. `SGLANG_TEST_MAX_RETRY = EnvInt(None)`, `SGLANG_DISAGGREGATION_THREAD_POOL_SIZE = EnvInt(None)`) rely on this to distinguish "user opted out" from "default applies".
|
||||
|
||||
### Test overrides
|
||||
|
||||
```python
|
||||
with envs.SGLANG_TEST_RETRACT.override(True):
|
||||
...
|
||||
```
|
||||
|
||||
Don't mutate `os.environ` directly in tests — `override` restores the original cleanly, including the explicit-None state, even if the block raises.
|
||||
|
||||
For multiple overrides composed dynamically, use `ExitStack`:
|
||||
|
||||
```python
|
||||
with ExitStack() as stack:
|
||||
for name, value in test_envs.items():
|
||||
stack.enter_context(getattr(envs, name).override(value))
|
||||
run_test()
|
||||
```
|
||||
|
||||
### Subprocess inheritance
|
||||
|
||||
`override` mutates the real `os.environ`, so child processes spawned **inside** the `with` block inherit the override. This is the supported way to seed a `subprocess.Popen`:
|
||||
|
||||
```python
|
||||
with envs.SGLANG_TEST_RETRACT.override(True):
|
||||
subprocess.Popen([...]).wait()
|
||||
```
|
||||
|
||||
A child started outside the `with` block sees the original value.
|
||||
|
||||
### `temp_set_env` is for non-sglang keys only
|
||||
|
||||
The module-level `temp_set_env(**env_vars)` helper exists for overriding **non-sglang** env vars (e.g. `CUDA_LAUNCH_BLOCKING`, `NCCL_DEBUG`) in tests. It explicitly rejects `SGLANG_*` / `SGL_*` keys:
|
||||
|
||||
```python
|
||||
# Wrong — raises ValueError
|
||||
with temp_set_env(SGLANG_TEST_RETRACT="true"): ...
|
||||
|
||||
# Right — sglang keys go through the descriptor
|
||||
with envs.SGLANG_TEST_RETRACT.override(True): ...
|
||||
|
||||
# Right — non-sglang keys go through temp_set_env
|
||||
with temp_set_env(CUDA_LAUNCH_BLOCKING="1"): ...
|
||||
```
|
||||
|
||||
The `allow_sglang=True` escape hatch exists for the rare case where you must bypass `Envs` (e.g. setting an env var **name** that's only constructed at runtime); don't use it just to skip writing a descriptor.
|
||||
|
||||
## Rule 4 — Naming: `SGLANG_` prefix + verb category
|
||||
|
||||
Prefix is always `SGLANG_` for new entries. `SGL_*` is auto-translated to `SGLANG_*` with a `DeprecationWarning` in `_convert_SGL_to_SGLANG`; never add a new `SGL_*` key.
|
||||
|
||||
The second token signals intent. Pick the right verb up front — renames require an alias entry (Rule 5).
|
||||
|
||||
| Verb | Meaning | Example |
|
||||
|---|---|---|
|
||||
| `ENABLE_FOO` | Knob that turns feature foo on/off. Default in the `EnvBool` encodes prod behavior. | `SGLANG_ENABLE_TORCH_COMPILE`, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM` |
|
||||
| `DISABLE_FOO` | Kill-switch. `DISABLE_FOO=True` turns foo off. | `SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP` |
|
||||
| `USE_FOO` | Selects which implementation / backend | `SGLANG_USE_AITER`, `SGLANG_USE_DEEPGEMM_BMM` |
|
||||
| `FORCE_FOO` | Overrides autodetection | `SGLANG_FORCE_FP8_MARLIN`, `SGLANG_FORCE_STREAM_INTERVAL` |
|
||||
| `LOG_FOO` | Logging-only knob | `SGLANG_LOG_GC`, `SGLANG_LOG_MS` |
|
||||
| `TEST_FOO` | Test-only hook | `SGLANG_TEST_RETRACT`, `SGLANG_TEST_MAX_RETRY` |
|
||||
| `DEBUG_FOO` | Debug-only instrumentation | `SGLANG_DEBUG_MEMORY_POOL`, `SGLANG_DEBUG_SYMM_MEM` |
|
||||
| `OPT_FOO` | Perf-optimization toggle (heavily used by DSV4 work) | `SGLANG_OPT_USE_FUSED_HASH_TOPK`, `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2` |
|
||||
|
||||
Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only forbidden combination is `DISABLE_FOO = EnvBool(True)`, because it produces a true double-negative at the call site (`if not envs.SGLANG_DISABLE_FOO.get():` reads as "if not disabled"). All other combinations are fine:
|
||||
|
||||
| Pattern | Call site | Verdict |
|
||||
|---|---|---|
|
||||
| `ENABLE_FOO = EnvBool(False)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — opt-in feature |
|
||||
| `ENABLE_FOO = EnvBool(True)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — on in prod, user opts out via `False` |
|
||||
| `DISABLE_FOO = EnvBool(False)` | `if not envs.SGLANG_DISABLE_FOO.get():` | OK — single negation, reads as "if enabled" |
|
||||
| `DISABLE_FOO = EnvBool(True)` | `if not envs.SGLANG_DISABLE_FOO.get():` | **Forbidden** — true double-negative |
|
||||
|
||||
`SGLANG_*` is the canonical sglang prefix. Vendor-integration keys (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`) keep their upstream prefix and live in the same `Envs` class — these are integration aliases, not sglang-owned feature flags.
|
||||
|
||||
## Rule 5 — Renames go through `*WithAlias` or `_print_deprecated_env`
|
||||
|
||||
For a rename where the old key must keep working with a warning:
|
||||
|
||||
```python
|
||||
SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK")
|
||||
```
|
||||
|
||||
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
|
||||
|
||||
For a full removal where the env var is going away, add to `_convert_SGL_to_SGLANG`:
|
||||
|
||||
```python
|
||||
_print_deprecated_env("SGLANG_OLD_NAME", "SGLANG_NEW_NAME") # mapped to a replacement
|
||||
_print_deprecated_env("SGLANG_OLD_NAME") # no replacement, gone
|
||||
```
|
||||
|
||||
For env-var to CLI-flag migration, add at module top-level:
|
||||
|
||||
```python
|
||||
_warn_deprecated_env_to_cli_flag(
|
||||
"SGLANG_FOO",
|
||||
"Please use '--foo' instead.",
|
||||
)
|
||||
```
|
||||
|
||||
Don't silently flip a default during a rename. If the new default disagrees with the old one, that's a behavior change — call it out in the PR body separately from the rename.
|
||||
|
||||
## Rule 6 — Env var vs CLI flag
|
||||
|
||||
| If the knob is… | Goes in |
|
||||
|---|---|
|
||||
| User-facing (documented, expected to flip per deployment) | `server_args.py` CLI flag |
|
||||
| Expert toggle, A-B kill-switch, vendor integration | `environ.py` env var |
|
||||
| Test / debug hook | `environ.py` env var with `TEST_` / `DEBUG_` prefix |
|
||||
| Temporary env var rolling out to a CLI flag | env var first, then migrate via `_warn_deprecated_env_to_cli_flag` |
|
||||
|
||||
Don't add a CLI flag that just forwards to an env var, and don't add an env var that duplicates an existing CLI flag. Pick one surface.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **External / vendor env vars consumed raw** (`HF_HUB_*`, `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `RANK`, `MASTER_ADDR`, etc.): see the decision table in Rule 1 — `os.getenv` is correct, don't pull them into `Envs`.
|
||||
- **Pre-existing `get_bool_env_var(...)` / `get_int_env_var(...)` call sites**: leave them as is; new code shouldn't add more, but mass-migration is out of scope for a feature PR.
|
||||
- **Upstream-aliased keys already in `Envs`** (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` — see Rule 1 decision table): the `SGLANG_` prefix rules in Rule 4 don't apply — the upstream prefix is the canonical name.
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
name: generate-profile
|
||||
description: Generate an e2e profiling trace of an SGLang server run. Launches a server, validates accuracy, captures a Chrome-compatible trace, and returns the profile path.
|
||||
---
|
||||
|
||||
# Generate an E2E Profile of an SGLang Server Run
|
||||
|
||||
This skill launches an SGLang server, validates it with a quick accuracy test, generates a profiling trace, and returns the profile file path.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||
- At least one available CUDA GPU
|
||||
|
||||
## Step-by-step Workflow
|
||||
|
||||
### Step 1: Launch the server
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=<gpu_id> sglang serve --model-path <model> --port <port> &
|
||||
```
|
||||
|
||||
- Default model: `Qwen/Qwen3-8B` (good balance of speed and quality)
|
||||
- Default port: `30000`
|
||||
- The server runs in the background. Save the PID for cleanup.
|
||||
- Use the GPU specified by the user's preferences (check memory files for GPU preferences).
|
||||
|
||||
### Step 2: Wait for server readiness
|
||||
|
||||
Poll the health endpoint until the server is ready:
|
||||
|
||||
```bash
|
||||
for i in $(seq 1 120); do
|
||||
if curl -s http://127.0.0.1:<port>/health 2>/dev/null | grep -q "ok\|healthy"; then
|
||||
echo "Server ready"
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
```
|
||||
|
||||
The server prints **"The server is fired up and ready to roll!"** to stdout when ready. The health endpoint returns 200 once the server can accept requests.
|
||||
|
||||
Typical startup time: 30-90 seconds depending on model size and whether CUDA graphs are being compiled.
|
||||
|
||||
### Step 3: Validate accuracy (sanity check)
|
||||
|
||||
```bash
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
|
||||
```
|
||||
|
||||
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
|
||||
- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
### Step 4: Generate the profile
|
||||
|
||||
```bash
|
||||
python3 -m sglang.test.send_one --profile
|
||||
```
|
||||
|
||||
This command:
|
||||
1. Sends a request to the server
|
||||
2. Triggers the profiler for 5 steps (default)
|
||||
3. Generates a trace file under `/tmp/<timestamp>/`
|
||||
4. The trace directory contains:
|
||||
- `<timestamp>-TP-0.trace.json.gz` — Chrome trace format (open in `chrome://tracing` or Perfetto)
|
||||
- `server_args.json` — the server configuration used
|
||||
|
||||
**Output format:**
|
||||
```
|
||||
Dump profiling traces to /tmp/<timestamp>
|
||||
```
|
||||
|
||||
The profile path is printed to stdout. Parse it from the output.
|
||||
|
||||
**Optional flags:**
|
||||
- `--profile-steps N` — number of profiling steps (default: 5)
|
||||
- `--profile-by-stage` — profile by stage (prefill/decode separately)
|
||||
- `--profile-prefix <path>` — custom output prefix
|
||||
|
||||
### Step 5: Kill the server
|
||||
|
||||
```bash
|
||||
pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
|
||||
```
|
||||
|
||||
Wait a moment and verify no sglang processes remain:
|
||||
```bash
|
||||
sleep 2 && pgrep -af "sglang serve" || echo "Server killed"
|
||||
```
|
||||
|
||||
### Step 6: Report the profile path
|
||||
|
||||
Return the profile directory path (e.g., `/tmp/1773999986.4769795`) and list its contents so the user knows what files were generated.
|
||||
|
||||
## Example Full Run
|
||||
|
||||
```bash
|
||||
# 1. Launch server
|
||||
source cleanup/bin/activate
|
||||
CUDA_VISIBLE_DEVICES=1 sglang serve --model-path Qwen/Qwen3-8B --port 30000 &
|
||||
|
||||
# 2. Wait for ready
|
||||
for i in $(seq 1 120); do
|
||||
curl -s http://127.0.0.1:30000/health | grep -q "ok" && break
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
|
||||
# Expected: Accuracy > 0.8
|
||||
|
||||
# 4. Profile
|
||||
python3 -m sglang.test.send_one --profile
|
||||
# Output: "Dump profiling traces to /tmp/1773999986.4769795"
|
||||
|
||||
# 5. Cleanup
|
||||
pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
|
||||
sleep 2
|
||||
|
||||
# 6. Check output
|
||||
ls -la /tmp/1773999986.4769795/
|
||||
# 1773999986.4851577-TP-0.trace.json.gz (Chrome trace)
|
||||
# server_args.json (server config)
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
- **Different port**: Pass `--port <port>` and use `--host 127.0.0.1 --port <port>` for test commands
|
||||
- **Multi-GPU**: Use `--tp <N>` for tensor parallelism; trace files will be generated per TP rank
|
||||
- **Longer profile**: Use `--profile-steps 10` for more steps in the trace
|
||||
- **Stage profiling**: Use `--profile-by-stage` to separate prefill and decode phases
|
||||
|
||||
## Viewing the Profile
|
||||
|
||||
Open the `.trace.json.gz` file in:
|
||||
- **Perfetto UI**: https://ui.perfetto.dev/ (drag and drop the file)
|
||||
- **Chrome tracing**: `chrome://tracing` (load the file)
|
||||
|
||||
Both support the gzipped Chrome trace format natively.
|
||||
+15
-15
@@ -46,18 +46,18 @@ in-flight row as shipped.
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/kernels/ops/layernorm/fused_rmsnorm.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
||||
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
||||
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
||||
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
||||
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
||||
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
||||
@@ -69,18 +69,18 @@ in-flight row as shipped.
|
||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/kernels/ops/attention/linear/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
|
||||
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
|
||||
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/kernels/ops/attention/linear/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/kernels/ops/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
|
||||
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
|
||||
|
||||
@@ -146,10 +146,10 @@ Stable entries should be folded into the mainline family rows above.
|
||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
|
||||
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
|
||||
|
||||
@@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package.
|
||||
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
|
||||
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
|
||||
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
|
||||
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
||||
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
||||
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
|
||||
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
|
||||
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
|
||||
+4
-1
@@ -18,7 +18,7 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
||||
|
||||
## Scheduler-side trace writing
|
||||
|
||||
- `python/sglang/srt/managers/scheduler_components/profiler_manager.py`
|
||||
- `python/sglang/srt/managers/scheduler_profiler_mixin.py`
|
||||
- actual trace start/stop behavior
|
||||
- filename pattern for `TP/DP/PP/EP` and optional stage suffixes
|
||||
- `CUDA_PROFILER` and torch profiler handling
|
||||
@@ -37,3 +37,6 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
||||
|
||||
- `test/registered/profiling/test_start_profile.py`
|
||||
- validates `/start_profile` behavior, including `CUDA_PROFILER`
|
||||
|
||||
- `test/registered/profiling/test_profile_v2.py`
|
||||
- validates stage-scoped trace outputs under `SGLANG_PROFILE_V2`
|
||||
+7
-9
@@ -391,7 +391,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
|
||||
"<br>python/sglang/srt/layers/layernorm.py"
|
||||
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||
"<br>python/sglang/srt/layers/communicator.py"
|
||||
),
|
||||
active_keywords=(
|
||||
"allreduce_fusion",
|
||||
@@ -419,7 +419,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
pattern="AITER allreduce fusion",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/distributed/communication_op.py"
|
||||
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||
"<br>python/sglang/srt/layers/communicator.py"
|
||||
"<br>python/sglang/srt/layers/layernorm.py"
|
||||
),
|
||||
active_keywords=(
|
||||
@@ -482,9 +482,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="Fused QK RoPE reshape + KV cache write",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py"
|
||||
),
|
||||
candidate_path="python/sglang/srt/layers/attention/utils.py",
|
||||
active_keywords=("fused_qk_rope_reshape_and_cache",),
|
||||
split_groups=(
|
||||
("rotary", "rope", "mrope"),
|
||||
@@ -535,7 +533,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused metadata copy for graph replay",
|
||||
candidate_path="python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py",
|
||||
candidate_path="python/sglang/kernels/ops/attention/fused_metadata_copy.py",
|
||||
active_keywords=(
|
||||
"fused_metadata_copy",
|
||||
"fused_metadata_copy_multi",
|
||||
@@ -702,7 +700,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
pattern="Fused MoE activation + quant / re-quant",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
|
||||
"<br>python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
|
||||
),
|
||||
active_keywords=(
|
||||
@@ -723,7 +721,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
FusionPatternSpec(
|
||||
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||
"python/sglang/srt/layers/communicator.py"
|
||||
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
||||
"forward_mla.py"
|
||||
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
||||
@@ -763,7 +761,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused quantize + indexed K-cache store",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py"
|
||||
"python/sglang/kernels/ops/attention/fused_store_index_cache.py"
|
||||
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
||||
),
|
||||
active_keywords=("fused_store_index_k_cache",),
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user