Merge b300/integration into main: 2026-11-17 c580 baseline
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
Promote the validated ds41-gist-exp baseline: - image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD) - P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T - D: fastokens / tokenizer-worker 16 - gateway: normal health/circuit, no P admission caps - c580 baseline 24.045 rps; c620 and buffer_only results recorded - Feishu baseline doc and deploy manifests included This is the base for the next Dynamo layer.
This commit is contained in:
@@ -0,0 +1,774 @@
|
|||||||
|
---
|
||||||
|
name: add-jit-kernel
|
||||||
|
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang.kernels JIT infrastructure and public operator groups
|
||||||
|
---
|
||||||
|
|
||||||
|
# Tutorial: Adding a New JIT Kernel to SGLang
|
||||||
|
|
||||||
|
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||||
|
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||||
|
The implementation tutorial below does not replace that API contract.
|
||||||
|
|
||||||
|
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Add a new operation that scales each element of a tensor by a scalar factor:
|
||||||
|
|
||||||
|
- Input: tensor `x` (CUDA) and scalar `factor` (float, passed at runtime)
|
||||||
|
- Output: `x * factor` (element-wise), allocated internally
|
||||||
|
- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
|
||||||
|
|
||||||
|
## When to use JIT vs AOT (`sgl-kernel`)
|
||||||
|
|
||||||
|
- **JIT (`jit_kernel`)**: prefer this first for kernels that do **not** depend on CUTLASS or another large C++ project. It is the default choice for lightweight kernels that benefit from rapid iteration and first-use compilation.
|
||||||
|
- **AOT (`sgl-kernel`)**: prefer this when the kernel **does** depend on CUTLASS or another large C++ project, or when it should live in `python/sglang/kernels/aot/` and participate in the wheel build / torch op registration flow.
|
||||||
|
- **Exception**: kernels that depend on `flashinfer`, or on CUTLASS that is already provided through `flashinfer`, can still be implemented as `jit_kernel`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
These hold for every step below.
|
||||||
|
|
||||||
|
- **`namespace sglang` is where JIT code lives.** Open it after the include block and close it at the end of the file, with the device kernels, traits and host wrapper inside. The shared `host::` / `device::` helpers are nested in it too, so they resolve unqualified. `load_jit` emits the `TVM_FFI_DLL_EXPORT_TYPED_FUNC` wrapper inside `namespace sglang` as well, so the `kernel_name` you pass from Python needs no `sglang::` prefix.
|
||||||
|
- **Tuning policy belongs in Python; C++ gets the answer, not the decision.** For a hyperparameter — split factor, threads per item, block size, vector width, an algorithm variant — prefer (not required) making it a **template parameter** and letting the `@cache_once` module factory choose the value, over a runtime `if`/`switch` in the launcher that picks among pre-instantiated kernels. The kernel then compiles exactly one specialisation and asserts its own preconditions with `static_assert`, while the heuristic that produced the value sits in Python where it is readable, adjustable, and inspectable without recompiling CUDA. `store_cache`'s `num_threads` is the worked example: a `get_kernel(num_split)` ladder guarded by byte-alignment `if constexpr`s became one template argument plus a heuristic in `_jit_kvcache_module`. The exception is a knob that genuinely varies per call with the runtime shape — that has to stay a kernel argument.
|
||||||
|
- **Check where the check is cheapest: `static_assert` > C++ host check > cached Python > per-call Python.** Anything fixed at compile time is a `static_assert`. Anything about the tensors is a `TensorMatcher` / `CHECK_HOST` in the C++ launcher, free next to a kernel launch. A check Python cannot delegate goes inside the `@cache_once` module factory, where it runs once per specialisation. What remains in the per-call entry point costs interpreter time on *every* forward, so it should be nothing but picking the module and allocating `out`.
|
||||||
|
- **Fixed-width integer types.** Prefer `int32_t` / `int64_t` / `uint32_t` / `size_t` over `int`, `long`, or `long long`, so an index has the same width on both sides of the FFI boundary. Bare `int` is fine only where the width plainly cannot matter — an unrolled loop counter over a `constexpr` bound, a template `int` parameter. Shapes arrive as `int64_t` (`SymbolicSize::unwrap()`); narrowing to `uint32_t` for in-kernel indexing is a deliberate act, so write the `static_cast` explicitly and only where the range is known.
|
||||||
|
- **Doxygen comments in C++.** Document exported entities with `///` or `/** ... */` blocks using `\brief`, `\param`, `\tparam`, `\return`, the way `include/sgl_kernel/` does. `python -m sglang.kernels.jit` writes `CommentFormat: Doxygen` into `.clangd` when clangd is 21 or newer, so these render on hover in the editor. Plain `//` remains fine for implementation notes inside a function body.
|
||||||
|
- **ASCII only in C++ and CUDA sources.** Write `--`, `->`, `<=` instead of `—`, `→`, `≤`, including in comments. `grep -nP '[^\x00-\x7F]' <file>` before committing.
|
||||||
|
- **`namespace details` is private.** Anything inside one is an implementation detail of its own header, free to change without notice. Do not name `details::` from another module, and never `using namespace details`. If you find yourself wanting something in there, that is the signal to promote it to a documented name instead.
|
||||||
|
- **`const T* __restrict__` for read-only pointers.** This is what `csrc/` does throughout, and it lets the compiler emit non-coherent (`LDG`) loads.
|
||||||
|
- **Watch the register budget.** For memory-bound kernels, keep to roughly 64 registers per thread so occupancy does not become the limit; prefer recomputing a value over letting it spill. Run `SGLANG_JIT_LOG_RESOURCE_USAGE=1 SGLANG_JIT_FORCE_RECOMPILE=1` to log per-kernel registers, spills and shared memory at INFO. Both halves are load-bearing: a cache hit has no compiler output to report, and passing `-Xptxas -v` by hand does nothing on its own because the build captures compiler output and replays it only on failure.
|
||||||
|
- **Alignment is a contract you enforce, not a property you hope for.** A vectorized copy derives its access width from the row *size* (`load_bytes` picks 16B for a 1024B row), but the address it lands on is `base + index * row_stride` — and a stride is not constrained by the size. `TensorMatcher` admits any stride unless you say otherwise, so a padded cache row silently produces `misaligned address`. Every kernel that vectorizes is highly recommended to end its validation with `.ensure_alignment(w)`, where `w` is the width the kernel actually uses.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Abstractions in `python/sglang/kernels/jit/include/sgl_kernel/`
|
||||||
|
|
||||||
|
**Always prefer these abstractions over raw CUDA primitives.** They provide safety, readability, and consistency with the rest of the codebase. The only reason to drop to raw primitives is performance the abstraction cannot reach — a trade you make deliberately, and justify in a comment.
|
||||||
|
|
||||||
|
### `utils.h` — Host-side utilities
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/utils.h>
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`CHECK_HOST(cond) << "msg " << value`** — **Preferred** runtime check: stream-style, throws `PanicError` with file/line info on failure. Zero overhead on the true path — the message expressions are only evaluated when the check fails.
|
||||||
|
- **`host::RuntimeCheck(cond, args...)`** — Function-style alternative to `CHECK_HOST`. Note its message args are always evaluated (even when the check passes), so prefer `CHECK_HOST` — especially on hot paths.
|
||||||
|
- **`host::Panic(args...)`** — Unconditionally throw a `PanicError` with a descriptive message.
|
||||||
|
- **`host::div_ceil(a, b)`** — Integer ceiling division `(a + b - 1) / b`.
|
||||||
|
- **`host::irange(n)`** / **`host::irange(start, end)`** — Range views for cleaner loops.
|
||||||
|
- **`host::pointer::offset(ptr, offsets...)`** — Byte-safe pointer arithmetic on `void*`. Use this instead of raw casts.
|
||||||
|
|
||||||
|
### `utils.cuh` — Device-side utilities + `LaunchKernel`
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/utils.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Type aliases**: `fp16_t`, `bf16_t`, `fp32_t`, `fp8_e4m3_t`, `fp8_e5m2_t` and their packed variants `fp16x2_t`, `bf16x2_t`, `fp32x2_t`, etc.
|
||||||
|
- **`SGL_DEVICE`** — Expands to `__forceinline__ __device__`. Use on all device functions.
|
||||||
|
- **`SGL_DEVICE_HOST`** — `__forceinline__ __device__ __host__`. Use it when a `constexpr` helper must be callable from both the kernel and its launcher, so the two cannot drift.
|
||||||
|
- **`device::kWarpThreads`** — Constant `32` (the codebase's *logical* warp width). `kWarpSize` is an alias. Note this is a group size, not the hardware wave: on gfx950 `warpSize` is 64, and `warp::kFullWidth` reflects that.
|
||||||
|
- **`device::get_lane_id<kNumThreads = kWarpThreads>()`** — this thread's index within its logical group. **Prefer it over `threadIdx.x % kNumThreads` whenever the value feeds an address.** On CUDA the lane register is one read that folds into the address computation, whereas the modulo is a derived value the compiler re-materialises at *every* address scale — 8 instructions and 2 registers on a two-tile warp copy, measured on sm_100a. It picks the cheaper form per platform, so callers never need to care. Only equals the true in-warp lane when `blockDim.x` is a multiple of `kNumThreads`.
|
||||||
|
- **`device::load_as<T>(ptr, offset)`** / **`device::store_as<T>(ptr, val, offset)`** — Type-safe loads/stores from `void*`.
|
||||||
|
- **`device::pointer::offset(ptr, offsets...)`** — Pointer arithmetic on device.
|
||||||
|
- **`host::LaunchKernel(grid, block, device_or_stream [, smem])`** — RAII kernel launcher that:
|
||||||
|
- Resolves the CUDA stream from a `DLDevice` via TVM-FFI automatically.
|
||||||
|
- Checks the CUDA error with file/line info after launch via `operator()(kernel, args...)`.
|
||||||
|
- Supports `.enable_pdl(bool)` for PDL (Programmatic Dependent Launch, SM90+).
|
||||||
|
- **`device::PDLWaitPrimary<kUsePDL>()`** / **`device::PDLTriggerSecondary<kUsePDL>()`** — The two halves of PDL, on sm_90+ (no-ops on older archs and ROCm). Their guarantees are **not** symmetric:
|
||||||
|
- `PDLTriggerSecondary` (`griddepcontrol.launch_dependents`) only lets the next kernel in the stream *start* early. It carries no memory ordering and publishes nothing — matching that, the header's asm has no `"memory"` clobber.
|
||||||
|
- `PDLWaitPrimary` (`griddepcontrol.wait`) is the ordering point: it waits until the preceding kernel has fully finished and its writes are visible.
|
||||||
|
|
||||||
|
So every read of data the preceding kernel produced must come after `PDLWaitPrimary()`. What overlaps with the primary's tail is whatever you put *before* the wait — loading parameters, computing indices, touching buffers the primary never wrote — so a kernel that waits on its first line gains nothing. Neither call is a barrier: threads may reach or skip them independently. See "Programmatic Dependent Launch and Synchronization" in the CUDA C++ Programming Guide.
|
||||||
|
- **`CHECK_CUDA(expr) << "context"`** — Stream-style CUDA error check; evaluates `expr` once and throws `PanicError` with `cudaGetErrorString` + file/line info if it is not `cudaSuccess`. Extra streamed context is optional.
|
||||||
|
- **`host::RuntimeDeviceCheck(cudaError_t)`** — Function-style alternative to `CHECK_CUDA`. It takes no context message, so prefer `CHECK_CUDA`, which builds its error object only on the failure path.
|
||||||
|
|
||||||
|
### `tensor.h` — Tensor validation (`TensorMatcher`, Symbolic types)
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/tensor.h>
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the **primary validation API** for all kernel launchers. Use it to validate every `tvm::ffi::TensorView` argument.
|
||||||
|
|
||||||
|
- **`host::SymbolicSize{"name"}`** — A named symbolic dimension. Call `.set_value(n)` to pin it, `.unwrap()` to extract after verification.
|
||||||
|
- **`host::SymbolicDType`** — Symbolic dtype. Use `.set_options<Ts...>()` to restrict allowed types.
|
||||||
|
- **`host::SymbolicDevice`** — Symbolic device. Use `.set_options<kDLCUDA>()` to restrict to CUDA.
|
||||||
|
- **`host::TensorMatcher({dims...})`** — Fluent builder for tensor validation:
|
||||||
|
- `.with_dtype<T>()` — require a specific C++ type (e.g. `fp16_t`)
|
||||||
|
- `.with_dtype<T1, T2, ...>()` — allow a set of types
|
||||||
|
- `.with_device<kDLCUDA>(device_sym)` — require CUDA and bind the checked device to a `SymbolicDevice`
|
||||||
|
- `.with_strides({strides...})` — validate strides (omit to require contiguous)
|
||||||
|
- `.ensure_alignment(bytes)` — require the data pointer **and every stride except the innermost** to be a multiple of `bytes` (a power of two). Size-1 dimensions are skipped, since their strides are arbitrary. This is how a vectorized kernel states its precondition; see the alignment convention above.
|
||||||
|
- `.verify(tensor_view)` — execute the check; throws `PanicError` with full context on failure; **chainable** (`verify(a).verify(b)` to check multiple tensors with the same shape)
|
||||||
|
- **`host::is_type<T>(dtype)`** — whether a `DLDataType` denotes the C++ type `T` (e.g. `fp16_t`).
|
||||||
|
|
||||||
|
**Typical pattern:**
|
||||||
|
```cpp
|
||||||
|
auto N = SymbolicSize{"num_elements"};
|
||||||
|
auto device = SymbolicDevice{};
|
||||||
|
device.set_options<kDLCUDA>();
|
||||||
|
TensorMatcher({N}) //
|
||||||
|
.with_dtype<fp16_t>()
|
||||||
|
.with_device<kDLCUDA>(device)
|
||||||
|
.ensure_alignment(16) // e.g. for 128-bit vectorized load/store
|
||||||
|
.verify(dst)
|
||||||
|
.verify(src); // same shape, dtype, device as dst
|
||||||
|
const int64_t n = N.unwrap();
|
||||||
|
const DLDevice dev = device.unwrap();
|
||||||
|
const int64_t last_dim = 128;
|
||||||
|
TensorMatcher({N, last_dim}) // a fixed dimension can be a plain integer
|
||||||
|
.with_dtype<fp16_t>()
|
||||||
|
.with_device<kDLCUDA>(device)
|
||||||
|
.verify(tensor_2d);
|
||||||
|
```
|
||||||
|
|
||||||
|
### `ffi.h` — Tensor allocation and blob wrapping (`host::ffi::`)
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/ffi.h>
|
||||||
|
```
|
||||||
|
|
||||||
|
The counterpart to `tensor.h`: that one validates what came in, this one produces new `tvm::ffi::Tensor` values. Allocation goes through the environment allocator (`TVMFFIEnvTensorAlloc`), so buffers come from PyTorch's caching allocator rather than a raw `cudaMalloc`.
|
||||||
|
|
||||||
|
- **`host::alloc_workspace_tensor(nbytes, device)`** (declared in `utils.cuh`) — **the way to get scratch memory**: a 1-D `uint8` tensor of `nbytes`, or an empty tensor when `nbytes == 0`. Hold the returned `Tensor` in a local across every launch that touches it — it frees on destruction.
|
||||||
|
- **`host::ffi::empty(shape, dtype, device)`** — Uninitialized tensor; `shape` accepts a braced list, so `ffi::empty({rows, sizeof(Plan)}, dtype, device)` works for a typed scratch array.
|
||||||
|
- **`host::ffi::empty_like(tensor_view)`** — Same shape, dtype, and device as an existing tensor.
|
||||||
|
- **`host::ffi::from_blob(data, shape, dtype, device[, deleter, stride, byte_offset])`** / **`from_blob_like(data, tensor_view, ...)`** — View memory you already own as a `Tensor`, no copy. The default deleter does nothing, so ownership stays with the caller; pass one only when the `Tensor` should own the block. Strides default to contiguous.
|
||||||
|
|
||||||
|
### `type.cuh` — `DTypeTrait<T>`, `packed_t<T>`, and reduction traits
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/type.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`DTypeTrait<T>`** — Static trait struct, specialized for integral types, `fp32_t`, `fp16_t`, `bf16_t`, `fp8_e4m3_t`, and their packed x2/x4 variants. Provides:
|
||||||
|
- `DTypeTrait<T>::from(value)` — convert from another type via the right CUDA intrinsic (e.g. `fp32_t` → `fp16_t`)
|
||||||
|
- `DTypeTrait<T>::abs/max/min` — type-dispatched math (fp32, fp16/bf16 scalar and x2, integrals)
|
||||||
|
- `DTypeTrait<T>::sqrt/rsqrt/exp/sin/cos(x)` — `fp32_t` only
|
||||||
|
- Metadata: `packed_t` / `unpacked_t` / `kVecSize` (packed layout), `kFloatMax` (dtype max as float, e.g. 448.0f for fp8-e4m3), `kZeroBits`
|
||||||
|
- **`packed_t<T>`** — Two-element packed alias: `packed_t<fp16_t>` = `fp16x2_t`, `packed_t<bf16_t>` = `bf16x2_t`, `packed_t<fp32_t>` = `fp32x2_t`. Use for vectorized loads/stores.
|
||||||
|
- **`device::cast<To, From>(value)`** — Type-safe cast using `DTypeTrait`, e.g. `cast<fp32x2_t, fp16x2_t>(v)`.
|
||||||
|
- **`device::unpack(value)`** — View a packed value as an `unpacked_t[kVecSize]` array reference (e.g. `fp32x2_t` → `fp32_t[2]`); element writes propagate back to the packed value.
|
||||||
|
- **`device::ReductionOp` (`SUM`/`MAX`/`MIN`) and `device::ReductionTrait<Op, T>::reduce(x, y)`** — One binary reduction step, dispatched through `DTypeTrait` (packed types reduce elementwise). This is the engine behind `warp::reduce`; use it directly when writing custom reductions.
|
||||||
|
|
||||||
|
### `vec.cuh` — Vectorized memory access (`AlignedVector`)
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/vec.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`device::AlignedVector<T, N>`** — Aligned storage for N elements of type T. N must be a power of two, `sizeof(T)*N <= 32`. Enables vectorized loads/stores for bandwidth efficiency. In terms of API/codegen constraints, the upper bound is 256-bit; in practice, 128-bit is the portable default, while 256-bit vectorization is typically only viable on `SM100+` and should be gated by an architecture check when needed.
|
||||||
|
- `.load(ptr, offset)` — vectorized load from `ptr[offset]`
|
||||||
|
- `.store(ptr, offset)` — vectorized store to `ptr[offset]`
|
||||||
|
- `.fill(value)` — fill all N elements with `value`
|
||||||
|
- `operator[](i)` — element access
|
||||||
|
- **`device::LoadStoreBytes`** — named widths to pass around instead of bare integers: `MAX_GMEM` (arch-dependent, 32 on Blackwell), `MAX_SMEM` (16), `MAX_PORTABLE` (16, safe on CUDA and HIP), `MIN_COALALESCED` (4), and `RAW_1B` .. `RAW_32B`.
|
||||||
|
|
||||||
|
### `tile.cuh` — `tile::Memory` (strided memory access pattern)
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/tile.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `tile::Memory<T>` is fundamentally a **1D cooperative accessor** over a contiguous region.
|
||||||
|
- **`device::tile::Memory<T>::cta(blockDim.x)`** — Creates a tile accessor where each thread handles `tid = threadIdx.x` with stride `tsize` (for `cta(blockDim.x)`, this is `blockDim.x`). Common for loops over a 1D array.
|
||||||
|
- **`device::tile::Memory<T>::warp()`** — the 32-lane flavour, and what `warp::load_bytes` is built on. It takes its lane index from `get_lane_id()` rather than `threadIdx.x % 32`, for the addressing reason described under `utils.cuh`. Use `warp(int n)` for a narrower sub-group; that overload keeps the modulo, since a hardware lane index is the wrong grouping below 32.
|
||||||
|
- **`device::tile::Memory<T>::thread()`** — single-thread accessor, no cooperation.
|
||||||
|
- **`.load(ptr, offset)`** — loads `ptr[tid + offset * tsize]`
|
||||||
|
- **`.store(ptr, val, offset)`** — stores to `ptr[tid + offset * tsize]`
|
||||||
|
- **`.in_bound(n, offset)`** — boundary check
|
||||||
|
|
||||||
|
For a **2D tile**, either flatten `(row, col)` into a linear tile index first, or compute the address manually with `ptr[row * stride + col]` using your thread/block coordinates.
|
||||||
|
|
||||||
|
### `bits.h` — Compile-time bit helpers (`host::`)
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/bits.h>
|
||||||
|
```
|
||||||
|
|
||||||
|
`constexpr` wrappers over `<bit>`, usable in `static_assert` and in template arguments: `host::is_pow2(x)`, `log2_floor(x)`, `log2_ceil(x)` (both `-1` for `x == 0`), `round_up_pow2(x)`, `round_down_pow2(x)`. They live in `host::` but are constant-expression-only, so device-side `static_assert` can call them.
|
||||||
|
|
||||||
|
### `math.cuh` — Device math (`device::math::`)
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/math.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `device::math::max/min<T>(a, b)` — type-dispatched binary math via `DTypeTrait`
|
||||||
|
- `device::math::abs/sqrt/rsqrt/exp/sin/cos<T>(x)` — type-dispatched unary math via `DTypeTrait`
|
||||||
|
|
||||||
|
### `warp.cuh` — Warp-level primitives
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/warp.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vectorized row copy — reach for this before hand-rolling.** A warp cooperatively moving one contiguous row is the single most common shape in this tree, and it used to be re-implemented per kernel.
|
||||||
|
|
||||||
|
- **`warp::load_bytes<kBytes, kPattern>(src)`** / **`warp::store_bytes<kBytes, kPattern>(dst, val)`** — the whole warp moves `kBytes` from/to one contiguous row. `kBytes` is arbitrary, down to 1; the helper picks the widest vector that divides it and handles the ragged tail. The returned value is opaque and encodes the width, so a load and its matching store **must use the same `<kBytes, kPattern>`**.
|
||||||
|
- **`warp::LoadStorePattern`** — the `kPattern` values, and the sign carries the meaning. A **negative** one (`WARP_UNIFORM_16B` and friends: `_GMEM`, `_SMEM`, `_4B`/`_8B`/`_32B`) says *the whole warp splits this row*, so the width is derived from the per-lane share (`gcd(kBytes / 32, |kPattern|)`), falling back to `gcd(kBytes, 4)` when the row is too narrow to give every lane 4 bytes. A **positive** one (a bare `LoadStoreBytes` value) makes no warp-splitting assumption and just caps the per-thread vector at `gcd(kBytes, kPattern)`. `WARP_UNIFORM_16B` is the usual choice for a row copy.
|
||||||
|
- **`LoadStorePattern::get_vec_bytes<kBytes, kPattern>()`** — the width actually chosen. `SGL_DEVICE_HOST`, so **the launcher must call this to feed `.ensure_alignment(...)`**. Pass it the same value the kernel passes `load_bytes` — for a split row that is the *per-warp share*, not the whole row; the full row can resolve to a different width and would under-constrain the strides.
|
||||||
|
|
||||||
|
**Reductions.**
|
||||||
|
|
||||||
|
- `warp::reduce<Op, kStart, kFinish>(value, active_mask)` — `Op` is a `device::ReductionOp` (`SUM`/`MAX`/`MIN`). `kStart` and `kFinish` bound a range of lane-index bits: `<N, 1>` reduces within contiguous groups of `N` lanes, `<kFullWidth, N>` reduces *across* groups at the same offset. The operation is **symmetric** — `<A, B>` and `<B, A>` reduce the same lane set.
|
||||||
|
- `warp::reduce_sum/reduce_max/reduce_min<kStart, kFinish>(value)` — wrappers. Work for any type with a `ReductionTrait`: floats, integers, packed x2. This is more widely used in practice than generic `reduce`.
|
||||||
|
- `warp::inclusive_reduce<Op, kWidth, kStart, kFinish>(val, lane_id, mask)` and `inclusive_sum` / `inclusive_max` / `inclusive_min` — segmented inclusive scan: every lane keeps its own running total, unlike `reduce`. Forward when `kStart < kFinish`, backward when `kStart > kFinish`. `lane_id` defaults to `get_lane_id<kWidth>()`, which is correct by construction — only pass it if you already have the **segment-relative** index (`threadIdx.x % kWidth`); a warp-relative one silently corrupts every segment past the first when `kWidth < 32`.
|
||||||
|
- `warp::broadcast<kWidth>(value, src_lane)` — `src_lane` is **segment-relative**: each `kWidth` segment reads its own lane, not one warp-wide source.
|
||||||
|
- `warp::get_lane_id`, `warp::elect_one_lane()` (one elected lane, for gating a single-thread TMA issue; CUDA-only), `warp::kFullMask` and `warp::kFullWidth` (32 on CUDA, 64 on HIP — note this differs from `kWarpThreads`).
|
||||||
|
|
||||||
|
### `cta.cuh` — CTA-level primitives
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/cta.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `device::cta::reduce_max<T>(value, smem, min_value)` — CTA-wide max using shared memory + warp reduction. Caller is responsible for a `__syncthreads()` after if the result in `smem[0]` is needed.
|
||||||
|
|
||||||
|
### `atomic.cuh` — Atomic operations
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/atomic.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `device::atomic::max(float* addr, float value)` — float atomic max (handles negative values correctly via bit tricks).
|
||||||
|
- **`device::atomic::Event`** — a cross-CTA arrive/wait counter in one 32-bit word. Producers call `arrive()`; consumers call `wait(num_producers)` (exactly one consumer) or `wait_multi<kConsumerBits>(num_producers, num_consumers, n)` (several). The last released consumer resets the word, so one `Event` is reusable across launches with no host re-zero. Preconditions, all of them load-bearing:
|
||||||
|
- the word must be **zeroed by the host** before first use, and must live in **global** memory — a shared or local `Event` is an illegal address;
|
||||||
|
- `arrive()` is a release and the waits are acquires, so a producer's writes before `arrive()` are visible after the wait;
|
||||||
|
- **generations must be explicitly ordered** — the word has no phase bit, so overlapping generation N+1 with a consumer still in generation N is undefined behaviour (a lagging consumer never exits);
|
||||||
|
- `wait()` and `wait_multi()` lay the word out incompatibly; mixing them on one `Event` is undefined behaviour.
|
||||||
|
|
||||||
|
### `runtime.cuh` — Occupancy and device info
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <sgl_kernel/runtime.cuh>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `host::runtime::get_blocks_per_sm(kernel, block_dim)` — max active blocks per SM (occupancy)
|
||||||
|
- `host::runtime::get_sm_count(device_id [, use_cache])` — number of SMs on the device. Memoized per device ordinal, so it is cheap enough to call on a launch path; pass `use_cache=false` to force a driver query.
|
||||||
|
- `host::runtime::get_cc_major` / `get_cc_minor` / `get_sm_version` — compute capability, same caching.
|
||||||
|
|
||||||
|
**Do not query the architecture at runtime.** The JIT compiles for the exact local GPU, so the arch is a *compile-time* fact: `SGL_CUDA_ARCH` is injected by `load_jit`, and `SGL_ARCH_HOPPER_OR_GREATER` / `SGL_ARCH_BLACKWELL_OR_GREATER` / `device::kMaxVecBytes` are derived from it. Reach for `get_cc_*` only for something the arch genuinely does not determine. SM *count* is the opposite case — it varies within an arch (B200 vs B300 vs a MIG slice), so it has to stay a runtime query.
|
||||||
|
|
||||||
|
**Persistent kernel pattern** (cap blocks to SM count × occupancy):
|
||||||
|
```cpp
|
||||||
|
static const uint32_t max_occ = runtime::get_blocks_per_sm(kernel, kBlockSize);
|
||||||
|
static const uint32_t num_sm = runtime::get_sm_count(device.unwrap().device_id);
|
||||||
|
const auto num_blocks = std::min(num_sm * max_occ, div_ceil(n, kBlockSize));
|
||||||
|
LaunchKernel(num_blocks, kBlockSize, device.unwrap())(kernel, params);
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 0 (optional): Generate a `.clangd` config for better IDE support
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m sglang.kernels.jit -h # for verbose help info about clangd configuration
|
||||||
|
python -m sglang.kernels.jit
|
||||||
|
python -m sglang.kernels.jit --dep cutlass flashinfer # with cutlass/flashinfer dependency
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1: Implement the CUDA kernel in `kernels/jit/csrc/`
|
||||||
|
|
||||||
|
Create `python/sglang/kernels/jit/csrc/elementwise/scale.cuh`.
|
||||||
|
|
||||||
|
The implementation fully uses the project abstractions described above:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// NOTE: Comments for headers are not common in practice.
|
||||||
|
// It is only shown here for tutorial purposes to highlight the key abstractions.
|
||||||
|
#include <sgl_kernel/tensor.h> // For TensorMatcher, SymbolicSize, SymbolicDevice
|
||||||
|
#include <sgl_kernel/type.cuh> // For DTypeTrait, fp16_t, bf16_t, fp32_t
|
||||||
|
#include <sgl_kernel/utils.h> // For CHECK_HOST, div_ceil
|
||||||
|
#include <sgl_kernel/utils.cuh> // For LaunchKernel, SGL_DEVICE
|
||||||
|
#include <sgl_kernel/vec.cuh> // For AlignedVector
|
||||||
|
|
||||||
|
#include <dlpack/dlpack.h>
|
||||||
|
#include <tvm/ffi/container/tensor.h>
|
||||||
|
|
||||||
|
namespace sglang {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* \brief Element-wise scale using vectorized 128-bit loads/stores.
|
||||||
|
*
|
||||||
|
* \tparam T Element type: fp16_t | bf16_t | fp32_t
|
||||||
|
* \tparam kVecN Elements per vector load (e.g. 8 for fp16)
|
||||||
|
* \tparam kUsePDL Whether to emit the PDL wait/trigger pair
|
||||||
|
* \param dst Output buffer, `n_total` elements
|
||||||
|
* \param src Input buffer, `n_total` elements
|
||||||
|
* \param factor Runtime scale factor
|
||||||
|
* \param n_total Number of elements to scale
|
||||||
|
*/
|
||||||
|
template <typename T, int kVecN, bool kUsePDL>
|
||||||
|
__global__ void scale_kernel(T* __restrict__ dst,
|
||||||
|
const T* __restrict__ src,
|
||||||
|
float factor,
|
||||||
|
uint32_t n_total) {
|
||||||
|
using vec_t = device::AlignedVector<T, kVecN>;
|
||||||
|
const uint32_t n_vecs = n_total / kVecN;
|
||||||
|
|
||||||
|
// If using PDL, wait for primary kernel before any global memory load.
|
||||||
|
// This is NOT a synchronization point, which means some threads can early exit before this.
|
||||||
|
device::PDLWaitPrimary<kUsePDL>();
|
||||||
|
|
||||||
|
// --- vectorised body ---
|
||||||
|
const uint32_t vec_stride = blockDim.x * gridDim.x;
|
||||||
|
for (uint32_t vi = blockIdx.x * blockDim.x + threadIdx.x;
|
||||||
|
vi < n_vecs;
|
||||||
|
vi += vec_stride) {
|
||||||
|
vec_t v;
|
||||||
|
v.load(src, vi);
|
||||||
|
#pragma unroll
|
||||||
|
for (int i = 0; i < kVecN; ++i) {
|
||||||
|
v[i] = static_cast<T>(static_cast<float>(v[i]) * factor);
|
||||||
|
}
|
||||||
|
v.store(dst, vi);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- scalar tail ---
|
||||||
|
const uint32_t base = n_vecs * kVecN;
|
||||||
|
const uint32_t scalar_stride = blockDim.x * gridDim.x;
|
||||||
|
for (uint32_t i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||||
|
base + i < n_total;
|
||||||
|
i += scalar_stride) {
|
||||||
|
dst[base + i] = static_cast<T>(static_cast<float>(src[base + i]) * factor);
|
||||||
|
}
|
||||||
|
|
||||||
|
// If using PDL, signal for the secondary kernel to start after all threads have finished
|
||||||
|
// This is NOT a synchronization point, which means some threads can early exit before this.
|
||||||
|
device::PDLTriggerSecondary<kUsePDL>();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* \brief Validate the tensors, select the vector width, launch `scale_kernel`.
|
||||||
|
*
|
||||||
|
* \tparam T Element type: fp16_t | bf16_t | fp32_t
|
||||||
|
* \tparam kUsePDL Whether to launch with PDL enabled
|
||||||
|
* \param dst Output tensor; same shape / dtype / device as `src`
|
||||||
|
* \param src Input tensor on CUDA
|
||||||
|
* \param factor Runtime scale factor
|
||||||
|
*/
|
||||||
|
template <typename T, bool kUsePDL>
|
||||||
|
void scale(tvm::ffi::TensorView dst, tvm::ffi::TensorView src, float factor) {
|
||||||
|
using namespace host;
|
||||||
|
|
||||||
|
// 1. Validate input tensors with TensorMatcher
|
||||||
|
SymbolicSize N = {"num_elements"};
|
||||||
|
SymbolicDevice device_;
|
||||||
|
device_.set_options<kDLCUDA>();
|
||||||
|
|
||||||
|
TensorMatcher({N}) //
|
||||||
|
.with_dtype<T>()
|
||||||
|
.with_device<kDLCUDA>(device_)
|
||||||
|
.verify(dst)
|
||||||
|
.verify(src); // same shape / dtype / device as dst
|
||||||
|
|
||||||
|
const uint32_t n = static_cast<uint32_t>(N.unwrap());
|
||||||
|
const DLDevice device = device_.unwrap();
|
||||||
|
|
||||||
|
CHECK_HOST(n > 0) << "scale: num_elements must be > 0, got " << n;
|
||||||
|
|
||||||
|
// 2. Choose vector width for 128-bit loads (16 bytes)
|
||||||
|
// fp16/bf16: 8 elements x 2 bytes = 16 bytes
|
||||||
|
// fp32: 4 elements x 4 bytes = 16 bytes
|
||||||
|
// We encourage using `device::kMaxVecBytes`, which will change according to
|
||||||
|
// the target architecture and can enable 256-bit vectorization on SM100+ if desired.
|
||||||
|
// But 128-bit is more commonly adapted for better compatibility,
|
||||||
|
// so it's still ok to hardcode 16 here just for simplicity.
|
||||||
|
constexpr int kVecN = 16 / sizeof(T);
|
||||||
|
const uint32_t n_work_items = div_ceil(n, static_cast<uint32_t>(kVecN));
|
||||||
|
|
||||||
|
// 3. Launch
|
||||||
|
constexpr uint32_t kBlockSize = 256;
|
||||||
|
const uint32_t grid = div_ceil(n_work_items, kBlockSize);
|
||||||
|
|
||||||
|
// PDL feature is 100% optional. Without `enable_pdl`, the code should still be correct.
|
||||||
|
// Try to enable it if profiling shows that it can benefit the performance of this kernel.
|
||||||
|
LaunchKernel(grid, kBlockSize, device).enable_pdl(kUsePDL)(
|
||||||
|
scale_kernel<T, kVecN, kUsePDL>,
|
||||||
|
static_cast<T*>(dst.data_ptr()),
|
||||||
|
static_cast<const T*>(src.data_ptr()),
|
||||||
|
factor,
|
||||||
|
n);
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace sglang
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
|
||||||
|
- Include headers from `sgl_kernel/` — **not** raw CUDA headers for anything already covered
|
||||||
|
- Use `TensorMatcher` for all tensor validation; never manually check shape/dtype/device
|
||||||
|
- Use `AlignedVector` for vectorised 128-bit loads/stores — significant bandwidth win
|
||||||
|
- Use `LaunchKernel` — it resolves the stream and checks errors automatically
|
||||||
|
- Use `CHECK_HOST(cond) << ...` for runtime assertions with useful error messages (zero overhead when the check passes)
|
||||||
|
- Prefer passing runtime scalars like `factor` directly unless compile-time specialisation is genuinely required
|
||||||
|
- `fp16_t` / `bf16_t` / `fp32_t` are the project's type aliases (from `utils.cuh`)
|
||||||
|
- `device::cast<To, From>` or `DTypeTrait<T>::from(val)` for cross-type conversions
|
||||||
|
- `device::math::` functions for device math instead of bare `__` intrinsics if possible.
|
||||||
|
- Consider PDL — it can help when the kernel has prologue work to overlap. Place `PDLWaitPrimary()` right before the first read of upstream data, not at the top of the kernel
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2: Add the Python wrapper in `kernels/ops/`
|
||||||
|
|
||||||
|
The wrapper lives next to its functional group under `python/sglang/kernels/ops/`, not beside the CUDA source — `kernels/jit/` holds only the JIT infrastructure (`csrc/`, `include/`, `utils/`, `benchmark/`). Create `python/sglang/kernels/ops/elementwise/scale.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from sglang.kernels.jit.utils import (
|
||||||
|
cache_once,
|
||||||
|
is_arch_support_pdl,
|
||||||
|
load_jit,
|
||||||
|
make_cpp_args,
|
||||||
|
)
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from tvm_ffi.module import Module
|
||||||
|
|
||||||
|
|
||||||
|
@cache_once
|
||||||
|
def _jit_scale_module(dtype: torch.dtype) -> Module:
|
||||||
|
"""Compile and cache the JIT scale module for a given dtype."""
|
||||||
|
# Checks on the compile key live here, not in `scale`: `cache_once` keys on
|
||||||
|
# `dtype`, so this runs once per specialisation instead of once per call.
|
||||||
|
if dtype not in (torch.float16, torch.bfloat16, torch.float32):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Unsupported dtype {dtype}. Supported: float16, bfloat16, float32"
|
||||||
|
)
|
||||||
|
args = make_cpp_args(dtype, is_arch_support_pdl())
|
||||||
|
return load_jit(
|
||||||
|
"scale",
|
||||||
|
*args,
|
||||||
|
cuda_files=["elementwise/scale.cuh"],
|
||||||
|
cuda_wrappers=[("scale", f"scale<{args}>")],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) -> torch.Tensor:
|
||||||
|
"""
|
||||||
|
Element-wise scale: dst = src * factor.
|
||||||
|
|
||||||
|
Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
src : CUDA tensor (FP16 / BF16 / FP32)
|
||||||
|
factor : scale factor
|
||||||
|
out : optional pre-allocated output tensor (same shape/dtype as src)
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
Scaled tensor (dst = src * factor).
|
||||||
|
"""
|
||||||
|
# DO NOT add proactive validation here: every check costs interpreter time
|
||||||
|
# on a per-forward path. Tensor invariants belong in the C++ launcher, and
|
||||||
|
# anything about the compile key belongs in `_jit_scale_module`.
|
||||||
|
if out is None:
|
||||||
|
out = torch.empty_like(src)
|
||||||
|
|
||||||
|
module = _jit_scale_module(src.dtype)
|
||||||
|
module.scale(out, src, factor)
|
||||||
|
return out
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
|
||||||
|
- Use `cache_once` — **not** `functools.lru_cache` (incompatible with `torch.compile`)
|
||||||
|
- `load_jit` first arg(s) form the unique build marker; same marker = same cached binary
|
||||||
|
- Only include compile-time specialisation knobs in the build marker; runtime values like `factor` should stay runtime unless the kernel truly needs templating
|
||||||
|
- `cuda_wrappers`: `(export_name, kernel_symbol)` — `export_name` is called from Python
|
||||||
|
- `make_cpp_args(dtype, ...)` converts `torch.dtype` to C++ type alias:
|
||||||
|
- `is_arch_support_pdl()` checks if the current architecture supports PDL, which is typically passed as a template argument to the kernel.
|
||||||
|
- Keep the entry point thin (see Conventions). What Python must still check goes in the `@cache_once` module factory, not in the entry point: `cache_once` keys on its arguments, so a check there costs one evaluation per specialisation instead of one per call — that is where the supported-dtype guard lives. Tensor invariants belong in the C++ launcher; if something here never reaches a `.verify(...)`, close that gap on the C++ side rather than in Python
|
||||||
|
|
||||||
|
| `torch.dtype` | C++ type |
|
||||||
|
|--------------------|------------|
|
||||||
|
| `torch.float16` | `fp16_t` |
|
||||||
|
| `torch.bfloat16` | `bf16_t` |
|
||||||
|
| `torch.float32` | `fp32_t` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Register the public entry point (required)
|
||||||
|
|
||||||
|
Add metadata in `python/sglang/kernels/ops/elementwise/__init__.py` so the
|
||||||
|
operator appears in the registry without importing its implementation:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sglang.kernels.registry import register_kernel
|
||||||
|
from sglang.kernels.spec import CapabilityRequirement, KernelBackend, KernelSpec
|
||||||
|
|
||||||
|
register_kernel(
|
||||||
|
KernelSpec(
|
||||||
|
op="elementwise.scale",
|
||||||
|
backend=KernelBackend.JIT,
|
||||||
|
target="sglang.kernels.ops.elementwise.scale:scale",
|
||||||
|
capabilities=frozenset({CapabilityRequirement.CUDA}),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Record the devices the implementation actually supports; the CUDA-only scale
|
||||||
|
example above does not imply that every JIT kernel is CUDA-only. Runtime and
|
||||||
|
correctness tests import `scale` from `sglang.kernels.ops.elementwise.scale`.
|
||||||
|
Do not import the implementation eagerly in the group initializer. When an
|
||||||
|
existing `BaseFusedOp` owns the operation, add the implementation there instead
|
||||||
|
of registering a second conflicting op/backend pair.
|
||||||
|
|
||||||
|
## Step 3 (optional): Tune JIT build flags
|
||||||
|
|
||||||
|
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
|
||||||
|
|
||||||
|
```python
|
||||||
|
return load_jit(
|
||||||
|
"scale",
|
||||||
|
*args,
|
||||||
|
cuda_files=["elementwise/scale.cuh"],
|
||||||
|
cuda_wrappers=[("scale", f"scale<{args}>")],
|
||||||
|
extra_cuda_cflags=["-O3", "--use_fast_math"],
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
If your kernel requires SM90+, raise a clear Python error before calling `load_jit`. Arch gating is one of the checks that has to live in Python — it decides whether to compile at all, so the C++ launcher never gets to run:
|
||||||
|
|
||||||
|
```python
|
||||||
|
if torch.cuda.get_device_capability()[0] < 9:
|
||||||
|
raise RuntimeError("This kernel requires SM90 (Hopper) or later")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 4: Write tests (required)
|
||||||
|
|
||||||
|
JIT kernel correctness tests and benchmarks live under `test/registered/kernels/ops/<group>/` and `test/registered/kernels/benchmark/<group>/`, mirroring the wrapper's group under `python/sglang/kernels/ops/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/kernels/jit/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
|
||||||
|
|
||||||
|
- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`.
|
||||||
|
- **Nightly kernel suite**: register with `stage="nightly"` plus the `runner_config` of the machine it needs (e.g. `1-gpu-large`), giving the `nightly-test-1-gpu-large` suite. `.github/workflows/nightly-test-nvidia.yml` sets `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` for the whole nightly run, so the expanded parameter grids apply automatically (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). There is no separate kernel-only nightly job: every nightly test on one machine type shares that machine's suite.
|
||||||
|
|
||||||
|
Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing):
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sglang.test.ci.ci_register import register_cuda_ci
|
||||||
|
|
||||||
|
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||||
|
# Optional B200/SM100 registration for tests that cover Blackwell-specific code paths
|
||||||
|
# register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
|
||||||
|
# Optional second registration: same file also runs nightly, same form,
|
||||||
|
# stage is just "nightly" there (and no `nightly=True`)
|
||||||
|
# register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
|
||||||
|
```
|
||||||
|
|
||||||
|
CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. Nightly uses the same shape with `stage="nightly"`; the single-string `suite=` form is left only for `stress` and non-CUDA pools.
|
||||||
|
|
||||||
|
Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery.
|
||||||
|
|
||||||
|
Use `register_cuda_ci(..., disabled="reason")` if the file must stay in-tree but should be skipped in CI (e.g. multi-GPU only).
|
||||||
|
|
||||||
|
**Run like CI** (from repo root):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-1-gpu-large)
|
||||||
|
# For B200/SM100-specific coverage:
|
||||||
|
(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-4-gpu-b200)
|
||||||
|
```
|
||||||
|
|
||||||
|
For fast iteration you can still run `pytest` on a single file locally; CI coverage is via `run_suite.py`.
|
||||||
|
|
||||||
|
Create `test/registered/kernels/ops/elementwise/test_scale.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
import torch
|
||||||
|
from sglang.kernels.ops.elementwise.scale import scale
|
||||||
|
from sglang.test.ci.ci_register import register_cuda_ci
|
||||||
|
|
||||||
|
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||||
|
@pytest.mark.parametrize("size", [1, 127, 128, 1024, 4097]) # cover tail remainder
|
||||||
|
@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0, 3.0])
|
||||||
|
def test_scale_correctness(dtype, size, factor):
|
||||||
|
src = torch.randn(size, dtype=dtype, device="cuda")
|
||||||
|
out = scale(src, factor)
|
||||||
|
expected = src * factor
|
||||||
|
|
||||||
|
rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
|
||||||
|
torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||||
|
def test_scale_out_param(dtype):
|
||||||
|
src = torch.randn(1024, dtype=dtype, device="cuda")
|
||||||
|
out = torch.empty_like(src)
|
||||||
|
result = scale(src, 2.0, out=out)
|
||||||
|
assert result is out
|
||||||
|
torch.testing.assert_close(out, src * 2.0, rtol=1e-2, atol=1e-2)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scale_cpu_error():
|
||||||
|
src = torch.randn(128, dtype=torch.float16) # CPU tensor
|
||||||
|
with pytest.raises(RuntimeError, match="CUDA"):
|
||||||
|
scale(src, 2.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scale_unsupported_dtype():
|
||||||
|
src = torch.randint(0, 10, (128,), dtype=torch.int32, device="cuda")
|
||||||
|
with pytest.raises(RuntimeError, match="dtype"):
|
||||||
|
scale(src, 2.0)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import sys
|
||||||
|
sys.exit(pytest.main([__file__, "-v", "-s"]))
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 5: Add a benchmark (required)
|
||||||
|
|
||||||
|
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
|
||||||
|
|
||||||
|
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
|
||||||
|
|
||||||
|
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
|
||||||
|
|
||||||
|
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
|
||||||
|
- **`@marker.parametrize(names, vals, ci_vals=None)`** — stackable decorator that adds a row axis (pytest-style). Each `@parametrize` adds one (or more, correlated) parameter the benchmark is swept over (Cartesian product across all `parametrize` decorators). `names` may be a single name (`"size"`) or a comma-separated correlated tuple axis (`"h,d"`, with `vals` then a list of tuples like `[(1, 64), (2, 128)]`). Pass the optional third `ci_vals` for a smaller sweep that is auto-selected under `is_in_ci()` — this is the built-in CI-shrinking mechanism, so you usually don't need `get_benchmark_range` for swept axes.
|
||||||
|
- **`marker.do_bench(fn, *, input_args=(), input_kwargs={}, ...)`** — runs `fn` under CUDA graph (default) or a naive loop, returns a `BenchResult`. Key knobs:
|
||||||
|
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
|
||||||
|
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
|
||||||
|
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
|
||||||
|
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"`; if you narrow it, keep every read, in-place, and output tensor.
|
||||||
|
- `use_cuda_graph=False` for kernels that can't be captured.
|
||||||
|
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
|
||||||
|
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
|
||||||
|
- **`utils.create_random(*shape)` / `utils.create_empty(*shape)`** — shorthand for `torch.randn` / `torch.empty` with `DEFAULT_DTYPE` (`bfloat16`) and `DEFAULT_DEVICE` (`"cuda"`). Override via the `dtype=` / `device=` kwargs.
|
||||||
|
- **`utils.get_benchmark_range(full_range, ci_range)`** — returns the smaller `ci_range` under CI (`is_in_ci()`), the `full_range` locally. Still available for the `benchmark(...)` column axis (which has no `ci_vals`); for `parametrize` row axes prefer the built-in `ci_vals` argument.
|
||||||
|
|
||||||
|
Create `test/registered/kernels/benchmark/elementwise/bench_scale.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
|
||||||
|
from sglang.kernels.jit.benchmark import marker
|
||||||
|
from sglang.kernels.jit.benchmark.utils import create_random
|
||||||
|
from sglang.kernels.ops.elementwise.scale import scale as jit_scale
|
||||||
|
from sglang.test.ci.ci_register import register_cuda_ci
|
||||||
|
|
||||||
|
register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
|
||||||
|
|
||||||
|
|
||||||
|
@torch.compile()
|
||||||
|
def torch_impl_scale(src: torch.Tensor, factor: float) -> torch.Tensor:
|
||||||
|
return src * factor
|
||||||
|
|
||||||
|
|
||||||
|
FN_MAP = {
|
||||||
|
"jit": jit_scale,
|
||||||
|
"torch": torch_impl_scale,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# `parametrize(name, full_vals, ci_vals)`: the 3rd arg is the smaller sweep
|
||||||
|
# auto-selected under CI; the full range runs locally.
|
||||||
|
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096, 65536]) # 1K .. 512K
|
||||||
|
@marker.benchmark("impl", ["jit", "torch"])
|
||||||
|
def benchmark(size: int, impl: str):
|
||||||
|
src = create_random(size)
|
||||||
|
factor = 2.0
|
||||||
|
return marker.do_bench(
|
||||||
|
FN_MAP[impl],
|
||||||
|
input_args=(src, factor),
|
||||||
|
# `src` is read -> clone it per iter to avoid L2 reuse; factor is a scalar.
|
||||||
|
graph_clone_args=(0,),
|
||||||
|
# Defaults already report bandwidth: memory_args="all" counts src,
|
||||||
|
# memory_output="out" counts the returned tensor -> bytes(src)+bytes(out).
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
benchmark.run()
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
|
||||||
|
- The `line_arg` name passed to `benchmark` (`"impl"` here) must match a parameter on `benchmark(...)`; same for every `parametrize` name (`"size"`).
|
||||||
|
- Stack `@parametrize` once per swept axis. The required `@marker.benchmark` is the **innermost** decorator (bottom of the stack, directly above the function) — `@parametrize` rows go above it.
|
||||||
|
- Prefer `create_random` / `create_empty` from `utils.py` over open-coding `torch.randn(..., dtype=..., device=...)`.
|
||||||
|
- The GB/s column appears by default (`memory_args="all"` + `memory_output="out"`). For memory-bound kernels it's the most informative number; scope `memory_args` / `memory_output` to the tensors actually touched if the defaults over- or under-count. For compute-bound kernels where bandwidth is misleading, set `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1` (or `disable_log_bandwidth=True`).
|
||||||
|
- For in-place kernels (which return `None`), pass the written tensors via `memory_output=(...)` since the `"out"` default would capture nothing.
|
||||||
|
- Tune `graph_clone_args` / `graph_clone_kwargs` to all the arguments that might be read by the kernel. We can only skip cloning for write-only args. For in-place modified args, we still need to clone them to get accurate timing (reusing the same buffer keeps it L2-hot and skews results).
|
||||||
|
- Call `benchmark.run()` (no `print_data=` kwarg — the marker framework prints directly).
|
||||||
|
|
||||||
|
Run locally:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python test/registered/kernels/benchmark/elementwise/bench_scale.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the benchmark suite the way CI does:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config`; starred args and non-literal values break AST collection
|
||||||
|
- **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations
|
||||||
|
- **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...`
|
||||||
|
- **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way.
|
||||||
|
- **Missing GB/s column**: the column is on by default; check that `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH` is not `1` and `disable_log_bandwidth` is not `True`. For in-place kernels (return `None`) the `memory_output="out"` default counts nothing — pass the written tensors via `memory_output=(...)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/docs/developer_guide/development_jit_kernel_guide.mdx`
|
||||||
|
- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
|
||||||
|
- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
|
||||||
|
- `python/sglang/kernels/jit/utils/compile/` — `load_jit` (`loader.py`), `make_cpp_args` (`cpp_args.py`)
|
||||||
|
- `python/sglang/kernels/jit/utils/common.py` — `cache_once`, `should_run_full_tests`, `get_ci_test_range`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`, `is_type`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/ffi.h` — `ffi::empty`, `ffi::empty_like`, `ffi::from_blob`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/utils.cuh` — type aliases, `LaunchKernel`, `SGL_DEVICE`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/vec.cuh` — `AlignedVector`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/tile.cuh` — `tile::Memory`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/bits.h` — compile-time bit helpers
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/type.cuh` — `DTypeTrait`, `packed_t`, `device::cast`, `device::unpack`, `ReductionTrait`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/math.cuh` — `device::math::`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/warp.cuh` — `warp::reduce<Op>` and `reduce_sum/max/min` wrappers
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
|
||||||
|
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
|
||||||
|
- `python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh` — minimal runnable reference
|
||||||
|
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
|
||||||
|
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
|
||||||
|
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
|
||||||
|
- `python/sglang/kernels/jit/benchmark/utils.py` — `create_random` / `create_empty` / `get_benchmark_range` helpers and `DEFAULT_DTYPE` / `DEFAULT_DEVICE`
|
||||||
|
- `test/registered/kernels/benchmark/layernorm/bench_qknorm.py` — real example: multi-axis `parametrize` (with `ci_vals`) + in-place `memory_output`
|
||||||
|
- `test/registered/kernels/benchmark/kvcache/bench_store_cache.py` — real example: scoped `memory_args` / `memory_output` + selective `graph_clone_args`
|
||||||
|
|
||||||
|
## Summary of Files Created
|
||||||
|
|
||||||
|
```
|
||||||
|
python/sglang/kernels/jit/csrc/elementwise/scale.cuh # NEW: CUDA kernel
|
||||||
|
python/sglang/kernels/ops/elementwise/scale.py # NEW: Python wrapper
|
||||||
|
test/registered/kernels/ops/elementwise/test_scale.py # NEW: Tests
|
||||||
|
test/registered/kernels/benchmark/elementwise/bench_scale.py # NEW: Benchmark
|
||||||
|
```
|
||||||
@@ -0,0 +1,370 @@
|
|||||||
|
---
|
||||||
|
name: add-sgl-kernel
|
||||||
|
description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel to sgl-kernel (including tests & benchmarks)
|
||||||
|
---
|
||||||
|
|
||||||
|
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
|
||||||
|
|
||||||
|
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||||
|
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||||
|
The implementation tutorial below does not replace that API contract.
|
||||||
|
|
||||||
|
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Add a new operation that scales each element of a tensor by a scalar factor:
|
||||||
|
|
||||||
|
- Input: tensor `x` (CUDA) and scalar `factor` (float)
|
||||||
|
- Output: `x * factor` (element-wise, in-place or into pre-allocated `out`)
|
||||||
|
- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
|
||||||
|
- Dispatched via `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro (defined in `python/sglang/kernels/aot/include/utils.h`)
|
||||||
|
|
||||||
|
## Two rules of thumb (must follow)
|
||||||
|
|
||||||
|
1. **Prefer `python/sglang/kernels/jit` first** when the kernel does **not** depend on CUTLASS or another large C++ project. This is the default path for lightweight kernels that benefit from rapid iteration.
|
||||||
|
2. **Prefer `sgl-kernel`** when the kernel **does** depend on CUTLASS or another large C++ project, or when it should be part of the AOT wheel / torch op registration flow.
|
||||||
|
3. **Exception**: if the dependency is `flashinfer`, or CUTLASS that is already provided through `flashinfer`, the kernel can still be implemented as `jit_kernel`.
|
||||||
|
|
||||||
|
In addition, every new kernel must ship with:
|
||||||
|
|
||||||
|
- **Tests** (pytest)
|
||||||
|
- **A benchmark script** (`marker.do_bench` from `sglang.kernels.jit.benchmark` — it works for any callable, not only JIT kernels)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repository integration map
|
||||||
|
|
||||||
|
You will typically touch these files/areas:
|
||||||
|
|
||||||
|
- Implementation: `python/sglang/kernels/aot/csrc/elementwise/scale.cu` (pick the right subdirectory)
|
||||||
|
- Public declarations: `python/sglang/kernels/aot/include/sgl_kernel_ops.h`
|
||||||
|
- Torch extension registration: `python/sglang/kernels/aot/csrc/common_extension.cc`
|
||||||
|
- Build: `python/sglang/kernels/aot/CMakeLists.txt` (`set(SOURCES ...)`)
|
||||||
|
- Python API: `python/sglang/kernels/aot/python/sgl_kernel/` and `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||||
|
- Tests: `python/sglang/kernels/aot/tests/test_scale.py`
|
||||||
|
- Benchmarks: `python/sglang/kernels/aot/benchmark/bench_scale.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1: Implement the kernel in `csrc/`
|
||||||
|
|
||||||
|
Pick the right subdirectory:
|
||||||
|
|
||||||
|
- `csrc/elementwise/` — for element-wise ops (our example)
|
||||||
|
- `csrc/gemm/`, `csrc/attention/`, `csrc/moe/` — for other categories
|
||||||
|
|
||||||
|
Create `python/sglang/kernels/aot/csrc/elementwise/scale.cu`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
#include <ATen/cuda/CUDAContext.h>
|
||||||
|
#include <c10/cuda/CUDAGuard.h>
|
||||||
|
#include <torch/all.h>
|
||||||
|
|
||||||
|
#include "utils.h" // DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16
|
||||||
|
|
||||||
|
// scale_kernel: out[i] = input[i] * factor
|
||||||
|
// Supports float, half (__half), __nv_bfloat16 via template T
|
||||||
|
template <typename T>
|
||||||
|
__global__ void scale_kernel(T* __restrict__ out,
|
||||||
|
const T* __restrict__ input,
|
||||||
|
float factor,
|
||||||
|
int64_t n) {
|
||||||
|
int64_t idx = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
|
||||||
|
if (idx < n) {
|
||||||
|
out[idx] = static_cast<T>(static_cast<float>(input[idx]) * factor);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void scale(at::Tensor& out, const at::Tensor& input, double factor) {
|
||||||
|
TORCH_CHECK(input.is_cuda(), "input must be a CUDA tensor");
|
||||||
|
TORCH_CHECK(input.is_contiguous(), "input must be contiguous");
|
||||||
|
TORCH_CHECK(out.is_cuda(), "out must be a CUDA tensor");
|
||||||
|
TORCH_CHECK(out.is_contiguous(), "out must be contiguous");
|
||||||
|
TORCH_CHECK(out.sizes() == input.sizes(), "out and input must have the same shape");
|
||||||
|
TORCH_CHECK(out.scalar_type() == input.scalar_type(),
|
||||||
|
"out and input must have the same dtype");
|
||||||
|
|
||||||
|
const int64_t n = input.numel();
|
||||||
|
const int threads = 256;
|
||||||
|
const int blocks = (n + threads - 1) / threads;
|
||||||
|
|
||||||
|
const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
|
||||||
|
const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
|
||||||
|
|
||||||
|
// Dispatches over float, float16, bfloat16
|
||||||
|
DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16(input.scalar_type(), c_type, [&] {
|
||||||
|
scale_kernel<c_type><<<blocks, threads, 0, stream>>>(
|
||||||
|
static_cast<c_type*>(out.data_ptr()),
|
||||||
|
static_cast<const c_type*>(input.data_ptr()),
|
||||||
|
static_cast<float>(factor),
|
||||||
|
n);
|
||||||
|
cudaError_t status = cudaGetLastError();
|
||||||
|
TORCH_CHECK(status == cudaSuccess,
|
||||||
|
"scale_kernel launch failed: ", cudaGetErrorString(status));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
|
||||||
|
- Use `at::Tensor` (PyTorch tensors), `TORCH_CHECK` for validation, `at::cuda::getCurrentCUDAStream()` for stream
|
||||||
|
- Keep Python wrappers thin; do shape/dtype/device validation in C++ right around the launch path
|
||||||
|
- `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` covers `float`, `half` (FP16), `__nv_bfloat16` (BF16)
|
||||||
|
- Add device error checking after every kernel launch
|
||||||
|
- If a kernel only works on certain architectures, enforce that with `TORCH_CHECK` and skip logic in tests
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2: Add a C++ declaration in `include/sgl_kernel_ops.h`
|
||||||
|
|
||||||
|
Edit `python/sglang/kernels/aot/include/sgl_kernel_ops.h`, add to the elementwise section:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
void scale(at::Tensor& out, const at::Tensor& input, double factor);
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 3: Register the op in `csrc/common_extension.cc`
|
||||||
|
|
||||||
|
Edit `python/sglang/kernels/aot/csrc/common_extension.cc`, inside `TORCH_LIBRARY_FRAGMENT(sgl_kernel, m)`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// From csrc/elementwise
|
||||||
|
m.def("scale(Tensor! out, Tensor input, float factor) -> ()");
|
||||||
|
m.impl("scale", torch::kCUDA, &scale);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
|
||||||
|
- `Tensor!` means in-place / mutable output argument
|
||||||
|
- The schema is important for `torch.compile` and for consistent call signatures
|
||||||
|
- Keep the torch schema in PyTorch scalar types (`float` here), but note that the C++ launcher signature still needs `double` for scalar arguments accepted by `torch::Library`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 4: Add the new source file to `CMakeLists.txt`
|
||||||
|
|
||||||
|
Edit `python/sglang/kernels/aot/CMakeLists.txt`, add to `set(SOURCES ...)`:
|
||||||
|
|
||||||
|
```cmake
|
||||||
|
csrc/elementwise/scale.cu
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
|
||||||
|
- Keep the list **alphabetically sorted** (the file explicitly requires this)
|
||||||
|
- If the kernel has arch constraints, reflect that in tests/benchmarks via skip logic
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 5: Expose a Python API under `python/sglang/kernels/aot/python/sgl_kernel/`
|
||||||
|
|
||||||
|
Prefer following the existing module organization first. For elementwise kernels, the usual pattern is:
|
||||||
|
|
||||||
|
- implement the Python wrapper in `python/sglang/kernels/aot/python/sgl_kernel/elementwise.py`
|
||||||
|
- then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||||
|
|
||||||
|
For example, in `python/sglang/kernels/aot/python/sgl_kernel/elementwise.py`, add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
|
||||||
|
def scale(
|
||||||
|
input: torch.Tensor,
|
||||||
|
factor: float,
|
||||||
|
out: torch.Tensor | None = None,
|
||||||
|
) -> torch.Tensor:
|
||||||
|
"""
|
||||||
|
Element-wise scale: out = input * factor.
|
||||||
|
|
||||||
|
Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
input : CUDA input tensor
|
||||||
|
factor : scale factor (float)
|
||||||
|
out : optional pre-allocated CUDA output tensor (same shape/dtype as input)
|
||||||
|
"""
|
||||||
|
if out is None:
|
||||||
|
out = torch.empty_like(input)
|
||||||
|
torch.ops.sgl_kernel.scale.default(out, input, factor)
|
||||||
|
return out
|
||||||
|
```
|
||||||
|
|
||||||
|
Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py` following the existing import style used by other kernels.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SGLang integration (required for runtime use)
|
||||||
|
|
||||||
|
After exposing the AOT wheel symbol, add a lazy wrapper and a `KernelSpec` under
|
||||||
|
`python/sglang/kernels/ops/<group>/`. Set `backend=KernelBackend.AOT`, record
|
||||||
|
actual device/architecture support, and keep `sgl_kernel` imports inside the
|
||||||
|
implementation path. SGLang runtime and integration tests import that wrapper.
|
||||||
|
For this example, use `sglang.kernels.ops.elementwise.scale`.
|
||||||
|
|
||||||
|
The wheel-level tests below validate its standalone API/build. They do not
|
||||||
|
replace CI-registered SGLang correctness tests in
|
||||||
|
`test/registered/kernels/ops/elementwise/test_scale.py` and benchmarks in
|
||||||
|
`test/registered/kernels/benchmark/elementwise/bench_scale.py`. Follow
|
||||||
|
`write-sglang-test` for their registration and CI budget; do not register tests
|
||||||
|
under the shipped `python/sglang/` package.
|
||||||
|
|
||||||
|
## Step 6: Write tests (required)
|
||||||
|
|
||||||
|
Create `python/sglang/kernels/aot/tests/test_scale.py`:
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import sgl_kernel
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
|
||||||
|
@pytest.mark.parametrize("size", [128, 1024, 4096, 65536])
|
||||||
|
@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0])
|
||||||
|
def test_scale_correctness(dtype, size, factor):
|
||||||
|
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||||
|
out = torch.empty_like(input)
|
||||||
|
|
||||||
|
result = sgl_kernel.scale(input, factor, out=out)
|
||||||
|
assert result is out
|
||||||
|
|
||||||
|
expected = input * factor
|
||||||
|
rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
|
||||||
|
torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scale_shape_mismatch():
|
||||||
|
input = torch.randn(128, dtype=torch.float16, device="cuda")
|
||||||
|
out = torch.empty(256, dtype=torch.float16, device="cuda")
|
||||||
|
with pytest.raises(RuntimeError, match="same shape"):
|
||||||
|
sgl_kernel.scale(input, 2.0, out=out)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scale_cpu_input():
|
||||||
|
input = torch.randn(128, dtype=torch.float16) # CPU
|
||||||
|
out = torch.empty_like(input)
|
||||||
|
with pytest.raises(RuntimeError, match="CUDA"):
|
||||||
|
sgl_kernel.scale(input, 2.0, out=out)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import sys
|
||||||
|
sys.exit(pytest.main([__file__, "-q"]))
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 7: Add a benchmark (required)
|
||||||
|
|
||||||
|
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
|
||||||
|
|
||||||
|
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
|
||||||
|
import sgl_kernel
|
||||||
|
from sglang.kernels.jit.benchmark import marker
|
||||||
|
|
||||||
|
|
||||||
|
def sglang_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
|
||||||
|
sgl_kernel.scale(input, factor, out=out)
|
||||||
|
|
||||||
|
|
||||||
|
def torch_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
|
||||||
|
torch.mul(input, factor, out=out)
|
||||||
|
|
||||||
|
|
||||||
|
FN_MAP = {"sglang": sglang_scale, "torch": torch_scale}
|
||||||
|
|
||||||
|
|
||||||
|
@marker.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32], [torch.float16])
|
||||||
|
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096]) # 1K .. 512K
|
||||||
|
@marker.benchmark("provider", ["sglang", "torch"])
|
||||||
|
def benchmark(dtype: torch.dtype, size: int, provider: str):
|
||||||
|
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||||
|
out = torch.empty_like(input)
|
||||||
|
return marker.do_bench(
|
||||||
|
FN_MAP[provider],
|
||||||
|
# Pass every tensor through input_args (not a closure) so marker rotates
|
||||||
|
# them across CUDA-graph calls to defeat L2 reuse.
|
||||||
|
input_args=(input, 2.0, out),
|
||||||
|
# Bandwidth = bytes(input) + bytes(out), both already in input_args.
|
||||||
|
memory_output=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
benchmark.run()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 8: Build
|
||||||
|
|
||||||
|
Build:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd python/sglang/kernels/aot
|
||||||
|
make build -j16
|
||||||
|
```
|
||||||
|
|
||||||
|
If you need to limit host resource usage:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd python/sglang/kernels/aot
|
||||||
|
make build -j1 MAX_JOBS=2 CMAKE_ARGS="-DSGL_KERNEL_COMPILE_THREADS=1"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 9: Validate
|
||||||
|
|
||||||
|
After building successfully, run the test and benchmark:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest python/sglang/kernels/aot/tests/test_scale.py -q
|
||||||
|
python python/sglang/kernels/aot/benchmark/bench_scale.py
|
||||||
|
```
|
||||||
|
|
||||||
|
PR CI also runs `pr-test-sgl-kernel.yml`, including the B200 job
|
||||||
|
`sgl-kernel-b200-test` when kernel changes are detected. Use that job as the
|
||||||
|
Blackwell coverage signal for AOT `sgl-kernel` changes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **Async CUDA errors**: `CUDA_LAUNCH_BLOCKING=1`
|
||||||
|
- **Memory errors**: `compute-sanitizer --tool memcheck python ...`
|
||||||
|
- **Build is too slow / OOM**: reduce `MAX_JOBS` and `SGL_KERNEL_COMPILE_THREADS`
|
||||||
|
- **Binary bloat**: use `python/sglang/kernels/aot/analyze_whl_kernel_sizes.py`
|
||||||
|
- **CMake sources list**: if your `.cu` file is missing from `SOURCES`, the symbol will be undefined at link time
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `python/sglang/kernels/aot/README.md`
|
||||||
|
- `python/sglang/kernels/aot/include/sgl_kernel_ops.h`
|
||||||
|
- `python/sglang/kernels/aot/csrc/common_extension.cc`
|
||||||
|
- `python/sglang/kernels/aot/CMakeLists.txt`
|
||||||
|
- `python/sglang/kernels/aot/include/utils.h` — `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro and friends
|
||||||
|
- `python/sglang/kernels/aot/csrc/elementwise/activation.cu` — reference for the FP16/BF16/FP32 dispatch pattern
|
||||||
|
|
||||||
|
## Summary of Files Created/Modified
|
||||||
|
|
||||||
|
```
|
||||||
|
python/sglang/kernels/aot/csrc/elementwise/scale.cu # NEW: CUDA kernel + launcher
|
||||||
|
python/sglang/kernels/aot/include/sgl_kernel_ops.h # MODIFIED: C++ declaration
|
||||||
|
python/sglang/kernels/aot/csrc/common_extension.cc # MODIFIED: schema + dispatch registration
|
||||||
|
python/sglang/kernels/aot/CMakeLists.txt # MODIFIED: add source file (alphabetical)
|
||||||
|
python/sglang/kernels/aot/python/sgl_kernel/elementwise.py # MODIFIED: Python wrapper
|
||||||
|
python/sglang/kernels/aot/python/sgl_kernel/__init__.py # MODIFIED: re-export Python API
|
||||||
|
python/sglang/kernels/aot/tests/test_scale.py # NEW: tests
|
||||||
|
python/sglang/kernels/aot/benchmark/bench_scale.py # NEW: benchmark
|
||||||
|
```
|
||||||
@@ -0,0 +1,414 @@
|
|||||||
|
---
|
||||||
|
name: ci-workflow-guide
|
||||||
|
description: Guide to SGLang CI workflow orchestration — stage ordering, fail-fast, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages.
|
||||||
|
---
|
||||||
|
|
||||||
|
# SGLang CI Workflow Orchestration Guide
|
||||||
|
|
||||||
|
This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and aborted on failure across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Naming Conventions
|
||||||
|
|
||||||
|
- **Suite**: `base-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `base-b-test-1-gpu-small`)
|
||||||
|
- **Test group**: Directory-level registered test group under `test/registered/` (e.g., `hicache` maps to `test/registered/hicache/test_*.py`)
|
||||||
|
- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Files
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|------|------|
|
||||||
|
| `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions |
|
||||||
|
| `.github/workflows/pr-test-extra.yml` | Extra workflow — gated by BOTH `run-ci` and `run-ci-extra` labels |
|
||||||
|
| `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting |
|
||||||
|
| `.github/actions/check-pr-test-health/action.yml` | Cross-job fail-fast: queries API for any failed job |
|
||||||
|
| `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete |
|
||||||
|
| `.github/actions/check-maintenance/action.yml` | Maintenance mode check |
|
||||||
|
| `.github/scripts/ci-labels.cjs` | Resolves the four CI control labels into dispatch axes |
|
||||||
|
| `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests |
|
||||||
|
| `python/sglang/test/ci/ci_register.py` | Test registration (AST-parsed markers), LPT auto-partition |
|
||||||
|
| `python/sglang/test/ci/ci_utils.py` | `run_unittest_files()`: execution, retry, continue-on-error |
|
||||||
|
| `scripts/ci/utils/slash_command_handler.py` | Handles slash commands from PR comments |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────┐
|
||||||
|
│ build kernel │
|
||||||
|
└──────┬───────┘
|
||||||
|
│
|
||||||
|
├─ check-changes ──── detects which packages changed
|
||||||
|
│ (main_package, sgl_kernel, jit_kernel, multimodal_gen)
|
||||||
|
│
|
||||||
|
├─ call-gate ──────── pr-gate.yml (draft? label? rate limit?)
|
||||||
|
│
|
||||||
|
├─────────────────────────────────────────────────────┐
|
||||||
|
│ │
|
||||||
|
▼ │
|
||||||
|
┌─────────────────────────────────────┐ │
|
||||||
|
│ Base A (~3 min) │ │
|
||||||
|
│ pre-flight check │ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌─────────────────────────────┐ │ │
|
||||||
|
│ │ base-a-test-1-gpu-small │ │ │
|
||||||
|
│ │ (small GPUs) │ │ │
|
||||||
|
│ └─────────────────────────────┘ │ │
|
||||||
|
│ ┌─────────────────────────────┐ │ │
|
||||||
|
│ │ base-a-test-cpu │ │ │
|
||||||
|
│ │ (CPU) │ │ │
|
||||||
|
│ └─────────────────────────────┘ │ │
|
||||||
|
└──────┬──────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌─────────────────────────────────────┐ ┌──────────────────────────┐
|
||||||
|
│ Base B (~30 min) │ │ kernel test │
|
||||||
|
│ base tests │ └──────────────────────────┘
|
||||||
|
│ │ ┌──────────────────────────┐
|
||||||
|
│ ┌─────────────────────────────┐ │ │ multimodal gen test │
|
||||||
|
│ │ base-b-test-1-gpu-small │ │ └──────────────────────────┘
|
||||||
|
│ │ (small GPUs, e.g. 5090) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ base-b-test-1-gpu-large │ │
|
||||||
|
│ │ (large GPUs, e.g. H100) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ base-b-test-2-gpu-large │ │
|
||||||
|
│ │ (large GPUs, e.g. H100) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
└──────┬──────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────┐
|
||||||
|
│ Base C (~30 min) │
|
||||||
|
│ advanced tests │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ base-c-test-4-gpu-h100 │ │
|
||||||
|
│ │ (H100 GPUs) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ base-c-test-8-gpu-h200 │ │
|
||||||
|
│ │ (8 x H200 GPUs) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ base-c-test-4-gpu-b200 │ │
|
||||||
|
│ │ (4 x B200 GPUs) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ Other advanced tests │ │
|
||||||
|
│ │ (DeepEP, PD Disagg, GB300) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
└──────┬──────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────┐
|
||||||
|
│ pr-test-finish │
|
||||||
|
│ aggregates all results, fails if │
|
||||||
|
│ any job failed/cancelled │
|
||||||
|
└─────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fails fast (red X) with a root cause annotation.
|
||||||
|
|
||||||
|
**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fail-fast is also disabled.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fail-Fast Layers
|
||||||
|
|
||||||
|
4 layers of fail-fast, from fine to coarse:
|
||||||
|
|
||||||
|
| Layer | Mechanism | Granularity | Disabled on schedule? |
|
||||||
|
|-------|-----------|-------------|----------------------|
|
||||||
|
| **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes |
|
||||||
|
| **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes |
|
||||||
|
| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fail-fast (red X) | Yes |
|
||||||
|
| **4. Stage → stage (cross-stage)** | `wait-for-base-*` + `needs` | Base A fails → base B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
|
||||||
|
|
||||||
|
- **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py`
|
||||||
|
- **Layer 2**: `--continue-on-error` flag in `run_suite.py` — off for PRs, on for scheduled runs
|
||||||
|
- **Layer 3**: `check-pr-test-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
|
||||||
|
- **Layer 4**: `wait-for-base-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Execution Modes
|
||||||
|
|
||||||
|
| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | Manual dispatch (`workflow_dispatch`) |
|
||||||
|
|--------|---------------------|------------------------------|--------------------------------------|
|
||||||
|
| **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only |
|
||||||
|
| **Cross-job fail-fast** | Yes (`check-pr-test-health`) | Yes | Yes |
|
||||||
|
| **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No |
|
||||||
|
| **Retry** | Enabled | Enabled | Enabled |
|
||||||
|
| **max_parallel** | 3 (default), 14 if `max-concurrency` label | 14 | 3 (default), 14 if `max-concurrency` |
|
||||||
|
| **PR gate** | Yes (draft, label, rate limit) | Skipped | Skipped |
|
||||||
|
| **Concurrency** | `cancel-in-progress: true` per PR | Queue (no cancel) | Isolated per stage+SHA |
|
||||||
|
|
||||||
|
Four labels relax these limits for one PR: `bypass-fail-fast`, `parallel-stages`,
|
||||||
|
`max-concurrency`, and `highest-priority` (all three). `.github/scripts/ci-labels.cjs`
|
||||||
|
resolves them; the [contribution guide](https://docs.sglang.io/developer_guide/contribution_guide.html#ci-control-labels)
|
||||||
|
describes what each one does.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage Gating (`wait-for-jobs` action)
|
||||||
|
|
||||||
|
`wait-for-base-a` and `wait-for-base-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
|
||||||
|
|
||||||
|
**How it works:**
|
||||||
|
1. Calls `listJobsForWorkflowRun` to list all jobs in the current run
|
||||||
|
2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `base-b-test-1-gpu-small (3)`)
|
||||||
|
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
|
||||||
|
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||||
|
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||||
|
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
|
||||||
|
|
||||||
|
**Job specs example** (base-b):
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{"prefix": "base-b-test-1-gpu-small", "expected_count": 8},
|
||||||
|
{"prefix": "base-b-test-1-gpu-large", "expected_count": 14},
|
||||||
|
{"prefix": "base-b-test-2-gpu-large", "expected_count": 4},
|
||||||
|
{"prefix": "base-b-test-4-gpu-b200", "expected_count": 1}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Critical**: `expected_count` must match the matrix size. If you add/remove matrix entries, update the wait job's spec accordingly.
|
||||||
|
|
||||||
|
**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and manual dispatches skip these entirely, allowing parallel execution.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cross-Job Fail-Fast (`check-pr-test-health` action)
|
||||||
|
|
||||||
|
Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`).
|
||||||
|
|
||||||
|
**How it works:**
|
||||||
|
1. Queries `listJobsForWorkflowRun` for the current workflow run
|
||||||
|
2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-pr-test-health` (excludes cascade failures)
|
||||||
|
3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names
|
||||||
|
4. If none → does nothing (step succeeds)
|
||||||
|
|
||||||
|
**Cascade filtering**: When job A fails fast due to the health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fail-fast. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list.
|
||||||
|
|
||||||
|
**Usage pattern:**
|
||||||
|
```yaml
|
||||||
|
steps:
|
||||||
|
- name: Checkout code
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
...
|
||||||
|
|
||||||
|
- uses: ./.github/actions/check-pr-test-health
|
||||||
|
id: pr-test-health
|
||||||
|
|
||||||
|
- name: Install dependencies # skipped automatically if health check failed
|
||||||
|
... # (default if: success() is false)
|
||||||
|
|
||||||
|
- name: Run test # also skipped
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Visual effect**: Job shows **red X** (failure) with error annotation showing root cause job names. Subsequent steps are naturally skipped (default `if: success()` is false after a failed step). No per-step `if` guards needed.
|
||||||
|
|
||||||
|
**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fail-fast.
|
||||||
|
|
||||||
|
**Error message example:**
|
||||||
|
```
|
||||||
|
Fail-fast: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Within-Suite Failure Handling
|
||||||
|
|
||||||
|
Controlled by `run_unittest_files()` in `python/sglang/test/ci/ci_utils.py`.
|
||||||
|
|
||||||
|
### Flags
|
||||||
|
|
||||||
|
| Flag | PR default | Scheduled default | Effect |
|
||||||
|
|------|------------|-------------------|--------|
|
||||||
|
| `--continue-on-error` | Off | On | Off: stop at first failure. On: run all files, report all failures at end |
|
||||||
|
| `--enable-retry` | On | On | Retry retriable failures (accuracy/perf assertions) |
|
||||||
|
| `--max-attempts` | 2 | 2 | Max attempts per file including initial run |
|
||||||
|
|
||||||
|
### Retry Classification
|
||||||
|
|
||||||
|
When a test fails and retry is enabled, the output is classified:
|
||||||
|
|
||||||
|
**Non-retriable** (checked first — real code errors):
|
||||||
|
`SyntaxError`, `ImportError`, `ModuleNotFoundError`, `NameError`, `TypeError`, `AttributeError`, `RuntimeError`, `CUDA out of memory`, `OOM`, `Segmentation fault`, `core dumped`, `ConnectionRefusedError`, `FileNotFoundError`
|
||||||
|
|
||||||
|
**Retriable** (accuracy/performance):
|
||||||
|
`AssertionError` with comparison patterns (`not greater than`, `not less than`, `not equal to`), `accuracy`, `score`, `latency`, `throughput`, `timeout`
|
||||||
|
|
||||||
|
**Default**: Unknown `AssertionError` → retriable. Other unknown failures → not retriable.
|
||||||
|
|
||||||
|
### How `continue_on_error` is set
|
||||||
|
|
||||||
|
In `pr-test.yml`'s `check-changes` job:
|
||||||
|
- `schedule` runs or `run_all_tests` flag → `continue_on_error = 'true'`
|
||||||
|
- PR runs → `continue_on_error = 'false'`
|
||||||
|
|
||||||
|
Each test job propagates via:
|
||||||
|
```yaml
|
||||||
|
env:
|
||||||
|
CONTINUE_ON_ERROR_FLAG: ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||||
|
run: |
|
||||||
|
python3 run_suite.py --hw cuda --suite <name> $CONTINUE_ON_ERROR_FLAG
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test Partitioning
|
||||||
|
|
||||||
|
Large suites are split across matrix jobs using the **LPT (Longest Processing Time) heuristic** in `ci_register.py:auto_partition()`:
|
||||||
|
|
||||||
|
1. Sort tests by `est_time` descending, filename as tie-breaker (deterministic)
|
||||||
|
2. Greedily assign each test to the partition with smallest cumulative time
|
||||||
|
3. Result: roughly equal total time per partition
|
||||||
|
|
||||||
|
**Partition table** (CUDA per-commit suites):
|
||||||
|
|
||||||
|
| Suite | Partitions | Runner | max_parallel |
|
||||||
|
|-------|-----------|--------|-------------|
|
||||||
|
| `base-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
|
||||||
|
| `base-a-test-cpu` | 4 | `ubuntu-latest` | — |
|
||||||
|
| `base-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
|
||||||
|
| `base-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
|
||||||
|
| `base-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
|
||||||
|
| `base-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||||
|
| `base-b-kernel-unit-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||||
|
| `base-b-kernel-unit-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||||
|
| `base-b-kernel-unit-test-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||||
|
| `base-b-kernel-benchmark-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||||
|
| `base-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
|
||||||
|
| `base-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
|
||||||
|
| `base-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
|
||||||
|
| `base-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
|
||||||
|
| `base-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
|
||||||
|
| `base-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
|
||||||
|
| `base-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
|
||||||
|
| `base-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
|
||||||
|
|
||||||
|
> **Suite names are generated**, not hand-written: each comes from a test's `register_*_ci(stage=..., runner_config=...)` as `{stage}-test-{runner_config}`, and `runner_config` maps to the `Runner` column via `scripts/ci/runner_configs.yml`.
|
||||||
|
>
|
||||||
|
> **Note**: Kernel suites (`base-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `base-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||||
|
|
||||||
|
**Workflow usage:**
|
||||||
|
```yaml
|
||||||
|
strategy:
|
||||||
|
matrix:
|
||||||
|
partition: [0, 1, 2, 3, 4, 5, 6, 7]
|
||||||
|
steps:
|
||||||
|
- run: python3 run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
|
||||||
|
--auto-partition-id ${{ matrix.partition }} --auto-partition-size 8
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## check-changes Job
|
||||||
|
|
||||||
|
Determines which test suites to run based on file changes.
|
||||||
|
|
||||||
|
### Detection Methods
|
||||||
|
|
||||||
|
| Trigger | Method | Details |
|
||||||
|
|---------|--------|---------|
|
||||||
|
| `pull_request` | `dorny/paths-filter` | Detects changes via GitHub diff |
|
||||||
|
| `workflow_dispatch` (with `pr_head_sha`) | GitHub API | `repos/{repo}/compare/main...{sha}` |
|
||||||
|
| `schedule` / `run_all_tests` | Force all true | Runs everything |
|
||||||
|
|
||||||
|
### Output Flags
|
||||||
|
|
||||||
|
| Output | Triggers |
|
||||||
|
|--------|----------|
|
||||||
|
| `main_package` | Base A/B/C test suites |
|
||||||
|
| `sgl_kernel` | Kernel wheel builds + kernel test suites; also switches B200 jobs to kernel-build runner labels outside `target_stage` mode |
|
||||||
|
| `jit_kernel` | JIT kernel test workflow |
|
||||||
|
| `multimodal_gen` | Multimodal-gen test workflow |
|
||||||
|
|
||||||
|
> **Note**: In `target_stage` mode, `sgl_kernel` is only active when `include_wheel_build=true`. Without that opt-in, kernel-change reruns fail validation instead of running a target stage without freshly built wheels. Outside `target_stage`, `sgl_kernel=true` switches B200 jobs from `4-gpu-b200` / `4-gpu-b200-low-disk` to `4-gpu-b200-kernel` / `4-gpu-b200-kernel-low-disk`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Concurrency Control
|
||||||
|
|
||||||
|
```
|
||||||
|
group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Segment | Source | Purpose |
|
||||||
|
|---------|--------|---------|
|
||||||
|
| `event_name` | `github.event_name` | Prevents scheduled runs colliding with fork PRs named `main` |
|
||||||
|
| `branch` | `github.head_ref \|\| github.ref_name` | Per-branch isolation |
|
||||||
|
| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates manual dispatches from main runs |
|
||||||
|
| `stage` | `inputs.target_stage \|\| 'all'` | Allows parallel stage dispatches |
|
||||||
|
|
||||||
|
`cancel-in-progress: true` for `pull_request` events (new push cancels old run), `false` for `workflow_call`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How To: Add a New Stage Job
|
||||||
|
|
||||||
|
1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-base-X, ...]`
|
||||||
|
2. Copy the `if:` condition pattern from an existing same-stage job (handles `target_stage`, `schedule`, `main_package`)
|
||||||
|
3. Add `checkout` step
|
||||||
|
4. Add `check-pr-test-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
|
||||||
|
5. Add `check-maintenance` step
|
||||||
|
6. Add `download-artifact` step if `sgl_kernel` changed
|
||||||
|
7. Add `install dependencies` step
|
||||||
|
8. Add `run test` step with `$CONTINUE_ON_ERROR_FLAG`
|
||||||
|
9. Add `upload-cuda-coredumps` step with `if: always()`
|
||||||
|
10. Register the suite name in `PER_COMMIT_SUITES` in `test/run_suite.py`
|
||||||
|
11. If using matrix, add `--auto-partition-id` and `--auto-partition-size` to the run command
|
||||||
|
12. **Update `wait-for-base-X`** job spec with the new job name and `expected_count` (if matrix)
|
||||||
|
13. **Add the job to `pr-test-finish.needs`** list
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How To: Debug CI Failures
|
||||||
|
|
||||||
|
| Symptom | Likely cause | What to check |
|
||||||
|
|---------|-------------|---------------|
|
||||||
|
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||||
|
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||||
|
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
|
||||||
|
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||||
|
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||||
|
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slash Commands
|
||||||
|
|
||||||
|
| Command | Effect |
|
||||||
|
|---------|--------|
|
||||||
|
| `/tag-run-ci-label` | Adds `run-ci` label to PR |
|
||||||
|
| `/tag-run-ci-label extra` | Adds both `run-ci` and `run-ci-extra` labels |
|
||||||
|
| `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run |
|
||||||
|
| `/tag-and-rerun-ci` | Adds `run-ci` label + reruns failed |
|
||||||
|
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||||
|
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
|
||||||
|
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||||
|
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
|
||||||
|
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||||
|
| `/run-cpu-ci` | Dispatches `run-cpu-ci.yml` on main: only the `base-a-test-cpu` suite, at the PR head SHA, no caches saved. Same gate as `/rerun-test`. Alias: `/rerun-cpu-ci` |
|
||||||
|
|
||||||
|
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||||
|
|
||||||
|
### Label-gated workflow dispatch (pr-test, pr-test-extra)
|
||||||
|
|
||||||
|
`pr-test.yml` and `pr-test-extra.yml` both listen for `pull_request.labeled` (in addition to `opened`/`synchronize`/`reopened`). The `check-changes.if` gate has two clauses:
|
||||||
|
|
||||||
|
1. **For `labeled` events**: the just-added label must be one of the gating labels (`run-ci` for pr-test, `run-ci` or `run-ci-extra` for pr-test-extra) — otherwise every unrelated label addition would dispatch a full CI run.
|
||||||
|
2. **All events**: the PR must currently carry the required labels.
|
||||||
|
|
||||||
|
This is what lets `/tag-run-ci-label` (and the `extra` variant) trigger a fresh CI run without an extra push.
|
||||||
|
|
||||||
|
**Caveat — skipped runs cannot be un-skipped by `run.rerun()`:** GitHub's rerun API reuses the original event payload, so rerunning a `pull_request`-event run that was skipped because of missing labels will skip again (label set in the frozen payload doesn't update). The only way to recover a label-skipped run is to add the missing label, which fires a fresh `labeled` event with the current label set. `handle_rerun_failed_ci` in the slash handler is for rerunning failed/non-label-skipped runs; it cannot revive label-skipped ones.
|
||||||
+1
-1
@@ -16,7 +16,7 @@ These are investigation hints, not a list of changes to apply.
|
|||||||
| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. |
|
| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. |
|
||||||
| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. |
|
| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. |
|
||||||
| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. |
|
| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. |
|
||||||
| Repeated template detection messages | `managers/template_detection.py` and `managers/template_manager.py` | Check whether a summary already contains the same information. |
|
| Repeated template detection messages | `parser/template_detection.py` and `parser/template_manager.py` | Check whether a summary already contains the same information. |
|
||||||
| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. |
|
| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. |
|
||||||
| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. |
|
| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. |
|
||||||
| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. |
|
| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. |
|
||||||
@@ -0,0 +1,268 @@
|
|||||||
|
---
|
||||||
|
name: cookbook-add-model
|
||||||
|
description: Add a new model to the SGLang Cookbook (docs/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the homepage vendor card. Interactive, multi-phase. Run with /cookbook-add-model.
|
||||||
|
disable-model-invocation: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Add a model to the SGLang Cookbook
|
||||||
|
|
||||||
|
> Migrating an **existing legacy-template page** (one that imports a monolithic
|
||||||
|
> `…/autoregressive/<slug>-deployment.jsx` generator)? Use the
|
||||||
|
> `cookbook-migrate-model` skill instead — same target format, but the legacy
|
||||||
|
> page (not the user) is the source of truth.
|
||||||
|
|
||||||
|
The cookbook is **config-driven**: two shared engines contain NO model-specific code —
|
||||||
|
`docs/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
|
||||||
|
`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
|
||||||
|
a per-model `config` (+ optional `benchmarks`) consumed by both engines, plus an MDX page
|
||||||
|
that imports them. No engine edits.
|
||||||
|
|
||||||
|
**Instantiate the model-agnostic template** (NOT a clone of any live cookbook — the
|
||||||
|
template is decoupled and covers all hardware + all axes):
|
||||||
|
- `templates/config.jsx.tmpl` → `docs/src/snippets/configs/<hf-org>/<model-slug>.jsx`
|
||||||
|
- `templates/benchmarks.jsx.tmpl` → `…/<model-slug>-benchmarks.jsx` (skip if no numbers)
|
||||||
|
- `templates/page.mdx.tmpl` → `docs/cookbook/<category>/<Vendor>/<ModelName>.mdx`
|
||||||
|
|
||||||
|
The template uses explicit `__TOKEN__` placeholders; you fill them, prune what the model
|
||||||
|
lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a populated
|
||||||
|
*instance* you can consult, but is not the template.
|
||||||
|
|
||||||
|
**Deep references (read on demand, don't inline):**
|
||||||
|
- [references/authoring-reference.md](references/authoring-reference.md) — field-by-field config / cells / playground / MDX contract.
|
||||||
|
- [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose.
|
||||||
|
- [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work).
|
||||||
|
- [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`).
|
||||||
|
- [references/diffusion-authoring.md](references/diffusion-authoring.md) — required opening, tag, command-picker, and feature-overlay contract for diffusion model pages. Read before editing any `docs/cookbook/diffusion/<Vendor>/<Model>.mdx` page.
|
||||||
|
|
||||||
|
## Architecture at a glance
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ cookbook/<category>/<Vendor>/<Model>.mdx │
|
||||||
|
│ import { Deployment } from "/src/snippets/_deployment.jsx"; │
|
||||||
|
│ import { Playground } from "/src/snippets/_playground.jsx"; │
|
||||||
|
│ import { config } from "/src/snippets/configs/.../X.jsx"; │
|
||||||
|
│ <Deployment config={config} /> <Playground config={config} />│
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
│ (config passed as React prop)
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ src/snippets/configs/<vendor>/<model>.jsx │
|
||||||
|
│ export const config = { │
|
||||||
|
│ supportedHardware, variants, quantizations, strategies, ... │
|
||||||
|
│ cells: [ { match:{hw,variant,quant,strategy,nodes}, │
|
||||||
|
│ env:[...], flags:[...] }, ... ], // 5-dim matrix │
|
||||||
|
│ playgroundFeatures: { attention, moe, parsers, ... }, │
|
||||||
|
│ }; │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
│ (consumed by BOTH engines — no model code in engines)
|
||||||
|
▼
|
||||||
|
┌──────────────────────────────────┬──────────────────────────────┐
|
||||||
|
│ _deployment.jsx │ _playground.jsx │
|
||||||
|
│ Renders the verified matrix; │ Renders override chips + │
|
||||||
|
│ one cell → its env/flags. │ diff against the cell. │
|
||||||
|
└──────────────────────────────────┴──────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
The two widgets stay in sync via: the **URL hash** (deploy mirrors its selection;
|
||||||
|
playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatches on
|
||||||
|
every change; playground listens — `replaceState` doesn't fire `hashchange`), and the
|
||||||
|
shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders).
|
||||||
|
|
||||||
|
> The main template is **autoregressive**. Diffusion pages use
|
||||||
|
> `templates/diffusion-page.mdx.tmpl` plus
|
||||||
|
> [references/diffusion-authoring.md](references/diffusion-authoring.md); do not force the
|
||||||
|
> autoregressive deployment matrix on them. Omni pages follow their own category structure.
|
||||||
|
> All categories still obey the Mintlify / NEW-tag / docs.json / category-card / validation
|
||||||
|
> rules below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Interactive, multi-step workflow. Collect inputs incrementally — don't ask for
|
||||||
|
everything upfront.** The real work is the verified `cells[]` recipes + measured
|
||||||
|
benchmarks (Phases 2 + 4); everything else is filling the template.
|
||||||
|
|
||||||
|
## Phase 1 — Collect inputs
|
||||||
|
|
||||||
|
1. **Model card** — HuggingFace repo/URL. **Fetch the page** and extract description,
|
||||||
|
param count, architecture, context length, **license**. (Fetching guards factual bugs
|
||||||
|
like an off-by-a-few-B param count.) If the model isn't public, ask the user.
|
||||||
|
2. **Variants / quantizations** — keep separate: variants are size/mode (e.g. Flash/Pro,
|
||||||
|
Instruct/Thinking); quantizations come from the HF card / linked repos (BF16/FP8/FP4/…).
|
||||||
|
Default to BF16 when a full-precision repo exists.
|
||||||
|
3. **Tested hardware + parallelism** — which platforms are actually tested, and TP/EP/DP
|
||||||
|
for each. List only tested hw (unlisted greys out).
|
||||||
|
4. **Verified launch recipes** — the full `sglang serve` flags per
|
||||||
|
(hw × variant × quant × strategy × nodes) combo → these become `cells[]`. Rewrite any
|
||||||
|
`python -m sglang.launch_server` to `sglang serve` form.
|
||||||
|
5. **sglang version / image tag** — ask which sglang build the recipes + benchmarks ran on
|
||||||
|
(a release like `0.5.x`, or `main`/nightly). **Never guess or hallucinate it.** This one
|
||||||
|
tag fills `dockerImages` and the benchmarks' `sglang_version`; when the user is unsure,
|
||||||
|
default the image to `lmsysorg/sglang:dev` (nightly) rather than inventing a release.
|
||||||
|
6. **Pre-flight**: `gh pr list --repo sgl-project/sglang --search "<model>"` (dup check).
|
||||||
|
|
||||||
|
**Hardware reference** (the shared `HARDWARE_CATALOG` in `_deployment.jsx`). A GPU **not** in
|
||||||
|
this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.hardware`
|
||||||
|
(`{id,label,vram,vendor}`) — the engine merges it in; don't edit the engine catalog:
|
||||||
|
|
||||||
|
| Platform | Vendor | VRAM | Docker image |
|
||||||
|
|---|---|---|---|
|
||||||
|
| H100 | NVIDIA | 80GB | `lmsysorg/sglang:<ver>` |
|
||||||
|
| H200 | NVIDIA | 141GB | `lmsysorg/sglang:<ver>` |
|
||||||
|
| B200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` |
|
||||||
|
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
|
||||||
|
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||||
|
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||||
|
| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:<ver>` — needs a **CUDA 13** build |
|
||||||
|
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||||
|
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||||
|
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||||
|
| MI355X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||||
|
|
||||||
|
- **Image tag (`<ver>`)**: don't guess — ask the user for the tag the recipes ran on, or
|
||||||
|
default to `dev` (nightly). The same tag goes in `dockerImages` and benchmarks'
|
||||||
|
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
|
||||||
|
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
|
||||||
|
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
|
||||||
|
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300
|
||||||
|
single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so
|
||||||
|
its only multi-GPU topology is TP=2 across 2 nodes.
|
||||||
|
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
|
||||||
|
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
|
||||||
|
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
|
||||||
|
- **EP** (MoE): 8-GPU NVIDIA `--tp 8 --ep 8`; AMD `EP = TP`; small NVIDIA (TP≤4) omit
|
||||||
|
`--ep` unless benchmarked. (The template's AMD example cell shows these.)
|
||||||
|
|
||||||
|
## Phase 2 — Instantiate the template
|
||||||
|
|
||||||
|
For a diffusion model, instantiate `templates/diffusion-page.mdx.tmpl` and keep the
|
||||||
|
shared `DiffusionModelTags` component, plus `templates/diffusion-config.jsx.tmpl` for the
|
||||||
|
opt-in scoped command builder. Put the compact install command and builder in §1 Quick
|
||||||
|
start. The first two paragraphs in §2 Model capabilities are not generic filler: they must
|
||||||
|
state the model's capability range, strongest differentiator, when to choose it, and at
|
||||||
|
least one real deployment or capability boundary. Put orthogonal runtime features in
|
||||||
|
`scope: "serve"` or `scope: "request"`, not in the base recipe; use the schema from
|
||||||
|
`references/diffusion-authoring.md`.
|
||||||
|
|
||||||
|
1. **Copy** the three template files to their target paths (above). Note the two
|
||||||
|
vendor-folder conventions: under `configs/` the folder is the **HuggingFace org**
|
||||||
|
(`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`).
|
||||||
|
2. **Replace every `__TOKEN__`**: `__MODEL_DISPLAY__`, `__MODEL_SLUG__`, `__HF_ORG__`,
|
||||||
|
`__HF_REPO__`, `__REASONING_PARSER__`, `__TOOLCALL_PARSER__`, `__ONE_LINER__`. Verify
|
||||||
|
none remain: `grep -rn '__[A-Z_]*__' <new files>`.
|
||||||
|
3. **Prune** to what the model supports (delete, don't stub) — using
|
||||||
|
[references/authoring-reference.md](references/authoring-reference.md):
|
||||||
|
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
|
||||||
|
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
|
||||||
|
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
|
||||||
|
- `playgroundFeatures` axes: remove the `megamoe` backend option + the
|
||||||
|
`megamoeQuant` block from the `moe` axis (non-Blackwell-MoE), delete `hisparse`
|
||||||
|
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
|
||||||
|
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
|
||||||
|
to single `default` if there's no variant axis (then drop the `variant` half of
|
||||||
|
`modelNames`/`defaultAccuracy` keys).
|
||||||
|
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||||
|
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||||
|
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||||
|
a guessed release; key by `hw`, or a more specific key — `hw|variant|quant`, `variant|quant`,
|
||||||
|
`hw|quant|strategy`, `hw|quant`, resolved in that order before `hw` — when one
|
||||||
|
variant/quant/strategy needs its own image), `multiNodeHints` only for fabric-specific hw
|
||||||
|
(e.g. gb200).
|
||||||
|
|
||||||
|
5. **Diffusion pages: add the ComfyUI section.** Every diffusion cookbook page ends with
|
||||||
|
a `## <n>. Run in ComfyUI` section so a reader never has to guess whether the model is
|
||||||
|
reachable from ComfyUI. It is one component; the per-model facts live in the component,
|
||||||
|
not the page:
|
||||||
|
|
||||||
|
```mdx
|
||||||
|
## <n>. Run in ComfyUI
|
||||||
|
|
||||||
|
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||||
|
|
||||||
|
<ComfyUISupport model="<key>" />
|
||||||
|
```
|
||||||
|
|
||||||
|
Pick `model` from the table in
|
||||||
|
`docs/src/snippets/diffusion/comfyui-support.jsx`. Use the model's own key when it has
|
||||||
|
an entry (its executor or dedicated node differs); otherwise use the generic `image` or
|
||||||
|
`video`. A model gets its own key only when the plugin actually treats it specially —
|
||||||
|
an entry in `executor_class_dict`
|
||||||
|
(`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a
|
||||||
|
dedicated node. Adding a key without the matching plugin support makes the page lie.
|
||||||
|
|
||||||
|
### Site-wiring (do all three)
|
||||||
|
|
||||||
|
- **`docs/docs.json`** — add the page under Cookbook → `<category>` → `<Vendor>`, at
|
||||||
|
the **top** of that vendor's `pages` (root-relative, no `.mdx`:
|
||||||
|
`cookbook/<category>/<Vendor>/<Model>`). New vendor group → insert in the section's
|
||||||
|
local ordering.
|
||||||
|
- **NEW-tag hygiene** — the new page keeps `tag: NEW` (from the template). Scan the
|
||||||
|
vendor dir for existing NEW and strip it from siblings; verify ≤1:
|
||||||
|
`grep -rn 'tag: NEW' docs/cookbook/<category>/<Vendor>/` → at most one result. (Scan
|
||||||
|
files; don't assume the first `docs.json` entry holds NEW.)
|
||||||
|
- **Homepage card** — `docs/cookbook/<category>/intro.mdx`: if the org already has a
|
||||||
|
`<Card>`, update only its `href` (keep `img`). If the org is **new**, add a `<Card>`
|
||||||
|
(title = nav-group name; keep card order aligned with `docs.json`) **and create its logo**:
|
||||||
|
ask the user for the brand logo, then generate the conforming **icon-only 940×525 RGBA
|
||||||
|
transparent** PNG → `docs/cards/logos/<org-slug>.png` per
|
||||||
|
[references/vendor-logo.md](references/vendor-logo.md) (track with `git add -f` — `*.png`
|
||||||
|
is gitignored repo-wide). Never invent or copy a logo.
|
||||||
|
|
||||||
|
## Phase 3 — Validate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd docs
|
||||||
|
mint validate # frontmatter, missing nav entries, MDX/JSX errors
|
||||||
|
mint broken-links
|
||||||
|
mint dev # visual smoke test at http://localhost:3000/cookbook/<category>/<Vendor>/<Model>
|
||||||
|
```
|
||||||
|
|
||||||
|
Spot-check: cells render sensible commands; URL-hash nav persists across reload; the
|
||||||
|
Playground inherits the Deploy selection live; each axis toggle produces the expected
|
||||||
|
diff; Docker mode wraps in `docker run` with the right image; multi-node cells emit the
|
||||||
|
hints + `--nnodes N`; cURL resolves the model name; the NEW badge shows on the new page
|
||||||
|
and is gone from same-vendor siblings; the homepage card points to the new model.
|
||||||
|
|
||||||
|
## Phase 4 — Interactive testing
|
||||||
|
|
||||||
|
The user deploys each cell, runs the benches, and pastes results; you fill the data:
|
||||||
|
- mark each tested `cells[]` entry `verified: true` (absent = yellow/unverified badge);
|
||||||
|
- fill the `<model>-benchmarks.jsx` entries (one per cell `match`) with measured
|
||||||
|
speed/accuracy + the `sglang_version` the user reports (don't invent one — the template's
|
||||||
|
`0.0.0` is a deliberate TODO); set model-level `defaultAccuracy` per variant. Leave a
|
||||||
|
cell's entry as a bare `match` stub if it has no numbers yet (the card shows "pending").
|
||||||
|
|
||||||
|
## Phase 5 — Prose & config tips
|
||||||
|
|
||||||
|
**Read [references/mintlify-authoring.md](references/mintlify-authoring.md) first** (it
|
||||||
|
carries the parser-output-shape / thinking-mode / Output-Example / no-hardcoded-sampling
|
||||||
|
rules + the Mintlify forbidden-syntax list). Then rewrite the MDX prose from the HF card +
|
||||||
|
user notes: §1 Model Introduction (description, links, params, license, variants table),
|
||||||
|
§2 Configuration Tips (hw-specific tuning, caveats), §3 Advanced Usage (Reasoning /
|
||||||
|
Tool-Calling / HiCache — keep only what applies; match the reasoning example to the
|
||||||
|
parser's output shape; each runnable block gets an `**Output Example:**`).
|
||||||
|
|
||||||
|
For diffusion pages, follow the category-specific Quick start and capability contract in
|
||||||
|
`references/diffusion-authoring.md`. Run `node docs/scripts/check_cookbook_configs.mjs` to
|
||||||
|
verify the tag widget and introduction structure before rendering the page.
|
||||||
|
|
||||||
|
## Phase 6 — Review
|
||||||
|
|
||||||
|
```
|
||||||
|
/cookbook-review-pr <PR number>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Git workflow
|
||||||
|
|
||||||
|
Always branch — never commit to main directly.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git checkout -b add-<model>-cookbook
|
||||||
|
git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||||
|
docs/src/snippets/configs/<hf-org>/<slug>-benchmarks.jsx \
|
||||||
|
docs/cookbook/<category>/<Vendor>/<Model>.mdx \
|
||||||
|
docs/docs.json docs/cookbook/<category>/intro.mdx
|
||||||
|
git commit -m "Add <Display-Name> cookbook"
|
||||||
|
git push -u origin add-<model>-cookbook
|
||||||
|
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
|
||||||
|
```
|
||||||
+3
-3
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
|
|||||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
then parallelism (`--tp`, `--attn-dp-size`), then MoE
|
||||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||||
engine assumes this ordering when inserting overrides (its anchors target
|
engine assumes this ordering when inserting overrides (its anchors target
|
||||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||||
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
|
|||||||
|
|
||||||
| Axis key | Widget | Use when |
|
| Axis key | Widget | Use when |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||||
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
|
|||||||
knobs/values that are disabled under the live facts, so stale picks never
|
knobs/values that are disabled under the live facts, so stale picks never
|
||||||
emit a blocked combination. Interleave prefill-CP + DP-Attention is
|
emit a blocked combination. Interleave prefill-CP + DP-Attention is
|
||||||
deliberately NOT grayed (combined support is planned upstream even though
|
deliberately NOT grayed (combined support is planned upstream even though
|
||||||
current releases assert `dp_size == 1` for interleave) — the engine shows
|
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
|
||||||
a warning hint below the command box instead.
|
a warning hint below the command box instead.
|
||||||
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
|
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
|
||||||
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
|
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
|
||||||
+3
-5
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
|
|||||||
playgroundFeatures: {
|
playgroundFeatures: {
|
||||||
|
|
||||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
|
||||||
attention: {
|
attention: {
|
||||||
knobs: [
|
knobs: [
|
||||||
{ id: "tp", label: "TP", values: [
|
{ id: "tp", label: "TP", values: [
|
||||||
@@ -357,8 +357,7 @@ sgl-eval run gsm8k \\
|
|||||||
"--trust-remote-code",
|
"--trust-remote-code",
|
||||||
"--model-path {{MODEL_NAME}}",
|
"--model-path {{MODEL_NAME}}",
|
||||||
"--tp 4",
|
"--tp 4",
|
||||||
"--dp 4",
|
"--attn-dp-size 4",
|
||||||
"--enable-dp-attention",
|
|
||||||
"--moe-a2a-backend deepep",
|
"--moe-a2a-backend deepep",
|
||||||
"--host {{HOST_IP}}",
|
"--host {{HOST_IP}}",
|
||||||
"--port {{PORT}}",
|
"--port {{PORT}}",
|
||||||
@@ -403,8 +402,7 @@ sgl-eval run gsm8k \\
|
|||||||
"--trust-remote-code",
|
"--trust-remote-code",
|
||||||
"--model-path {{MODEL_NAME}}",
|
"--model-path {{MODEL_NAME}}",
|
||||||
"--tp 16",
|
"--tp 16",
|
||||||
"--dp 16",
|
"--attn-dp-size 16",
|
||||||
"--enable-dp-attention",
|
|
||||||
"--moe-a2a-backend deepep",
|
"--moe-a2a-backend deepep",
|
||||||
"--host {{HOST_IP}}",
|
"--host {{HOST_IP}}",
|
||||||
"--port {{PORT}}",
|
"--port {{PORT}}",
|
||||||
+2
-2
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
|
|||||||
|
|
||||||
## 1. Quick start
|
## 1. Quick start
|
||||||
|
|
||||||
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
|
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
|
||||||
a verified recipe below.
|
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
|
||||||
|
|
||||||
<Deployment config={config} />
|
<Deployment config={config} />
|
||||||
|
|
||||||
@@ -0,0 +1,241 @@
|
|||||||
|
---
|
||||||
|
name: cookbook-migrate-model
|
||||||
|
description: Migrate a legacy-template SGLang cookbook page (monolithic per-model generator under docs/src/snippets/autoregressive/) onto the config-driven template (shared _deployment.jsx / _playground.jsx engines + per-model config). Use when asked to migrate, convert, or port an existing cookbook page — NOT for brand-new models (use cookbook-add-model for those). Run with /cookbook-migrate-model <Model page name, e.g. GLM-5.1>.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Cookbook Migrate Model
|
||||||
|
|
||||||
|
Convert one legacy cookbook page to the config-driven format, faithfully. The
|
||||||
|
legacy page — its generator widget and its measured benchmark blocks — is the
|
||||||
|
**single source of truth**. You are transcribing it into the new data model,
|
||||||
|
not improving it.
|
||||||
|
|
||||||
|
Reuses the `cookbook-add-model` skill's assets (read them on demand):
|
||||||
|
- `../cookbook-add-model/templates/config.jsx.tmpl`, `page.mdx.tmpl`, `benchmarks.jsx.tmpl`
|
||||||
|
- `../cookbook-add-model/references/authoring-reference.md` (config/cells/playground contract)
|
||||||
|
- `../cookbook-add-model/references/mintlify-authoring.md` (MDX rules)
|
||||||
|
|
||||||
|
Migration-specific references in this skill:
|
||||||
|
- [references/dimension-mapping.md](references/dimension-mapping.md) — legacy-control → new-dimension mapping rules, command rewrite table, per-family strategy sets, and the Qwen3.5 pilot as a worked example (PR #27848).
|
||||||
|
|
||||||
|
The round's per-model inventory (scope, batch order, quirks, measured-data
|
||||||
|
survey) is tracked by the migration maintainer outside the repo — expect it in
|
||||||
|
your dispatch prompt, or ask for it.
|
||||||
|
|
||||||
|
## Hard rules (non-negotiable)
|
||||||
|
|
||||||
|
1. **Never modernize.** Env vars, flags, TP values, docker tags, version strings
|
||||||
|
are copied verbatim from the legacy page — even when today's defaults differ
|
||||||
|
(e.g. a flag whose behavior has since become the default; keep it anyway). The recipe
|
||||||
|
that was verified is the recipe as written. Allowed normalizations are ONLY
|
||||||
|
the five alias rewrites in dimension-mapping.md §2 (`launch_server`→`sglang
|
||||||
|
serve`, `--model`→`--model-path`, `--tp-size`→`--tp`, abbreviated
|
||||||
|
`--speculative-algo`→`--speculative-algorithm`,
|
||||||
|
`--expert-parallel-size`→`--ep`). **Accuracy-degrading flags**
|
||||||
|
(`--kv-cache-dtype fp8_e4m3`, W4A4-style runtime quant) follow a
|
||||||
|
deterministic rule — enforced in migration, no asking: offered as a
|
||||||
|
legacy **selectable option** → never select it (cells mirror the
|
||||||
|
accuracy-safe side), and the option **survives as a Playground axis** —
|
||||||
|
an existing one where it fits, else add one via a separate prior engine
|
||||||
|
PR (rule 4); a legacy choice never degrades to a tips mention. Baked
|
||||||
|
into the recipe's **default/unconditional command** → keep it verbatim
|
||||||
|
(the recipe was measured with it, and fp8 KV halves KV memory —
|
||||||
|
stripping could OOM it). See dimension-mapping.md §2 caveats.
|
||||||
|
2. **Never invent versions or numbers.** Benchmark numbers only from the
|
||||||
|
legacy page's measured blocks, and a result migrates ONLY when its
|
||||||
|
`sglang_version` is a **reproducible anchor** — the bar is reproducibility,
|
||||||
|
not "must be a release":
|
||||||
|
- ✅ release tag/version (`v0.5.9` / `0.5.9`), commit hash, OR — for
|
||||||
|
**Day-0 support** (the enabling PR isn't merged and no release is cut
|
||||||
|
yet) — a specific **PR (`PR #27944`) or commit** you can `gh pr checkout`
|
||||||
|
/ `git checkout <sha>`. Commit is most precise; a PR pin is fine for day-0.
|
||||||
|
- ❌ a moving ref — `"main branch"`, `"main (2026-06-11)"`, open-ended
|
||||||
|
`"0.5.8+"` — is NOT reproducible: **drop the WHOLE result (speed AND
|
||||||
|
accuracy)**, not just speed. Keep `benchmarkCommands` so ⚡Reproduce still
|
||||||
|
guides re-measurement against a pinned build.
|
||||||
|
**Never inherit cross-model numbers** — measurements the legacy page
|
||||||
|
attributes to a *different model* (e.g. a K2.6 page carrying K2.5-measured
|
||||||
|
speed) are dropped regardless of version. When kept, `sglang_version` is the
|
||||||
|
legacy page's string verbatim. Docker tags only the ones the legacy page
|
||||||
|
pinned (unmapped hw falls back to `:dev`).
|
||||||
|
3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the
|
||||||
|
legacy page has concrete measured data for that exact 5-dim combo AND
|
||||||
|
(b) the cell's flags equal the deployment command used for that measurement
|
||||||
|
— modulo `{{HOST_IP}}`/`{{PORT}}`, the five alias rewrites, and **parser
|
||||||
|
flags**: `--reasoning-parser`/`--tool-call-parser` are stripped from every
|
||||||
|
cell (Playground-only feature; when the measured run had them on, say so in
|
||||||
|
the benchmarks file header). When the measured command diverges from the
|
||||||
|
generator default, **the verified cell follows the measured command**; the
|
||||||
|
generator default stays as the sibling strategy/cell or a tips note. Everything else is unverified (yellow) —
|
||||||
|
including combos that look memory-infeasible; keep them verbatim and list
|
||||||
|
them in the PR body for the re-verification track.
|
||||||
|
4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not
|
||||||
|
change in a migration PR. Model-specific features are config DATA consumed
|
||||||
|
by generic axis handlers (MegaMoE precedent), so they need NO engine change.
|
||||||
|
A **titled single-select that strips a flag family** — KV Cache DType
|
||||||
|
(`--kv-cache-dtype`), mamba (`--mamba-radix-cache-strategy`), … — is already
|
||||||
|
covered by the merged generic **`flagSelects`** axis: declare it in the
|
||||||
|
config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5
|
||||||
|
mamba example), **no engine PR**. Only a genuinely new control *shape* that
|
||||||
|
`flagSelects` can't express would need a one-time generic primitive (never a
|
||||||
|
model-named handler) on a separate prior PR (engine-axis.md).
|
||||||
|
5. **`github.cookbookModel` must be set** (`<hf-org>/<page-slug>`, e.g.
|
||||||
|
`qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels
|
||||||
|
as deepseek-v4. The issue template itself needs NO edits (free-form input).
|
||||||
|
6. **Strategy tiers are signal-driven.** A cell goes under `low-latency` /
|
||||||
|
`high-throughput` ONLY on a signal present in the legacy source (an
|
||||||
|
explicit performance toggle, a named recipe, or prose stating the
|
||||||
|
operating point); **no signal → `balanced`**. Never derive a slant from
|
||||||
|
your own hardware intuition — re-tiering on measured evidence is the
|
||||||
|
hardware owner's follow-up PR, not part of a migration
|
||||||
|
(dimension-mapping.md §4).
|
||||||
|
|
||||||
|
## Workflow (one model = one PR)
|
||||||
|
|
||||||
|
### 1. Inventory the legacy assets
|
||||||
|
- Read the legacy generator (`docs/src/snippets/autoregressive/<slug>-deployment.jsx`)
|
||||||
|
end-to-end: every option dimension (radio vs checkbox vs dynamic), every
|
||||||
|
gate/SUPPORT matrix, the full emitted command per reachable combo (env
|
||||||
|
prefixes, `# Error` pseudo-commands included).
|
||||||
|
- Read the legacy MDX: §2 install docker tags → `dockerImages` (pinned, not
|
||||||
|
upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
|
||||||
|
Output Examples verbatim); §5 benchmark blocks → transcribe each measured
|
||||||
|
block: deploy command used, bench command (dataset/isl/osl/num-prompts/
|
||||||
|
concurrency), P50 (median) TTFT/TPOT, output tok/s, hardware, version string.
|
||||||
|
- Inbound-anchor sweep: `grep -rn "<PageName>" docs/ --include='*.mdx'` —
|
||||||
|
find links/`#fragments` into this page (`mint broken-links` does NOT check
|
||||||
|
fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
|
||||||
|
- Check the maintainer-provided inventory notes for this model's known quirks —
|
||||||
|
but treat them (and the §4 family table) as a **survey snapshot**: re-verify
|
||||||
|
every dimension against the live legacy files before mapping. Pages keep
|
||||||
|
receiving updates (precedent: Kimi-K2.6's live generator has a speculative
|
||||||
|
toggle the 2026-06-10 survey notes lack).
|
||||||
|
|
||||||
|
### 2. Design the 5-dim mapping
|
||||||
|
Apply [references/dimension-mapping.md](references/dimension-mapping.md). Key
|
||||||
|
decision — which legacy toggle becomes the `strategies` dimension: a toggle
|
||||||
|
that **changes other parts of the command** (TP, mem) must (the Playground
|
||||||
|
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||||
|
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||||
|
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||||
|
even when its flags are uncoupled (`--attn-dp-size N` is a pure
|
||||||
|
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||||
|
Playground axis with the flags baked into cells when the legacy default was
|
||||||
|
ON — EXCEPT parsers:
|
||||||
|
`--reasoning-parser`/`--tool-call-parser` are NEVER baked into cells, they are
|
||||||
|
Playground-only (DSv4 convention). **Every legacy control survives as an
|
||||||
|
interactive control** — a dimension or a Playground axis, never a tips-only
|
||||||
|
mention — and a model-specific control is **config data, not engine code**
|
||||||
|
(MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure
|
||||||
|
config whenever it fits an existing axis's data schema. A **titled
|
||||||
|
single-select that strips a flag family** (Nemotron3's "KV Cache DType",
|
||||||
|
mamba `--mamba-radix-cache-strategy`, …) fits the merged generic **`flagSelects`**
|
||||||
|
axis — so it too is config-only (declare a `flagSelects` list). Only a control
|
||||||
|
whose *shape* `flagSelects` still can't express would need a ONE-TIME generic
|
||||||
|
primitive (never a model-named handler) on a separate PRIOR engine PR, keeping
|
||||||
|
the migration PR data-only (hard rule 4, engine-axis.md). The strategy count follows the page's
|
||||||
|
operating points: **one recipe → a single `balanced`; two → `low-latency` +
|
||||||
|
`high-throughput`; three → the full trio (the ideal)**. The tiers apply per
|
||||||
|
(hw × variant × quant) combination — a single-recipe combination on a
|
||||||
|
multi-strategy page parks under its semantically honest tier (no
|
||||||
|
latency/throughput slant → `balanced`; the page's list is the union). When the
|
||||||
|
legacy toggle is MTP / speculative decoding, the direction is a deterministic
|
||||||
|
default — apply without asking: **MTP on → `low-latency`, MTP off →
|
||||||
|
`high-throughput`** (reversed only with maintainer confirmation). Tier
|
||||||
|
placement is signal-driven (hard rule 6). Never invent a recipe just to fill
|
||||||
|
strategy chips (see dimension-mapping.md §4). Record the outcome as a
|
||||||
|
**strategy mapping table** for the PR body — one row per group of
|
||||||
|
combinations sharing the same legacy signal (e.g. "all GPU combos: MTP
|
||||||
|
toggle → low-latency / high-throughput"; "xeon: (none) → balanced"), with a
|
||||||
|
one-line rationale each; don't enumerate 60 identical rows. The table is what
|
||||||
|
hardware owners sign off on at review.
|
||||||
|
|
||||||
|
### 3. Generate the config (codegen, then audit)
|
||||||
|
- For >~30 cells, port the legacy `generateCommand()` into a throwaway Node
|
||||||
|
script that enumerates combos and emits the `cells:[...]` literal (output
|
||||||
|
must stay a pure literal — Mintlify forbids runtime spreads/calls). Apply the
|
||||||
|
verified-cell override in the script. See the pilot scripts embedded in the
|
||||||
|
worked example of dimension-mapping.md §5.
|
||||||
|
- **Independent equivalence audit (required):** extract the ORIGINAL generator
|
||||||
|
from git (`git show main:<path>` — NOT `HEAD:`, the migration branch deletes
|
||||||
|
the file, see dimension-mapping.md §5 item 7), stub React hooks, run it for
|
||||||
|
every combo, and diff token-by-token against the new cells. Expected deltas
|
||||||
|
only: the appended `--host {{HOST_IP}}`/`--port {{PORT}}`, the
|
||||||
|
engine-injected multi-node trio, the §2 alias rewrites (the entrypoint
|
||||||
|
rewrite doesn't appear in cell tokens — cells hold flags only; the audit
|
||||||
|
script normalizes it on the legacy side), and the intentional verified-cell
|
||||||
|
override. Paste the PASS count + the audit script in the PR body
|
||||||
|
(collapsed `<details>`).
|
||||||
|
- Hand-author the non-cells fields per authoring-reference.md. Structural
|
||||||
|
self-checks: every cell resolves a `modelNames` key; no `--nnodes/--node-rank/
|
||||||
|
--dist-init-addr/--host/--port` literals; every `{{KEY}}` declared; every
|
||||||
|
`supportedHardware` id has ≥1 cell.
|
||||||
|
|
||||||
|
### 4. Benchmarks file
|
||||||
|
One entry per measured block only (cells without entries already render
|
||||||
|
"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
|
||||||
|
total (in+out) tok/s/GPU = `output tok/s ÷ (tp × nnodes) ×
|
||||||
|
(isl+osl)/osl` — stored directly (the card shows it as-is). TTFT/TPOT
|
||||||
|
take the P50 (median) rows; set `config.latencyPercentile` (default `"P50"`; use
|
||||||
|
`"Mean"` only for legacy Mean-recorded data — temporary, being migrated to P50; an
|
||||||
|
entry-level `latencyPercentile` overrides the page value per cell).
|
||||||
|
Put the workload's
|
||||||
|
`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
|
||||||
|
the benchmarks carry accuracy data** — the engine ships no default eval set
|
||||||
|
(#27842), so missing labels means the accuracy rows silently don't render;
|
||||||
|
extra context (sample counts, suites that don't fit) goes in the entry's
|
||||||
|
`notes`. Zero-measured-data pages: skip the file and the `benchmarks` prop
|
||||||
|
entirely, but keep `benchmarkCommands` so ⚡Reproduce still guides users.
|
||||||
|
|
||||||
|
### 5. Rewrite the MDX
|
||||||
|
From `page.mdx.tmpl`: keep the original `title` (nav identity), write a fresh
|
||||||
|
SEO `description` (top-level — delete any legacy `metatags.description`), **no
|
||||||
|
`tag: NEW`** (a migration is not a launch), **no `mode:`**. Install accordion
|
||||||
|
carries the legacy install content + pinned images. Keep the template's
|
||||||
|
DSv4-style strategy bullets — serving semantics first (single-user chat /
|
||||||
|
typical multi-user / batch throughput), trimmed to the strategies the page
|
||||||
|
ships, plus at most a one-line note on what each strategy changes on this
|
||||||
|
model; do NOT rewrite them as toggle-/migration-centric explanations.
|
||||||
|
Legacy §5 benchmark prose is deleted (numbers → benchmark card, commands →
|
||||||
|
⚡Reproduce); legacy prose deploy commands are deleted (doc↔config parity —
|
||||||
|
fold their unique flags into §2 tips). Invocation examples + real outputs carry
|
||||||
|
over verbatim, but **wrapped in Accordions** — §3 commands and outputs are
|
||||||
|
collapsible (required, DeepSeek-V4 pattern): code in an
|
||||||
|
`<Accordion title="… (Python)">`, output in a following
|
||||||
|
`<Accordion title="Example Output">`; legacy pages kept them inline.
|
||||||
|
|
||||||
|
### 6. Delete the legacy generator
|
||||||
|
Remove `docs/src/snippets/autoregressive/<slug>-deployment.jsx` and its
|
||||||
|
import. `grep -rn "<slug>-deployment" docs/` must return nothing (config
|
||||||
|
provenance comments must not name the deleted path). Site wiring needs **no
|
||||||
|
changes**: docs.json path/title unchanged, vendor card + logo already exist.
|
||||||
|
|
||||||
|
### 7. Validate
|
||||||
|
- `grep -rn '__[A-Z_]*__'` on the new files (no template tokens).
|
||||||
|
- `cd docs && mint validate && mint broken-links` (pre-existing breaks on
|
||||||
|
main are not yours — say so in the PR).
|
||||||
|
- `mint dev` browser smoke: initial selection = the verified cell (first in
|
||||||
|
`cells[]`) with green badge; multi-node cells show the injected trio +
|
||||||
|
header; AMD cells show env prefixes; Docker mode wraps with the pinned image
|
||||||
|
and passes cell env as `--env`; condition-hidden combos grey out; benchmark
|
||||||
|
card values; NO parser flags in any Deploy command; Playground parser
|
||||||
|
toggles ADD the parser flags (green additions) while spec toggles strike
|
||||||
|
the baked spec flags (red); Submit ↗ prefills this model. **Probe pitfall:** drive at most
|
||||||
|
ONE programmatic click per evaluation and wait for React to settle —
|
||||||
|
multiple clicks in one synchronous script batch and read stale DOM.
|
||||||
|
- Token-level audit from step 3 passes.
|
||||||
|
|
||||||
|
### 8. PR + review
|
||||||
|
One PR per model. PR body: migration framing, verified policy applied, the
|
||||||
|
strategy mapping table (step 2), the audit PASS count + script, any
|
||||||
|
inherited-infeasible combos flagged for re-verification. Then run `/cookbook-review-pr <N>` and fix findings.
|
||||||
|
FYI: docs previews only build for in-repo (`sgl-project/sglang`) branches —
|
||||||
|
a fork-headed PR is perfectly fine but renders no preview; a maintainer can
|
||||||
|
re-push the branch in-repo if a preview is wanted for review.
|
||||||
|
|
||||||
|
### 9. Keep this skill current
|
||||||
|
Any new convention, engine behavior, or pitfall you discover while migrating
|
||||||
|
(naming decisions, audit-script gotchas, review-rule conflicts, …) MUST be fed
|
||||||
|
back into this skill — same PR if it's skill-file-only, or an immediate
|
||||||
|
follow-up commit on the skill's branch/PR. The next agent runs on what's
|
||||||
|
written here, not on your session's context.
|
||||||
+2
-2
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
|
|||||||
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
||||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
||||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
||||||
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
|
|||||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||||
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
||||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||||
|
|
||||||
Caveats discovered in the pilot:
|
Caveats discovered in the pilot:
|
||||||
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
||||||
@@ -0,0 +1,268 @@
|
|||||||
|
---
|
||||||
|
name: cookbook-review-pr
|
||||||
|
description: Review a pull request against the SGLang Cookbook (docs/, Mintlify) contribution checklist — the config-driven format (per-model config + benchmarks JSX consumed by the shared _deployment.jsx / _playground.jsx engines). Run with /cookbook-review-pr <PR number>.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Cookbook Review PR
|
||||||
|
|
||||||
|
Fetch the diff, run the checklist, report what you find. The cookbook is **config-driven**:
|
||||||
|
shared engines (`_deployment.jsx`, `_playground.jsx`) with NO model-specific code; each
|
||||||
|
model is a data `config` (+ optional `benchmarks`) under `src/snippets/configs/<vendor>/`
|
||||||
|
plus an MDX page. This checklist targets that layout. Field-schema detail lives in
|
||||||
|
`.claude/skills/cookbook-add-model/references/authoring-reference.md` — defer to it rather
|
||||||
|
than restating.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```
|
||||||
|
/cookbook-review-pr <PR number>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. `gh pr view <N> --repo sgl-project/sglang --json title,body,files,author,baseRefName,headRefName,commits,reviews`
|
||||||
|
2. `gh pr diff <N> --repo sgl-project/sglang`
|
||||||
|
3. `gh pr list --repo sgl-project/sglang --state open --search "<model name>"` (duplicate check)
|
||||||
|
4. Run every checklist item against the diff.
|
||||||
|
5. Output per-file verdicts + overall recommendation.
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
### 1. File hygiene
|
||||||
|
- A cookbook PR should only touch: `docs/src/snippets/configs/<vendor>/*.jsx`
|
||||||
|
(config + benchmarks), `docs/cookbook/**/*.mdx`, `docs/docs.json`,
|
||||||
|
`docs/cookbook/<category>/intro.mdx` (vendor card), `docs/cards/logos/<vendor>.png`
|
||||||
|
(new vendor only). Flag stray files (`settings.local.json`, lockfiles, IDE configs).
|
||||||
|
- Pages must be `.mdx`, not `.md`. Files end with a trailing newline. Check commit history
|
||||||
|
for unrelated commits accidentally included.
|
||||||
|
- **Engines untouched**: `_deployment.jsx` / `_playground.jsx` should NOT change in a
|
||||||
|
model-add PR (adding a model is data-only). Engine edits = a separate axis/feature PR
|
||||||
|
(see `cookbook-add-model/references/engine-axis.md`); review them against that checklist.
|
||||||
|
|
||||||
|
### 2. Config quality (the per-model `config`)
|
||||||
|
- Single `export const config = { ... }` literal — **no** function calls, spreads,
|
||||||
|
fragment refs, or IIFE (Mintlify re-evals at hydration → `ReferenceError`).
|
||||||
|
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
|
||||||
|
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
|
||||||
|
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
|
||||||
|
(`{id,label,vram,vendor}`), **not** added to the engine catalog — unless the platform is
|
||||||
|
genuinely shared, which is an engine change the PR body has to argue (and the
|
||||||
|
`cookbook-add-model` hardware table moves with it). A new catalog entry is inert for
|
||||||
|
other pages: each filters the catalog by its own `supportedHardware`.
|
||||||
|
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||||
|
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||||
|
- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
|
||||||
|
2 → `low-latency` + `high-throughput`; 3 → the full trio. Tiers apply per
|
||||||
|
(hw × variant × quant) combination: a single-recipe combination must park under its
|
||||||
|
semantically honest tier (clear slant → that tier, e.g. a workstation card under
|
||||||
|
`low-latency`; no slant → `balanced`, e.g. a CPU platform) — **flag a no-slant recipe
|
||||||
|
parked under low-latency/high-throughput**. Mixed unions
|
||||||
|
like [low-latency, balanced, high-throughput] with per-selection greying are fine. Also
|
||||||
|
flag model-specific ids (e.g. `mtp`), and flag an INVERTED speculative mapping — the
|
||||||
|
deterministic default is MTP/spec-decoding ON → `low-latency`, OFF → `high-throughput`
|
||||||
|
(at saturation the draft+verify overhead outweighs the speedup); the reverse needs an
|
||||||
|
explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics
|
||||||
|
in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal
|
||||||
|
toggles.
|
||||||
|
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback); a
|
||||||
|
`hw|quant` key (resolved before the plain `hw`) is valid when one quant on a shared GPU needs
|
||||||
|
a different image (e.g. an FP4 dev build) — don't flag those.
|
||||||
|
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||||
|
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||||
|
Hints render above BOTH run modes, so flag one that reads as docker-only ("add these
|
||||||
|
docker run flags") — `docker run` flags belong in the hardware entry's
|
||||||
|
`multiNodeDockerFlags`, which the engine emits into the Docker command itself.
|
||||||
|
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
|
||||||
|
template's `model` field is a free-form input prefilled from this value; if the config
|
||||||
|
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
|
||||||
|
page's submissions get mislabeled.
|
||||||
|
- `playgroundFeatures` is opt-OUT: the **general axes ship on every cookbook by default**
|
||||||
|
(`attention` TP/CP/DP-Attn, `moe` backend+EP for MoE models, `parsers`, `speculative`,
|
||||||
|
`pdDisagg`, `hicache`) — flag a missing general axis unless the model genuinely cannot
|
||||||
|
use it. Model-specific axes only where applicable (MegaMoE backend + `megamoeQuant`
|
||||||
|
only on Blackwell MoE, gated by `requiresHw`; `hisparse` only DSA-style). Knobs that are
|
||||||
|
meaningless for a subset of variants/hw are `disable`d with a reason, not silently live
|
||||||
|
(e.g. MoE knobs greyed on dense variants). No empty/stub axes.
|
||||||
|
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
|
||||||
|
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
|
||||||
|
nothing).
|
||||||
|
- **All-hardware considered**: every `supportedHardware` id (from the catalog or `config.hardware`) has ≥1 cell OR is a deliberate
|
||||||
|
greyed "coming soon"; AMD was pruned or kept on purpose (not a leftover template family).
|
||||||
|
|
||||||
|
### 3. Cells / 5-dim matrix
|
||||||
|
- Every cell `match` has EXACTLY the 5 keys (`hw`, `variant`, `quant`, `strategy`, `nodes`).
|
||||||
|
- `env` / `flags` are flat literals (only `{{PLACEHOLDER}}` subst) — no shared
|
||||||
|
`commonFlags` reference (Mintlify won't inline it).
|
||||||
|
- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
|
||||||
|
(the renderer injects them from `match.nodes`).
|
||||||
|
- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
|
||||||
|
- NO `--reasoning-parser` / `--tool-call-parser` in any cell — parsers are a
|
||||||
|
Playground-only feature added on top of the base command (DSv4 convention);
|
||||||
|
flag any cell that bakes them in.
|
||||||
|
- Accuracy-degrading flags in cells — runtime quant below the checkpoint
|
||||||
|
(e.g. MegaMoE W4A4 — DSv4 gates it behind the Playground's `megamoeQuant`)
|
||||||
|
and lossy `--kv-cache-dtype` (e.g. `fp8_e4m3` over a higher-precision-KV
|
||||||
|
checkpoint): **flag for explicit maintainer confirmation**. Output quality
|
||||||
|
should be exactly what the quant chip declares, so absent a recorded
|
||||||
|
sign-off in the PR (e.g. carried verbatim from a measured legacy recipe's
|
||||||
|
default command), request the flag move to Playground/tips.
|
||||||
|
- Flag order: `--model-path` first (an optional `--trust-remote-code` may precede it —
|
||||||
|
the DSv4 cells do), then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||||
|
last (the playground's insert anchors assume this).
|
||||||
|
- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
|
||||||
|
(BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
|
||||||
|
|
||||||
|
### 4. Benchmarks
|
||||||
|
- Each `benchmarks[]` entry's `match` tuple corresponds to a real cell.
|
||||||
|
- `accuracyLabels` is present whenever the benchmarks carry accuracy data — the engine
|
||||||
|
ships NO default eval set; without it the accuracy rows silently don't render.
|
||||||
|
`defaultAccuracy` / per-cell `accuracy` / `benchmarkCommands.accuracy` keys all
|
||||||
|
∈ `config.accuracyLabels`.
|
||||||
|
- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
|
||||||
|
that only released FP8/FP4 is a factual bug.
|
||||||
|
- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
|
||||||
|
from the `sglang serve` deploy command, and should carry `--flush-cache`: bench_serving's
|
||||||
|
`random` prompts are deterministic, so a warm rerun hits the radix cache and inflates
|
||||||
|
throughput — speed numbers are measured cache-cold.
|
||||||
|
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||||
|
guessed/placeholder value (no leftover `0.0.0`).
|
||||||
|
- **Latency percentile**: `config.latencyPercentile` (default `"P50"`, or `"Mean"`) matches the
|
||||||
|
percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. A benchmarks
|
||||||
|
entry may carry its own `latencyPercentile` to override the page value per cell
|
||||||
|
(entry → config → `"P50"`): on a P50 page, kept legacy Mean cells must set it — a
|
||||||
|
`sglang_version` tag alone doesn't convey the percentile. (`"Mean"` is temporary — legacy
|
||||||
|
data is being re-measured to P50.)
|
||||||
|
- **Throughput convention**: `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU**
|
||||||
|
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
|
||||||
|
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||||
|
column must be produced by the SAME harness — flag a page that, say, measures one
|
||||||
|
platform's GSM8K with a historical raw completion harness and another's with
|
||||||
|
`sgl-eval run gsm8k --num-examples 1319` and shows both as one "GSM8K %"
|
||||||
|
(the scores aren't comparable). Either standardize on one harness (matching
|
||||||
|
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
|
||||||
|
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
|
||||||
|
|
||||||
|
### 5. Doc ↔ config parity (the #1 finding)
|
||||||
|
- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
|
||||||
|
equal what the engine emits from the corresponding cell — same flags, same order. Drift
|
||||||
|
here is the most common review miss.
|
||||||
|
|
||||||
|
### 5b. ComfyUI section (diffusion pages)
|
||||||
|
- A diffusion page ends with `## <n>. Run in ComfyUI` rendering `<ComfyUISupport />`. A
|
||||||
|
reader must not have to guess whether the model is reachable from ComfyUI.
|
||||||
|
- The `model` prop is a key that exists in `docs/src/snippets/diffusion/comfyui-support.jsx`.
|
||||||
|
A model-specific key is only correct when the plugin really treats it specially — an entry
|
||||||
|
in `executor_class_dict`
|
||||||
|
(`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a dedicated
|
||||||
|
node. Otherwise the generic `image` / `video` key is the honest one; a model-specific key
|
||||||
|
without matching plugin support makes the page claim support that does not exist.
|
||||||
|
- Prose describing ComfyUI support inline instead of using the component is a finding: the
|
||||||
|
facts drift from the plugin.
|
||||||
|
|
||||||
|
### 6. Commands / port
|
||||||
|
- Launch uses `sglang serve` — flag any `python -m sglang.launch_server` /
|
||||||
|
`python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`;
|
||||||
|
guard against prose/cells reintroducing the old launcher.
|
||||||
|
- Port `30000` everywhere (launch, curl, client `base_url`, bench) — flag `8000`.
|
||||||
|
Launch port must match client/curl port on the same page.
|
||||||
|
|
||||||
|
### 7. Frontmatter
|
||||||
|
- Every new MDX page has `title:` and a **top-level** `description:` (a real one-line value
|
||||||
|
prop, not copied from another vendor) — NOT `metatags.description` (non-canonical; the
|
||||||
|
top-level field is what renders as the subtitle and SEO meta — see mintlify-authoring).
|
||||||
|
- **No `mode: wide` on a model page** — it hides the right-hand "On this page" ToC that every
|
||||||
|
other model page has. Leave `mode` unset (the Deploy/Playground panels self-cap at 900px, so
|
||||||
|
the default column holds them fine). `mode: wide` belongs only on category `intro.mdx` grids.
|
||||||
|
- `tag: NEW` only for genuine new launches; when one is added, stale `tag: NEW` on older
|
||||||
|
pages should be dropped in the same PR (`grep -RlE "^tag: NEW" docs/cookbook/`).
|
||||||
|
- MDX imports BOTH `Deployment` and `Playground` from `/src/snippets/...` (absolute).
|
||||||
|
- Deploy heading slugs to `deployment` (or `deploy`), Playground to `playground` — so
|
||||||
|
"↑ Switch base" and "Open the Playground →" scroll. No numbered headings for these two.
|
||||||
|
|
||||||
|
### 8. Navigation & homepage
|
||||||
|
- New page → `docs/docs.json` updated: under the right vendor group inside
|
||||||
|
`navigation` → Cookbook → Autoregressive Models, root-relative, **no `.mdx`**:
|
||||||
|
`cookbook/<category>/<Vendor>/<Model>`.
|
||||||
|
- Homepage `<Card href>` in `docs/cookbook/<category>/intro.mdx` points to the vendor's
|
||||||
|
flagship; new vendors get a new `<Card>` + a logo at `docs/cards/logos/<vendor>.png` —
|
||||||
|
**940×525 RGBA transparent, icon-only (no wordmark)**, lowercase filename, tracked via
|
||||||
|
`git add -f` (`*.png` is gitignored repo-wide). Card order matches the `docs.json` nav order.
|
||||||
|
- Don't change `docs/cookbook/intro.mdx` for individual model adds (top-level only).
|
||||||
|
|
||||||
|
### 9. Links & factual
|
||||||
|
- HuggingFace URLs resolve to a real model. License section matches the actual HF license
|
||||||
|
(don't copy from another model). Docker images from `lmsysorg/sglang`; no `sgl-project-dev`.
|
||||||
|
The image **tag** is a real build (a release the author ran, or `:dev`/nightly) — not a
|
||||||
|
guessed version.
|
||||||
|
- Internal links root-relative, no extension (`/cookbook/.../<Model>`); flag `.md`/`.mdx`
|
||||||
|
or `../`-relative links. `docs.sglang.io` is canonical.
|
||||||
|
- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
|
||||||
|
not `${VAR}` (a bash no-op).
|
||||||
|
- **Parser ids must exist in the code registries** on the PR's target branch: every
|
||||||
|
`--reasoning-parser X` / `--tool-call-parser Y` named in prose or in
|
||||||
|
`playgroundFeatures.parsers` flags is a registered key in
|
||||||
|
`python/sglang/srt/parser/reasoning_parser.py` (DetectorMap) /
|
||||||
|
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
|
||||||
|
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
|
||||||
|
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
|
||||||
|
(`python/sglang/srt/parser/template_detection.py`) actually resolve THIS model's
|
||||||
|
chat template to the right parser — no rule match means auto silently disables the
|
||||||
|
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
|
||||||
|
|
||||||
|
### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
|
||||||
|
- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
|
||||||
|
blocks (`> [!NOTE]`), markdown **pipe tables** (use JSX `<table>`), inline `<details>`,
|
||||||
|
or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
|
||||||
|
- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
|
||||||
|
the opening fence); a fenced block nested inside another uses four backticks outside.
|
||||||
|
- §3 commands and outputs are **collapsible** (DeepSeek-V4 pattern): every runnable
|
||||||
|
example wrapped in an `<Accordion>`, its real output in a following
|
||||||
|
`<Accordion title="Example Output">` (`Pending update...` only with user
|
||||||
|
acknowledgement). Flag bare/inline example blocks and `**Output Example:**` headings.
|
||||||
|
- Reasoning-parser example matches the parser's **output shape**: separate-field
|
||||||
|
(`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
|
||||||
|
- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
|
||||||
|
`generation_config.json` defaults); listing them in §1 informationally is fine.
|
||||||
|
|
||||||
|
### 10. Quantization rules
|
||||||
|
- **NVFP4** checkpoints are Blackwell-only (B200/B300/GB300) — never AMD. An AMD FP4 cell
|
||||||
|
is legitimate ONLY when the vendor published an **MXFP4** checkpoint for it (e.g.
|
||||||
|
`amd/Qwen3.5-397B-A17B-MXFP4` on MI355X) — verify the HF repo resolves; otherwise the
|
||||||
|
AMD FP4 chip must be absent/`disabled`.
|
||||||
|
- BF16 / FP8 work on NVIDIA and AMD. `--kv-cache-dtype fp8_e4m3` in a cell is an
|
||||||
|
accuracy-degrading flag — see §3 (needs explicit maintainer sign-off; default
|
||||||
|
home is Playground/tips).
|
||||||
|
|
||||||
|
### 11. Scope
|
||||||
|
- Changes match the PR title. Flag global changes hiding behind a platform-specific title
|
||||||
|
(e.g. an "H200 FP8" PR that adds a flag to ALL cells). Unmentioned side-fixes belong in
|
||||||
|
the PR body.
|
||||||
|
|
||||||
|
### 12. Duplicate PRs
|
||||||
|
- Another open PR for the same model? Flag it; compare completeness; note merge-conflict
|
||||||
|
risk on `docs.json` + the vendor card; flag a superseded older PR by the same author.
|
||||||
|
|
||||||
|
### 13. Build / validate
|
||||||
|
```bash
|
||||||
|
cd docs
|
||||||
|
mint validate
|
||||||
|
mint broken-links
|
||||||
|
```
|
||||||
|
Optional: `mint dev` for a visual smoke test.
|
||||||
|
|
||||||
|
### 14. Reviewer feedback
|
||||||
|
- `gh api repos/sgl-project/sglang/pulls/<N>/comments` — have prior reviewer requests been
|
||||||
|
addressed? Unresolved requested-changes should be flagged.
|
||||||
|
|
||||||
|
### 15. Grammar & spelling
|
||||||
|
- Check added/changed prose for typos and grammar (e.g. "recommend" vs "recommended").
|
||||||
|
Flag each with the exact wrong text + correction.
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Per file:
|
||||||
|
- ✅ PASS
|
||||||
|
- ⚠️ ISSUE: \<what\>
|
||||||
|
- 🔴 BLOCK: \<what\>
|
||||||
|
|
||||||
|
Overall: **APPROVE** / **REQUEST CHANGES** / **BLOCKED**
|
||||||
@@ -0,0 +1,203 @@
|
|||||||
|
---
|
||||||
|
name: env-var-conventions
|
||||||
|
description: Conventions for SGLang environment variables — where to define, how to access, how to name, and how to deprecate. Use when adding, renaming, or reviewing any `SGLANG_*` environment variable (or migrating a legacy `SGL_*` alias), or when touching `python/sglang/srt/environ.py`.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Environment Variables — Conventions
|
||||||
|
|
||||||
|
Apply this skill when adding, renaming, or reviewing any sglang-owned environment variable (`SGLANG_*`, or a legacy `SGL_*` alias being phased out), or when touching `python/sglang/srt/environ.py`.
|
||||||
|
|
||||||
|
## Rule 1 — Define in the `Envs` class in `python/sglang/srt/environ.py`
|
||||||
|
|
||||||
|
All sglang-owned env vars live as `EnvField` descriptors on the `Envs` class. Never add a new `os.getenv("SGLANG_...")`, `get_bool_env_var("SGLANG_...")`, or `get_int_env_var("SGLANG_...")` call site — the helpers in `python/sglang/srt/utils/common.py` carry an explicit `FIXME: move your environment variable to sglang.srt.environ` and exist only for pre-existing call sites.
|
||||||
|
|
||||||
|
Group the new entry under an existing section comment (e.g. `# Logging Options`, `# Scheduler: recv interval`, `# Flashinfer`). Add a new section comment only when none fits — never drop a new entry at the bottom of an unrelated block.
|
||||||
|
|
||||||
|
### Decision table: register in `Envs` or use `os.getenv`?
|
||||||
|
|
||||||
|
| Variable | Owner | Goes through `Envs`? |
|
||||||
|
|---|---|---|
|
||||||
|
| `SGLANG_*` | sglang | **Always.** The canonical prefix for all new entries. |
|
||||||
|
| `MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` | Upstream/vendor alias that sglang wants to centralize | **Yes** — register in `Envs` so `.get()` / `.override()` work uniformly. Keep the upstream prefix. |
|
||||||
|
| `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `HF_HUB_*` (raw upstream) | External tooling | **No.** Read with `os.getenv` — they're set by the launcher / driver, not by sglang. |
|
||||||
|
| `RANK`, `LOCAL_RANK`, `WORLD_SIZE`, `MASTER_ADDR`, `MASTER_PORT`, `HOME`, `PATH` | Distributed launcher / OS | **No.** `os.getenv` only. |
|
||||||
|
| Test runner internals (`PYTEST_CURRENT_TEST`, etc.) | Test framework | **No.** `os.getenv` only. |
|
||||||
|
|
||||||
|
`SGL_*` is **not** a parallel valid prefix — it's a deprecated legacy alias. `_convert_SGL_to_SGLANG` rewrites `SGL_*` to `SGLANG_*` at import time with a `DeprecationWarning`. Never define a new `SGL_*` descriptor or `os.getenv("SGL_...")` call site; if you see one in code, it's tech debt to migrate.
|
||||||
|
|
||||||
|
The rule of thumb: if the value's lifecycle is owned by sglang code (we read it, we may want to override it in tests, we may want to rename it), put it in `Envs`. If the value is set by something outside sglang and we only consume it as-is, use `os.getenv`.
|
||||||
|
|
||||||
|
## Rule 2 — Pick the typed descriptor
|
||||||
|
|
||||||
|
| Type | Use for |
|
||||||
|
|---|---|
|
||||||
|
| `EnvBool(default)` | boolean flag |
|
||||||
|
| `EnvInt(default)` | integer |
|
||||||
|
| `EnvFloat(default)` | float |
|
||||||
|
| `EnvStr(default)` | string |
|
||||||
|
| `EnvTuple(())` | comma-separated list, parsed via `s.split(",")` and stripped |
|
||||||
|
|
||||||
|
Default value:
|
||||||
|
- For a knob whose "unset" state must be distinguishable from any concrete value, use `None` as the default (e.g. `EnvStr(None)`, `EnvInt(None)`). The descriptor handles set-to-None correctly via `_set_to_none`.
|
||||||
|
- For a feature flag, the default encodes the production behavior. `ENABLE_FOO = EnvBool(False)` means foo is off in prod; `DISABLE_FOO = EnvBool(False)` means foo is on in prod. See Rule 4 on picking the verb.
|
||||||
|
|
||||||
|
### IntEnum for multi-state knobs
|
||||||
|
|
||||||
|
When a knob has more than two discrete states (e.g. an off / soft / strict ladder), define an `IntEnum` next to `Envs` and pass the enum member as the `EnvInt` default. Callers compare against the enum, not raw integers:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ToolStrictLevel(IntEnum):
|
||||||
|
OFF = 0
|
||||||
|
FUNCTION = 1
|
||||||
|
PARAMETER = 2
|
||||||
|
|
||||||
|
SGLANG_TOOL_STRICT_LEVEL = EnvInt(ToolStrictLevel.OFF)
|
||||||
|
|
||||||
|
# At the call site:
|
||||||
|
if envs.SGLANG_TOOL_STRICT_LEVEL.get() >= ToolStrictLevel.PARAMETER:
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Don't pile `SGLANG_ENABLE_FOO_STRICT` / `SGLANG_ENABLE_FOO_OFF` boolean knobs that fight each other — one ordered integer is cleaner.
|
||||||
|
|
||||||
|
## Rule 3 — Access via the `EnvField` API, never raw `os.environ`
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sglang.srt.environ import envs
|
||||||
|
|
||||||
|
if envs.SGLANG_FOO.get():
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
`envs.SGLANG_FOO` is the descriptor itself; its `__bool__` and `__len__` raise on purpose so that `if envs.SGLANG_FOO:` fails loudly instead of silently reading as truthy. The `.get()` is mandatory at every read site.
|
||||||
|
|
||||||
|
### Full API surface
|
||||||
|
|
||||||
|
| Method | Use |
|
||||||
|
|---|---|
|
||||||
|
| `.get()` | Read value (parsed). Returns `default` if unset, or `None` if explicitly set to None. |
|
||||||
|
| `.set(value)` | Set value. `set(None)` flips the internal `_set_to_none` flag so the next `.get()` returns `None`, not `default`. |
|
||||||
|
| `.clear()` | Unset entirely. Next `.get()` returns `default`. |
|
||||||
|
| `.is_set()` | True iff the key is present in `os.environ` (regardless of value, including the explicit-None case). |
|
||||||
|
| `.override(value)` | Context manager — set on enter, restore exactly what was there on exit. Use this in tests. |
|
||||||
|
|
||||||
|
The `_set_to_none` distinction matters: `clear()` → `is_set()=False, get()=default`; `set(None)` → `is_set()=True, get()=None`. A few descriptors (e.g. `SGLANG_TEST_MAX_RETRY = EnvInt(None)`, `SGLANG_DISAGGREGATION_THREAD_POOL_SIZE = EnvInt(None)`) rely on this to distinguish "user opted out" from "default applies".
|
||||||
|
|
||||||
|
### Test overrides
|
||||||
|
|
||||||
|
```python
|
||||||
|
with envs.SGLANG_TEST_RETRACT.override(True):
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Don't mutate `os.environ` directly in tests — `override` restores the original cleanly, including the explicit-None state, even if the block raises.
|
||||||
|
|
||||||
|
For multiple overrides composed dynamically, use `ExitStack`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
with ExitStack() as stack:
|
||||||
|
for name, value in test_envs.items():
|
||||||
|
stack.enter_context(getattr(envs, name).override(value))
|
||||||
|
run_test()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Subprocess inheritance
|
||||||
|
|
||||||
|
`override` mutates the real `os.environ`, so child processes spawned **inside** the `with` block inherit the override. This is the supported way to seed a `subprocess.Popen`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
with envs.SGLANG_TEST_RETRACT.override(True):
|
||||||
|
subprocess.Popen([...]).wait()
|
||||||
|
```
|
||||||
|
|
||||||
|
A child started outside the `with` block sees the original value.
|
||||||
|
|
||||||
|
### `temp_set_env` is for non-sglang keys only
|
||||||
|
|
||||||
|
The module-level `temp_set_env(**env_vars)` helper exists for overriding **non-sglang** env vars (e.g. `CUDA_LAUNCH_BLOCKING`, `NCCL_DEBUG`) in tests. It explicitly rejects `SGLANG_*` / `SGL_*` keys:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Wrong — raises ValueError
|
||||||
|
with temp_set_env(SGLANG_TEST_RETRACT="true"): ...
|
||||||
|
|
||||||
|
# Right — sglang keys go through the descriptor
|
||||||
|
with envs.SGLANG_TEST_RETRACT.override(True): ...
|
||||||
|
|
||||||
|
# Right — non-sglang keys go through temp_set_env
|
||||||
|
with temp_set_env(CUDA_LAUNCH_BLOCKING="1"): ...
|
||||||
|
```
|
||||||
|
|
||||||
|
The `allow_sglang=True` escape hatch exists for the rare case where you must bypass `Envs` (e.g. setting an env var **name** that's only constructed at runtime); don't use it just to skip writing a descriptor.
|
||||||
|
|
||||||
|
## Rule 4 — Naming: `SGLANG_` prefix + verb category
|
||||||
|
|
||||||
|
Prefix is always `SGLANG_` for new entries. `SGL_*` is auto-translated to `SGLANG_*` with a `DeprecationWarning` in `_convert_SGL_to_SGLANG`; never add a new `SGL_*` key.
|
||||||
|
|
||||||
|
The second token signals intent. Pick the right verb up front — renames require an alias entry (Rule 5).
|
||||||
|
|
||||||
|
| Verb | Meaning | Example |
|
||||||
|
|---|---|---|
|
||||||
|
| `ENABLE_FOO` | Knob that turns feature foo on/off. Default in the `EnvBool` encodes prod behavior. | `SGLANG_ENABLE_TORCH_COMPILE`, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM` |
|
||||||
|
| `DISABLE_FOO` | Kill-switch. `DISABLE_FOO=True` turns foo off. | `SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP` |
|
||||||
|
| `USE_FOO` | Selects which implementation / backend | `SGLANG_USE_AITER`, `SGLANG_USE_DEEPGEMM_BMM` |
|
||||||
|
| `FORCE_FOO` | Overrides autodetection | `SGLANG_FORCE_FP8_MARLIN`, `SGLANG_FORCE_STREAM_INTERVAL` |
|
||||||
|
| `LOG_FOO` | Logging-only knob | `SGLANG_LOG_GC`, `SGLANG_LOG_MS` |
|
||||||
|
| `TEST_FOO` | Test-only hook | `SGLANG_TEST_RETRACT`, `SGLANG_TEST_MAX_RETRY` |
|
||||||
|
| `DEBUG_FOO` | Debug-only instrumentation | `SGLANG_DEBUG_MEMORY_POOL`, `SGLANG_DEBUG_SYMM_MEM` |
|
||||||
|
| `OPT_FOO` | Perf-optimization toggle (heavily used by DSV4 work) | `SGLANG_OPT_USE_FUSED_HASH_TOPK`, `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2` |
|
||||||
|
|
||||||
|
Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only forbidden combination is `DISABLE_FOO = EnvBool(True)`, because it produces a true double-negative at the call site (`if not envs.SGLANG_DISABLE_FOO.get():` reads as "if not disabled"). All other combinations are fine:
|
||||||
|
|
||||||
|
| Pattern | Call site | Verdict |
|
||||||
|
|---|---|---|
|
||||||
|
| `ENABLE_FOO = EnvBool(False)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — opt-in feature |
|
||||||
|
| `ENABLE_FOO = EnvBool(True)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — on in prod, user opts out via `False` |
|
||||||
|
| `DISABLE_FOO = EnvBool(False)` | `if not envs.SGLANG_DISABLE_FOO.get():` | OK — single negation, reads as "if enabled" |
|
||||||
|
| `DISABLE_FOO = EnvBool(True)` | `if not envs.SGLANG_DISABLE_FOO.get():` | **Forbidden** — true double-negative |
|
||||||
|
|
||||||
|
`SGLANG_*` is the canonical sglang prefix. Vendor-integration keys (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`) keep their upstream prefix and live in the same `Envs` class — these are integration aliases, not sglang-owned feature flags.
|
||||||
|
|
||||||
|
## Rule 5 — Renames go through `*WithAlias` or `_print_deprecated_env`
|
||||||
|
|
||||||
|
For a rename where the old key must keep working with a warning:
|
||||||
|
|
||||||
|
```python
|
||||||
|
SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME")
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
|
||||||
|
|
||||||
|
For a full removal where the env var is going away, add to `_convert_SGL_to_SGLANG`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
_print_deprecated_env("SGLANG_OLD_NAME", "SGLANG_NEW_NAME") # mapped to a replacement
|
||||||
|
_print_deprecated_env("SGLANG_OLD_NAME") # no replacement, gone
|
||||||
|
```
|
||||||
|
|
||||||
|
For env-var to CLI-flag migration, add at module top-level:
|
||||||
|
|
||||||
|
```python
|
||||||
|
_warn_deprecated_env_to_cli_flag(
|
||||||
|
"SGLANG_FOO",
|
||||||
|
"Please use '--foo' instead.",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Don't silently flip a default during a rename. If the new default disagrees with the old one, that's a behavior change — call it out in the PR body separately from the rename.
|
||||||
|
|
||||||
|
## Rule 6 — Env var vs CLI flag
|
||||||
|
|
||||||
|
| If the knob is… | Goes in |
|
||||||
|
|---|---|
|
||||||
|
| User-facing (documented, expected to flip per deployment) | `server_args.py` CLI flag |
|
||||||
|
| Expert toggle, A-B kill-switch, vendor integration | `environ.py` env var |
|
||||||
|
| Test / debug hook | `environ.py` env var with `TEST_` / `DEBUG_` prefix |
|
||||||
|
| Temporary env var rolling out to a CLI flag | env var first, then migrate via `_warn_deprecated_env_to_cli_flag` |
|
||||||
|
|
||||||
|
Don't add a CLI flag that just forwards to an env var, and don't add an env var that duplicates an existing CLI flag. Pick one surface.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- **External / vendor env vars consumed raw** (`HF_HUB_*`, `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `RANK`, `MASTER_ADDR`, etc.): see the decision table in Rule 1 — `os.getenv` is correct, don't pull them into `Envs`.
|
||||||
|
- **Pre-existing `get_bool_env_var(...)` / `get_int_env_var(...)` call sites**: leave them as is; new code shouldn't add more, but mass-migration is out of scope for a feature PR.
|
||||||
|
- **Upstream-aliased keys already in `Envs`** (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` — see Rule 1 decision table): the `SGLANG_` prefix rules in Rule 4 don't apply — the upstream prefix is the canonical name.
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
---
|
||||||
|
name: generate-profile
|
||||||
|
description: Generate an e2e profiling trace of an SGLang server run. Launches a server, validates accuracy, captures a Chrome-compatible trace, and returns the profile path.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Generate an E2E Profile of an SGLang Server Run
|
||||||
|
|
||||||
|
This skill launches an SGLang server, validates it with a quick accuracy test, generates a profiling trace, and returns the profile file path.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||||
|
- At least one available CUDA GPU
|
||||||
|
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
|
||||||
|
|
||||||
|
## Step-by-step Workflow
|
||||||
|
|
||||||
|
### Step 1: Launch the server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
CUDA_VISIBLE_DEVICES=<gpu_id> sglang serve --model-path <model> --port <port> &
|
||||||
|
```
|
||||||
|
|
||||||
|
- Default model: `Qwen/Qwen3-8B` (good balance of speed and quality)
|
||||||
|
- Default port: `30000`
|
||||||
|
- The server runs in the background. Save the PID for cleanup.
|
||||||
|
- Use the GPU specified by the user's preferences (check memory files for GPU preferences).
|
||||||
|
|
||||||
|
### Step 2: Wait for server readiness
|
||||||
|
|
||||||
|
Poll the health endpoint until the server is ready:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for i in $(seq 1 120); do
|
||||||
|
if curl -s http://127.0.0.1:<port>/health 2>/dev/null | grep -q "ok\|healthy"; then
|
||||||
|
echo "Server ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
The server prints **"The server is fired up and ready to roll!"** to stdout when ready. The health endpoint returns 200 once the server can accept requests.
|
||||||
|
|
||||||
|
Typical startup time: 30-90 seconds depending on model size and whether CUDA graphs are being compiled.
|
||||||
|
|
||||||
|
### Step 3: Validate accuracy (sanity check)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
|
||||||
|
```
|
||||||
|
|
||||||
|
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||||
|
- This is a quick sanity check, not a rigorous benchmark.
|
||||||
|
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||||
|
|
||||||
|
### Step 4: Generate the profile
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m sglang.test.send_one --profile
|
||||||
|
```
|
||||||
|
|
||||||
|
This command:
|
||||||
|
1. Sends a request to the server
|
||||||
|
2. Triggers the profiler for 5 steps (default)
|
||||||
|
3. Generates a trace file under `/tmp/<timestamp>/`
|
||||||
|
4. The trace directory contains:
|
||||||
|
- `<timestamp>-TP-0.trace.json.gz` — Chrome trace format (open in `chrome://tracing` or Perfetto)
|
||||||
|
- `server_args.json` — the server configuration used
|
||||||
|
|
||||||
|
**Output format:**
|
||||||
|
```
|
||||||
|
Dump profiling traces to /tmp/<timestamp>
|
||||||
|
```
|
||||||
|
|
||||||
|
The profile path is printed to stdout. Parse it from the output.
|
||||||
|
|
||||||
|
**Optional flags:**
|
||||||
|
- `--profile-steps N` — number of profiling steps (default: 5)
|
||||||
|
- `--profile-by-stage` — profile by stage (prefill/decode separately)
|
||||||
|
- `--profile-prefix <path>` — custom output prefix
|
||||||
|
|
||||||
|
### Step 5: Kill the server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
|
||||||
|
```
|
||||||
|
|
||||||
|
Wait a moment and verify no sglang processes remain:
|
||||||
|
```bash
|
||||||
|
sleep 2 && pgrep -af "sglang serve" || echo "Server killed"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 6: Report the profile path
|
||||||
|
|
||||||
|
Return the profile directory path (e.g., `/tmp/1773999986.4769795`) and list its contents so the user knows what files were generated.
|
||||||
|
|
||||||
|
## Example Full Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Launch server
|
||||||
|
source cleanup/bin/activate
|
||||||
|
CUDA_VISIBLE_DEVICES=1 sglang serve --model-path Qwen/Qwen3-8B --port 30000 &
|
||||||
|
|
||||||
|
# 2. Wait for ready
|
||||||
|
for i in $(seq 1 120); do
|
||||||
|
curl -s http://127.0.0.1:30000/health | grep -q "ok" && break
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
# 3. Accuracy check
|
||||||
|
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
|
||||||
|
# Expected: score > 80%
|
||||||
|
|
||||||
|
# 4. Profile
|
||||||
|
python3 -m sglang.test.send_one --profile
|
||||||
|
# Output: "Dump profiling traces to /tmp/1773999986.4769795"
|
||||||
|
|
||||||
|
# 5. Cleanup
|
||||||
|
pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
|
||||||
|
sleep 2
|
||||||
|
|
||||||
|
# 6. Check output
|
||||||
|
ls -la /tmp/1773999986.4769795/
|
||||||
|
# 1773999986.4851577-TP-0.trace.json.gz (Chrome trace)
|
||||||
|
# server_args.json (server config)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Customization
|
||||||
|
|
||||||
|
- **Different port**: Pass `--port <port>` and use `--host 127.0.0.1 --port <port>` for test commands
|
||||||
|
- **Multi-GPU**: Use `--tp <N>` for tensor parallelism; trace files will be generated per TP rank
|
||||||
|
- **Longer profile**: Use `--profile-steps 10` for more steps in the trace
|
||||||
|
- **Stage profiling**: Use `--profile-by-stage` to separate prefill and decode phases
|
||||||
|
|
||||||
|
## Viewing the Profile
|
||||||
|
|
||||||
|
Open the `.trace.json.gz` file in:
|
||||||
|
- **Perfetto UI**: https://ui.perfetto.dev/ (drag and drop the file)
|
||||||
|
- **Chrome tracing**: `chrome://tracing` (load the file)
|
||||||
|
|
||||||
|
Both support the gzipped Chrome trace format natively.
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
---
|
||||||
|
name: kernel-organization
|
||||||
|
description: Apply the SGLang kernels RFC when adding, moving, splitting, or reviewing kernel APIs, registry metadata, kernel tests, benchmarks, and model-specific implementations. Use with add-jit-kernel, add-sgl-kernel, and write-sglang-test for placement and migration checks.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Kernel organization
|
||||||
|
|
||||||
|
Read `python/sglang/kernels/README.md` and `test/README.md` before choosing a
|
||||||
|
location. [RFC #29630](https://github.com/sgl-project/sglang/issues/29630)
|
||||||
|
establishes the namespace; subsequent migrations #32148 and #40922 clarify
|
||||||
|
logical grouping and remove the old `_jit_` filename prefix.
|
||||||
|
|
||||||
|
## Choose the owner
|
||||||
|
|
||||||
|
- Put callable SGLang kernel APIs under `sglang.kernels.ops.<group>`. Runtime
|
||||||
|
and integration tests import from that namespace, including its submodules.
|
||||||
|
The `sgl_kernel` wheel retains its own public API and packaging tests.
|
||||||
|
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
|
||||||
|
routing in `moe`, attention index selection in `attention`, sampling in
|
||||||
|
`sampling`, normalization in `layernorm`.
|
||||||
|
- Classify a fused operator by its complete contract; do not split a fused
|
||||||
|
kernel into separate launches just to separate its parts. Split unrelated
|
||||||
|
public entry points that happen to share a source file. Decided rules:
|
||||||
|
- Any cross-rank collective or peer-memory access (all-reduce, all-gather,
|
||||||
|
reduce-scatter, a2a, multicast/NVLS) puts the op in `communication`,
|
||||||
|
whatever compute it fuses.
|
||||||
|
- Producer + output quantization stays with the producer (norm + quant in
|
||||||
|
`layernorm`, activation + quant in `activation`, GEMM with a quant epilogue
|
||||||
|
in `gemm`, MoE dispatch/permute + quant in `moe`). `quantization` holds only
|
||||||
|
pure format conversion: quant, dequant, scale-layout conversion, repack.
|
||||||
|
- RoPE stays in `attention` (standalone RoPE, position ids, QK-norm + RoPE).
|
||||||
|
- `kvcache` owns ops that read/write KV-cache contents; `attention` owns the
|
||||||
|
index/shape/schedule tables (page tables, `cu_seqlens`, `kv_indices`, split
|
||||||
|
metadata) and index selection. A fused compute + store stays with its
|
||||||
|
compute group.
|
||||||
|
- Model-specific files/subpackages and tuning data are allowed **inside** a
|
||||||
|
logical group. Do not add a model bundle or an implementation file directly
|
||||||
|
under `ops/`. Propose a new logical group only for a distinct responsibility,
|
||||||
|
and update `ops._GROUPS`, the README group list, and tests together
|
||||||
|
(`test_kernels_namespace.py` checks the README against `_GROUPS` and the
|
||||||
|
README layout against the directories under `kernels/`).
|
||||||
|
- Agent-generated implementations live in `kernels/kda_kernels/` (Kernel Design
|
||||||
|
Agents, `KernelBackend.KDA`) and `kernels/cake_kernels/` (Cake adapters over
|
||||||
|
FlashInfer). They are backends, not groups: register their public ops in the
|
||||||
|
logical group with a `KernelSpec` whose target points into them, and do not
|
||||||
|
reorganize those directories.
|
||||||
|
- Keep shared build/runtime infrastructure (including the Helion runtime
|
||||||
|
helper) in `kernels/jit`; operator wrappers call it from their logical group.
|
||||||
|
Keep process groups, communicator state, workspace classes, model dispatch,
|
||||||
|
and buffer ownership in `srt` (`srt/layers/communication/` for the K3 and Muse
|
||||||
|
Spark adapters); kernels must not import srt runtime modules to inspect
|
||||||
|
model objects, so pass flags in. Adapters there need not pretend to be generic
|
||||||
|
interfaces.
|
||||||
|
|
||||||
|
## Register the public entry point
|
||||||
|
|
||||||
|
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
|
||||||
|
existing `BaseFusedOp` registration when the operation has interchangeable
|
||||||
|
backends (a `BaseFusedOp` built per layer registers `module:Class.forward_<backend>`
|
||||||
|
targets). Group imports must not import GPU implementations or compile kernels.
|
||||||
|
|
||||||
|
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
|
||||||
|
existing backend dispatch and describe actual device/architecture restrictions
|
||||||
|
with `CapabilityRequirement`; JIT/AOT are not device types. Input shape/dtype
|
||||||
|
checks remain part of the entry point's contract. Registering a torch custom op
|
||||||
|
does not register it in the SGLang kernel inventory. Predicates, private JIT
|
||||||
|
factories, reference helpers, and runtime classes are not separate kernel APIs.
|
||||||
|
|
||||||
|
## Place and preserve tests
|
||||||
|
|
||||||
|
- Kernel numerical tests: `test/registered/kernels/ops/<group>/`.
|
||||||
|
- CI microbenchmarks: `test/registered/kernels/benchmark/<group>/`.
|
||||||
|
- Inventory/selector tests: `test/registered/unit/kernels/`.
|
||||||
|
- Runtime unit tests: `test/registered/unit/<subsystem>/`, following the tested
|
||||||
|
runtime module. A CUDA allocation does not make a runtime test a kernel test.
|
||||||
|
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
|
||||||
|
standalone test/benchmark entry points in production operator modules.
|
||||||
|
- Shared test helpers and reference oracles: `sglang.test.kernels`. Preserve
|
||||||
|
vendored upstream trees and AOT wheel packaging boundaries instead of
|
||||||
|
reorganizing them incidentally.
|
||||||
|
|
||||||
|
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
|
||||||
|
execution entry points, and CI stages/runners. Apportion existing time estimates
|
||||||
|
across split files; do not duplicate the original budget for every output file
|
||||||
|
or silently drop a registration. Follow `write-sglang-test` for CI registration.
|
||||||
|
|
||||||
|
## Verify a migration
|
||||||
|
|
||||||
|
1. Search all runtime, test, benchmark, documentation, patch-string, and lazy
|
||||||
|
registry references before deleting the old path. Check relative imports and
|
||||||
|
package re-exports as well as direct imports. Do not leave forwarding shims.
|
||||||
|
2. Keep tuning files with the GEMM that loads them and preserve relative lookup
|
||||||
|
behavior. Check wheel/package inclusion as well as source-tree execution.
|
||||||
|
3. Preserve module singleton state: every runtime caller must import the same
|
||||||
|
new communication adapter, not a second copy of its buffer registry.
|
||||||
|
4. Separate relocation commits from semantic changes. Use
|
||||||
|
`mechanical-refactor-verify` to reproduce moves, and compare test inventories
|
||||||
|
and CI registration coverage before/after. Only delete an experimental path
|
||||||
|
after checking its call sites, flags, source, and dedicated tests together.
|
||||||
|
5. Run the namespace/dispatch CPU tests, the registered-test validation hook,
|
||||||
|
pre-commit, and relevant GPU tests when available. State exactly which GPU
|
||||||
|
checks ran; import/AST checks do not prove numerical or performance parity.
|
||||||
|
|
||||||
|
Do not infer violations solely from a model name inside a group, a missing
|
||||||
|
`_jit_` prefix, use of a runtime utility, or absence of `BaseFusedOp` inheritance.
|
||||||
+15
-15
@@ -46,18 +46,18 @@ in-flight row as shipped.
|
|||||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||||
| --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- |
|
||||||
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
|
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
|
||||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/kernels/ops/layernorm/fused_rmsnorm.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||||
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
||||||
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
||||||
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
||||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||||
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
||||||
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
||||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||||
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
||||||
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
||||||
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
||||||
@@ -69,18 +69,18 @@ in-flight row as shipped.
|
|||||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
|
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||||
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
||||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/kernels/ops/attention/linear/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||||
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
|
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
|
||||||
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
|
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
|
||||||
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
|
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
|
||||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/kernels/ops/attention/linear/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/kernels/ops/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||||
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
|
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
|
||||||
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
|
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
|
||||||
|
|
||||||
@@ -146,10 +146,10 @@ Stable entries should be folded into the mainline family rows above.
|
|||||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||||
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
|
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
|
||||||
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
|
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
|
||||||
|
|
||||||
@@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package.
|
|||||||
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
|
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
|
||||||
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
|
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
|
||||||
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
|
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
|
||||||
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
||||||
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
|
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
|
||||||
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
|
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
|
||||||
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
|
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
|
||||||
+1
-4
@@ -18,7 +18,7 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
|||||||
|
|
||||||
## Scheduler-side trace writing
|
## Scheduler-side trace writing
|
||||||
|
|
||||||
- `python/sglang/srt/managers/scheduler_profiler_mixin.py`
|
- `python/sglang/srt/managers/scheduler_components/profiler_manager.py`
|
||||||
- actual trace start/stop behavior
|
- actual trace start/stop behavior
|
||||||
- filename pattern for `TP/DP/PP/EP` and optional stage suffixes
|
- filename pattern for `TP/DP/PP/EP` and optional stage suffixes
|
||||||
- `CUDA_PROFILER` and torch profiler handling
|
- `CUDA_PROFILER` and torch profiler handling
|
||||||
@@ -37,6 +37,3 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
|||||||
|
|
||||||
- `test/registered/profiling/test_start_profile.py`
|
- `test/registered/profiling/test_start_profile.py`
|
||||||
- validates `/start_profile` behavior, including `CUDA_PROFILER`
|
- validates `/start_profile` behavior, including `CUDA_PROFILER`
|
||||||
|
|
||||||
- `test/registered/profiling/test_profile_v2.py`
|
|
||||||
- validates stage-scoped trace outputs under `SGLANG_PROFILE_V2`
|
|
||||||
+9
-7
@@ -391,7 +391,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
candidate_path=(
|
candidate_path=(
|
||||||
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
|
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
|
||||||
"<br>python/sglang/srt/layers/layernorm.py"
|
"<br>python/sglang/srt/layers/layernorm.py"
|
||||||
"<br>python/sglang/srt/layers/communicator.py"
|
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||||
),
|
),
|
||||||
active_keywords=(
|
active_keywords=(
|
||||||
"allreduce_fusion",
|
"allreduce_fusion",
|
||||||
@@ -419,7 +419,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
pattern="AITER allreduce fusion",
|
pattern="AITER allreduce fusion",
|
||||||
candidate_path=(
|
candidate_path=(
|
||||||
"python/sglang/srt/distributed/communication_op.py"
|
"python/sglang/srt/distributed/communication_op.py"
|
||||||
"<br>python/sglang/srt/layers/communicator.py"
|
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||||
"<br>python/sglang/srt/layers/layernorm.py"
|
"<br>python/sglang/srt/layers/layernorm.py"
|
||||||
),
|
),
|
||||||
active_keywords=(
|
active_keywords=(
|
||||||
@@ -482,7 +482,9 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
),
|
),
|
||||||
FusionPatternSpec(
|
FusionPatternSpec(
|
||||||
pattern="Fused QK RoPE reshape + KV cache write",
|
pattern="Fused QK RoPE reshape + KV cache write",
|
||||||
candidate_path="python/sglang/srt/layers/attention/utils.py",
|
candidate_path=(
|
||||||
|
"python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py"
|
||||||
|
),
|
||||||
active_keywords=("fused_qk_rope_reshape_and_cache",),
|
active_keywords=("fused_qk_rope_reshape_and_cache",),
|
||||||
split_groups=(
|
split_groups=(
|
||||||
("rotary", "rope", "mrope"),
|
("rotary", "rope", "mrope"),
|
||||||
@@ -533,7 +535,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
),
|
),
|
||||||
FusionPatternSpec(
|
FusionPatternSpec(
|
||||||
pattern="NSA fused metadata copy for graph replay",
|
pattern="NSA fused metadata copy for graph replay",
|
||||||
candidate_path="python/sglang/kernels/ops/attention/fused_metadata_copy.py",
|
candidate_path="python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py",
|
||||||
active_keywords=(
|
active_keywords=(
|
||||||
"fused_metadata_copy",
|
"fused_metadata_copy",
|
||||||
"fused_metadata_copy_multi",
|
"fused_metadata_copy_multi",
|
||||||
@@ -700,7 +702,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
pattern="Fused MoE activation + quant / re-quant",
|
pattern="Fused MoE activation + quant / re-quant",
|
||||||
candidate_path=(
|
candidate_path=(
|
||||||
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
|
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
|
||||||
"<br>python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
"<br>python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||||
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
|
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
|
||||||
),
|
),
|
||||||
active_keywords=(
|
active_keywords=(
|
||||||
@@ -721,7 +723,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
FusionPatternSpec(
|
FusionPatternSpec(
|
||||||
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
|
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
|
||||||
candidate_path=(
|
candidate_path=(
|
||||||
"python/sglang/srt/layers/communicator.py"
|
"python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||||
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
||||||
"forward_mla.py"
|
"forward_mla.py"
|
||||||
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
||||||
@@ -761,7 +763,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
|||||||
FusionPatternSpec(
|
FusionPatternSpec(
|
||||||
pattern="NSA fused quantize + indexed K-cache store",
|
pattern="NSA fused quantize + indexed K-cache store",
|
||||||
candidate_path=(
|
candidate_path=(
|
||||||
"python/sglang/kernels/ops/attention/fused_store_index_cache.py"
|
"python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py"
|
||||||
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
||||||
),
|
),
|
||||||
active_keywords=("fused_store_index_k_cache",),
|
active_keywords=("fused_store_index_k_cache",),
|
||||||
@@ -0,0 +1,249 @@
|
|||||||
|
---
|
||||||
|
name: processor-model-parity
|
||||||
|
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Processor model parity
|
||||||
|
|
||||||
|
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
|
||||||
|
server, sgl-router, the renderer). A model is supported only when, for the same
|
||||||
|
request body, the processor produces **the same prompt text and the same prompt
|
||||||
|
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
|
||||||
|
the implementation, and SGLang code exists only where the two differ.
|
||||||
|
|
||||||
|
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
|
||||||
|
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
|
||||||
|
Multimodal preprocessing is not covered here: `render_request` fails on media
|
||||||
|
content parts, so hosts send those chats to the engine. The fixture generator
|
||||||
|
serves every checkpoint text-only, so a media case records that rejection.
|
||||||
|
|
||||||
|
## Done means
|
||||||
|
|
||||||
|
- **All text should work**: every text chat the engine accepts renders and
|
||||||
|
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
|
||||||
|
the processor cannot reproduce goes to the engine; list it in the PR.
|
||||||
|
- **Chat and completions go through the engine's `/generate`**, with the router
|
||||||
|
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
|
||||||
|
engine's OpenAI route is only the `Unsupported` fallback. That needs the
|
||||||
|
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
|
||||||
|
against a live engine.
|
||||||
|
|
||||||
|
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
|
||||||
|
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
|
||||||
|
to `rust/sglang-processor/` unless they start with `python/`.
|
||||||
|
|
||||||
|
## Where a model's code goes
|
||||||
|
|
||||||
|
| Piece | File | Needed when |
|
||||||
|
|---|---|---|
|
||||||
|
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
|
||||||
|
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
|
||||||
|
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
|
||||||
|
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
|
||||||
|
| README row | the `render` table in `README.md` | always |
|
||||||
|
|
||||||
|
No new test code: `tests/parity.rs` checks every fixture in the directory.
|
||||||
|
|
||||||
|
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
|
||||||
|
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
|
||||||
|
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
|
||||||
|
renders the model:
|
||||||
|
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
|
||||||
|
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
|
||||||
|
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
|
||||||
|
`PromptFormatter`, replace that branch.
|
||||||
|
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
|
||||||
|
variant holding Dynamo's Jinja `PromptFormatter`, picked in
|
||||||
|
`ChatFormatter::from_tokenizer_config` from the template text (as
|
||||||
|
`parser/template_detection.py` picks it, where it has a detector), and build the
|
||||||
|
template context that `_process_messages` and `_apply_jinja_template` build, like
|
||||||
|
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
|
||||||
|
rewrites shapes its load-time probes find the template rejects (a later system
|
||||||
|
turn, for one); SGLang raises on them, so check those before rendering.
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
|
||||||
|
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
|
||||||
|
Note whether its tokenizer ships a `chat_template`; that can change the spec.
|
||||||
|
1. **Trace the Python path for the model.** Write down every transformation
|
||||||
|
between the HTTP body and `prompt_ids`, in order:
|
||||||
|
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
|
||||||
|
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
|
||||||
|
Reuse `render/reasoning.rs` for this rather than porting it again.
|
||||||
|
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
|
||||||
|
replaces the request effort; server `--default-chat-template-kwargs` merge in.
|
||||||
|
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
|
||||||
|
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
|
||||||
|
It also reads the server's `--tool-call-parser` and whether the tokenizer has
|
||||||
|
a chat template. If the deployment depends on the parser, set
|
||||||
|
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
|
||||||
|
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
|
||||||
|
effort profile reads `encoding/encoding_dsv4.py`).
|
||||||
|
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
|
||||||
|
message dumping, content flattening, `_handle_last_assistant_message`, system
|
||||||
|
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
|
||||||
|
call, and how the prompt is tokenized.
|
||||||
|
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
|
||||||
|
rules live there: dropped turns, `</think>` on empty reasoning, argument
|
||||||
|
formatting, and the errors it raises.
|
||||||
|
|
||||||
|
This list is the **divergence inventory**.
|
||||||
|
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
|
||||||
|
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
|
||||||
|
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
|
||||||
|
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
|
||||||
|
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
|
||||||
|
which often differ from SGLang's. Diff the low-level encoder against Python's
|
||||||
|
for the rules in step 1.
|
||||||
|
3. **Write one request case per inventory item** in the fixture (`name` +
|
||||||
|
`request` only), then generate the expected outputs from Python (below). Cover
|
||||||
|
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
|
||||||
|
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
|
||||||
|
each failing case, add the smallest SGLang step that fixes it, with a one-line
|
||||||
|
doc comment naming the Python source it mirrors. Never patch the fixture to
|
||||||
|
match Rust.
|
||||||
|
5. **Verify** text offline, then token ids with the checkpoint cached (commands
|
||||||
|
below). If sgl-router or the renderer uses the formatter, run their suites too.
|
||||||
|
|
||||||
|
## Checklist (DeepSeek-V4 case that covers each)
|
||||||
|
|
||||||
|
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
|
||||||
|
reduced to role and content; null content becomes `""`; history tool calls keep
|
||||||
|
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
|
||||||
|
- **Content parts**: which part types survive and the join separator. (`parts`)
|
||||||
|
- **Tools**: all request tools or only those `tool_choice` allows; field order
|
||||||
|
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
|
||||||
|
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
|
||||||
|
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
|
||||||
|
`tool_schema_type_aliases`)
|
||||||
|
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
|
||||||
|
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
|
||||||
|
- **Pydantic coercion**: typed request fields are coerced before rendering. For
|
||||||
|
example, `strict: 1` and `defer_loading: "false"` become booleans, and
|
||||||
|
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
|
||||||
|
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
|
||||||
|
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
|
||||||
|
JSON string; other encoders format them their own way. Keep key order
|
||||||
|
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
|
||||||
|
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
|
||||||
|
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
|
||||||
|
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
|
||||||
|
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
|
||||||
|
truthiness (`1` is on); strings use Python's yes-word set.
|
||||||
|
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
|
||||||
|
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
|
||||||
|
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
|
||||||
|
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
|
||||||
|
a separately tokenized prefix. Check whether content is flattened before the
|
||||||
|
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
|
||||||
|
assistant turn's tool calls are discarded, so object checks on their arguments
|
||||||
|
belong after the split. Hosts reach the model through `render_prompt` too, so
|
||||||
|
check their continuation handling as well.
|
||||||
|
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
|
||||||
|
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
|
||||||
|
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
|
||||||
|
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
|
||||||
|
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
|
||||||
|
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
|
||||||
|
continuation prefix is encoded alone and loses a leading BOS
|
||||||
|
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
|
||||||
|
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
|
||||||
|
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
|
||||||
|
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
|
||||||
|
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
|
||||||
|
includes pydantic rejections and Python crashes (a 500 is still a rejection).
|
||||||
|
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
|
||||||
|
PR that hosts fall back to the engine for it.
|
||||||
|
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
|
||||||
|
reach, fix it upstream and mark the case `known_gap` with the reason. The test
|
||||||
|
reports the case without failing, and fails once it matches so the marker gets
|
||||||
|
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
|
||||||
|
|
||||||
|
## Generating fixtures
|
||||||
|
|
||||||
|
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
|
||||||
|
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
|
||||||
|
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
|
||||||
|
renders it through the real `_process_messages`, under the case's
|
||||||
|
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
|
||||||
|
writes:
|
||||||
|
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
|
||||||
|
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
|
||||||
|
are kept as written). An `AttributeError` aborts instead: the stub server lacks
|
||||||
|
something, so extend it;
|
||||||
|
- the fixture-level `config` (the `config.json` fields the processor reads, plus
|
||||||
|
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
|
||||||
|
checkpoint its `chat_template`, so the test renders offline.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd rust/sglang-processor
|
||||||
|
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
|
||||||
|
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
|
||||||
|
```
|
||||||
|
|
||||||
|
When a new spec needs more, extend `serving_chat()` in the script (one place):
|
||||||
|
- **More server state:** set the extra attribute that `__init__` resolves, for
|
||||||
|
example `_dsv41_default_reasoning_effort`.
|
||||||
|
|
||||||
|
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
|
||||||
|
recorded by the template's text output. Its `render_request` returns the
|
||||||
|
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
|
||||||
|
|
||||||
|
Pin `revision` to a commit hash and regenerate only on purpose.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd rust
|
||||||
|
cargo test -p sglang-processor --locked # text, offline
|
||||||
|
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
|
||||||
|
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
|
||||||
|
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
|
||||||
|
```
|
||||||
|
|
||||||
|
The token check loads the pinned commit's snapshot from the cache. When the
|
||||||
|
snapshot is missing, the test prints `token ids not checked` (visible with
|
||||||
|
`-- --nocapture`), so confirm that line is absent before claiming token parity.
|
||||||
|
With the snapshot cached, the test also checks that the checkpoint resolves the
|
||||||
|
fixture's recorded DeepSeek-V4 profile.
|
||||||
|
|
||||||
|
## OpenAI layer
|
||||||
|
|
||||||
|
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
|
||||||
|
once its render parity holds. Two more fixtures then cover the rest of Python's
|
||||||
|
OpenAI layer:
|
||||||
|
- `tests/fixtures/reasoning_parity/<parser>.json`, from
|
||||||
|
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
|
||||||
|
chunked outputs. A parser is served once `src/think/models/` ports it.
|
||||||
|
- `tests/fixtures/tool_parity/<parser>.json`, from
|
||||||
|
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
|
||||||
|
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
|
||||||
|
parser is served once `src/tool_call/models/` ports it.
|
||||||
|
- `tests/fixtures/openai_parity/<model-id>.json`, from
|
||||||
|
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
|
||||||
|
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
|
||||||
|
output for it, and the response Python builds from that output. Launch the
|
||||||
|
engine with the model's cookbook flags, including its parsers. Without the
|
||||||
|
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
|
||||||
|
fixture does.
|
||||||
|
|
||||||
|
## Pitfalls
|
||||||
|
|
||||||
|
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
|
||||||
|
`dynamo-renderer` or `dynamo-tokenizers`.
|
||||||
|
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
|
||||||
|
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
|
||||||
|
sent, so such requests stay on the engine's route. Fixtures don't show this; a
|
||||||
|
live engine does.
|
||||||
|
- Python mutates the request before rendering (`_validate_request` rewrites tool
|
||||||
|
schemas), so fixture generators drive the server's own entry points.
|
||||||
|
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
|
||||||
|
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
|
||||||
|
generator pins them and `tests/parity.rs` clears them, so fixtures do not
|
||||||
|
depend on the machine.
|
||||||
|
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
|
||||||
|
such as `task` and message-level `tools`. Parity is defined on `render_request`;
|
||||||
|
report host-adapter gaps separately rather than bending the model code.
|
||||||
|
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
|
||||||
|
beyond the dispatch arm.
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user