Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
11e5ba6e8f | ||
|
|
fb341f8de5 |
@@ -1,82 +0,0 @@
|
||||
---
|
||||
name: kernel-organization
|
||||
description: Apply the SGLang kernels RFC when adding, moving, splitting, or reviewing kernel APIs, registry metadata, kernel tests, benchmarks, and model-specific implementations. Use with add-jit-kernel, add-sgl-kernel, and write-sglang-test for placement and migration checks.
|
||||
---
|
||||
|
||||
# Kernel organization
|
||||
|
||||
Read `python/sglang/kernels/README.md` and `test/README.md` before choosing a
|
||||
location. [RFC #29630](https://github.com/sgl-project/sglang/issues/29630)
|
||||
establishes the namespace; subsequent migrations #32148 and #40922 clarify
|
||||
logical grouping and remove the old `_jit_` filename prefix.
|
||||
|
||||
## Choose the owner
|
||||
|
||||
- Put callable SGLang kernel APIs under `sglang.kernels.ops.<group>`. Runtime
|
||||
and integration tests import from that namespace, including its submodules.
|
||||
The `sgl_kernel` wheel retains its own public API and packaging tests.
|
||||
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
|
||||
routing in `moe`, attention index selection in `attention`, sampling in
|
||||
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
|
||||
quantization alone belongs in `quantization`.
|
||||
- Model-specific files/subpackages and tuning data are allowed **inside** a
|
||||
logical group. Do not add a model bundle or an implementation file directly
|
||||
under `ops/`. Propose a new logical group only for a distinct responsibility,
|
||||
and update `ops._GROUPS`, documentation, and tests together.
|
||||
- Classify a fused operator by its complete contract. Do not split a fused
|
||||
kernel into separate launches just to separate norm, RoPE, or quantization.
|
||||
Split unrelated public entry points that happen to share a source file.
|
||||
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
|
||||
wrappers call it from their logical group. Keep process groups, communicator
|
||||
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
|
||||
`srt/layers/communication/` need not pretend to be generic interfaces.
|
||||
|
||||
## Register the public entry point
|
||||
|
||||
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
|
||||
existing `BaseFusedOp` registration when the operation has interchangeable
|
||||
backends. Group imports must not import GPU implementations or compile kernels.
|
||||
|
||||
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
|
||||
existing backend dispatch and describe actual device/architecture restrictions
|
||||
with `CapabilityRequirement`; JIT/AOT are not device types. Input shape/dtype
|
||||
checks remain part of the entry point's contract. Registering a torch custom op
|
||||
does not register it in the SGLang kernel inventory. Predicates, private JIT
|
||||
factories, reference helpers, and runtime classes are not separate kernel APIs.
|
||||
|
||||
## Place and preserve tests
|
||||
|
||||
- Kernel numerical tests: `test/registered/kernels/ops/<group>/`.
|
||||
- CI microbenchmarks: `test/registered/kernels/benchmark/<group>/`.
|
||||
- Inventory/selector tests: `test/registered/unit/kernels/`.
|
||||
- Runtime unit tests: `test/registered/unit/<subsystem>/`, following the tested
|
||||
runtime module. A CUDA allocation does not make a runtime test a kernel test.
|
||||
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
|
||||
standalone test/benchmark entry points in production operator modules.
|
||||
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
|
||||
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
|
||||
|
||||
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
|
||||
execution entry points, and CI stages/runners. Apportion existing time estimates
|
||||
across split files; do not duplicate the original budget for every output file
|
||||
or silently drop a registration. Follow `write-sglang-test` for CI registration.
|
||||
|
||||
## Verify a migration
|
||||
|
||||
1. Search all runtime, test, benchmark, documentation, patch-string, and lazy
|
||||
registry references before deleting the old path. Check relative imports and
|
||||
package re-exports as well as direct imports. Do not leave forwarding shims.
|
||||
2. Keep tuning files with the GEMM that loads them and preserve relative lookup
|
||||
behavior. Check wheel/package inclusion as well as source-tree execution.
|
||||
3. Preserve module singleton state: every runtime caller must import the same
|
||||
new communication adapter, not a second copy of its buffer registry.
|
||||
4. Separate relocation commits from semantic changes. Use
|
||||
`mechanical-refactor-verify` to reproduce moves, and compare test inventories
|
||||
and CI registration coverage before/after. Only delete an experimental path
|
||||
after checking its call sites, flags, source, and dedicated tests together.
|
||||
5. Run the namespace/dispatch CPU tests, the registered-test validation hook,
|
||||
pre-commit, and relevant GPU tests when available. State exactly which GPU
|
||||
checks ran; import/AST checks do not prove numerical or performance parity.
|
||||
|
||||
Do not infer violations solely from a model name inside a group, a missing
|
||||
`_jit_` prefix, use of a runtime utility, or absence of `BaseFusedOp` inheritance.
|
||||
@@ -1,249 +0,0 @@
|
||||
---
|
||||
name: processor-model-parity
|
||||
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
|
||||
---
|
||||
|
||||
# Processor model parity
|
||||
|
||||
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
|
||||
server, sgl-router, the renderer). A model is supported only when, for the same
|
||||
request body, the processor produces **the same prompt text and the same prompt
|
||||
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
|
||||
the implementation, and SGLang code exists only where the two differ.
|
||||
|
||||
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
|
||||
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
|
||||
Multimodal preprocessing is not covered here: `render_request` fails on media
|
||||
content parts, so hosts send those chats to the engine. The fixture generator
|
||||
serves every checkpoint text-only, so a media case records that rejection.
|
||||
|
||||
## Done means
|
||||
|
||||
- **All text should work**: every text chat the engine accepts renders and
|
||||
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
|
||||
the processor cannot reproduce goes to the engine; list it in the PR.
|
||||
- **Chat and completions go through the engine's `/generate`**, with the router
|
||||
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
|
||||
engine's OpenAI route is only the `Unsupported` fallback. That needs the
|
||||
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
|
||||
against a live engine.
|
||||
|
||||
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
|
||||
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
|
||||
to `rust/sglang-processor/` unless they start with `python/`.
|
||||
|
||||
## Where a model's code goes
|
||||
|
||||
| Piece | File | Needed when |
|
||||
|---|---|---|
|
||||
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
|
||||
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
|
||||
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
|
||||
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
|
||||
| README row | the `render` table in `README.md` | always |
|
||||
|
||||
No new test code: `tests/parity.rs` checks every fixture in the directory.
|
||||
|
||||
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
|
||||
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
|
||||
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
|
||||
renders the model:
|
||||
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
|
||||
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
|
||||
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
|
||||
`PromptFormatter`, replace that branch.
|
||||
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
|
||||
variant holding Dynamo's Jinja `PromptFormatter`, picked in
|
||||
`ChatFormatter::from_tokenizer_config` from the template text (as
|
||||
`parser/template_detection.py` picks it, where it has a detector), and build the
|
||||
template context that `_process_messages` and `_apply_jinja_template` build, like
|
||||
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
|
||||
rewrites shapes its load-time probes find the template rejects (a later system
|
||||
turn, for one); SGLang raises on them, so check those before rendering.
|
||||
|
||||
## Method
|
||||
|
||||
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
|
||||
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
|
||||
Note whether its tokenizer ships a `chat_template`; that can change the spec.
|
||||
1. **Trace the Python path for the model.** Write down every transformation
|
||||
between the HTTP body and `prompt_ids`, in order:
|
||||
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
|
||||
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
|
||||
Reuse `render/reasoning.rs` for this rather than porting it again.
|
||||
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
|
||||
replaces the request effort; server `--default-chat-template-kwargs` merge in.
|
||||
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
|
||||
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
|
||||
It also reads the server's `--tool-call-parser` and whether the tokenizer has
|
||||
a chat template. If the deployment depends on the parser, set
|
||||
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
|
||||
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
|
||||
effort profile reads `encoding/encoding_dsv4.py`).
|
||||
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
|
||||
message dumping, content flattening, `_handle_last_assistant_message`, system
|
||||
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
|
||||
call, and how the prompt is tokenized.
|
||||
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
|
||||
rules live there: dropped turns, `</think>` on empty reasoning, argument
|
||||
formatting, and the errors it raises.
|
||||
|
||||
This list is the **divergence inventory**.
|
||||
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
|
||||
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
|
||||
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
|
||||
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
|
||||
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
|
||||
which often differ from SGLang's. Diff the low-level encoder against Python's
|
||||
for the rules in step 1.
|
||||
3. **Write one request case per inventory item** in the fixture (`name` +
|
||||
`request` only), then generate the expected outputs from Python (below). Cover
|
||||
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
|
||||
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
|
||||
each failing case, add the smallest SGLang step that fixes it, with a one-line
|
||||
doc comment naming the Python source it mirrors. Never patch the fixture to
|
||||
match Rust.
|
||||
5. **Verify** text offline, then token ids with the checkpoint cached (commands
|
||||
below). If sgl-router or the renderer uses the formatter, run their suites too.
|
||||
|
||||
## Checklist (DeepSeek-V4 case that covers each)
|
||||
|
||||
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
|
||||
reduced to role and content; null content becomes `""`; history tool calls keep
|
||||
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
|
||||
- **Content parts**: which part types survive and the join separator. (`parts`)
|
||||
- **Tools**: all request tools or only those `tool_choice` allows; field order
|
||||
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
|
||||
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
|
||||
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
|
||||
`tool_schema_type_aliases`)
|
||||
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
|
||||
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
|
||||
- **Pydantic coercion**: typed request fields are coerced before rendering. For
|
||||
example, `strict: 1` and `defer_loading: "false"` become booleans, and
|
||||
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
|
||||
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
|
||||
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
|
||||
JSON string; other encoders format them their own way. Keep key order
|
||||
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
|
||||
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
|
||||
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
|
||||
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
|
||||
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
|
||||
truthiness (`1` is on); strings use Python's yes-word set.
|
||||
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
|
||||
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
|
||||
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
|
||||
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
|
||||
a separately tokenized prefix. Check whether content is flattened before the
|
||||
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
|
||||
assistant turn's tool calls are discarded, so object checks on their arguments
|
||||
belong after the split. Hosts reach the model through `render_prompt` too, so
|
||||
check their continuation handling as well.
|
||||
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
|
||||
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
|
||||
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
|
||||
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
|
||||
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
|
||||
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
|
||||
continuation prefix is encoded alone and loses a leading BOS
|
||||
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
|
||||
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
|
||||
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
|
||||
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
|
||||
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
|
||||
includes pydantic rejections and Python crashes (a 500 is still a rejection).
|
||||
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
|
||||
PR that hosts fall back to the engine for it.
|
||||
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
|
||||
reach, fix it upstream and mark the case `known_gap` with the reason. The test
|
||||
reports the case without failing, and fails once it matches so the marker gets
|
||||
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
|
||||
|
||||
## Generating fixtures
|
||||
|
||||
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
|
||||
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
|
||||
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
|
||||
renders it through the real `_process_messages`, under the case's
|
||||
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
|
||||
writes:
|
||||
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
|
||||
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
|
||||
are kept as written). An `AttributeError` aborts instead: the stub server lacks
|
||||
something, so extend it;
|
||||
- the fixture-level `config` (the `config.json` fields the processor reads, plus
|
||||
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
|
||||
checkpoint its `chat_template`, so the test renders offline.
|
||||
|
||||
```sh
|
||||
cd rust/sglang-processor
|
||||
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
|
||||
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
|
||||
```
|
||||
|
||||
When a new spec needs more, extend `serving_chat()` in the script (one place):
|
||||
- **More server state:** set the extra attribute that `__init__` resolves, for
|
||||
example `_dsv41_default_reasoning_effort`.
|
||||
|
||||
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
|
||||
recorded by the template's text output. Its `render_request` returns the
|
||||
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
|
||||
|
||||
Pin `revision` to a commit hash and regenerate only on purpose.
|
||||
|
||||
## Verify
|
||||
|
||||
```sh
|
||||
cd rust
|
||||
cargo test -p sglang-processor --locked # text, offline
|
||||
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
|
||||
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
|
||||
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
|
||||
```
|
||||
|
||||
The token check loads the pinned commit's snapshot from the cache. When the
|
||||
snapshot is missing, the test prints `token ids not checked` (visible with
|
||||
`-- --nocapture`), so confirm that line is absent before claiming token parity.
|
||||
With the snapshot cached, the test also checks that the checkpoint resolves the
|
||||
fixture's recorded DeepSeek-V4 profile.
|
||||
|
||||
## OpenAI layer
|
||||
|
||||
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
|
||||
once its render parity holds. Two more fixtures then cover the rest of Python's
|
||||
OpenAI layer:
|
||||
- `tests/fixtures/reasoning_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
|
||||
chunked outputs. A parser is served once `src/think/models/` ports it.
|
||||
- `tests/fixtures/tool_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
|
||||
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
|
||||
parser is served once `src/tool_call/models/` ports it.
|
||||
- `tests/fixtures/openai_parity/<model-id>.json`, from
|
||||
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
|
||||
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
|
||||
output for it, and the response Python builds from that output. Launch the
|
||||
engine with the model's cookbook flags, including its parsers. Without the
|
||||
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
|
||||
fixture does.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
|
||||
`dynamo-renderer` or `dynamo-tokenizers`.
|
||||
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
|
||||
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
|
||||
sent, so such requests stay on the engine's route. Fixtures don't show this; a
|
||||
live engine does.
|
||||
- Python mutates the request before rendering (`_validate_request` rewrites tool
|
||||
schemas), so fixture generators drive the server's own entry points.
|
||||
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
|
||||
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
|
||||
generator pins them and `tests/parity.rs` clears them, so fixtures do not
|
||||
depend on the machine.
|
||||
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
|
||||
such as `task` and message-level `tools`. Parity is defined on `render_request`;
|
||||
report host-adapter gaps separately rather than bending the model code.
|
||||
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
|
||||
beyond the dispatch arm.
|
||||
@@ -1,21 +0,0 @@
|
||||
# Serving a model? Read its cookbook page first
|
||||
|
||||
Before running, deploying, benchmarking, or reproducing a specific model, read
|
||||
its page under `docs/cookbook/`:
|
||||
|
||||
- `docs/cookbook/autoregressive/` — LLM and VLM serving
|
||||
- `docs/cookbook/diffusion/` — image, video, 3D, robotics
|
||||
- `docs/cookbook/omni/`, `docs/cookbook/vla/`, `docs/cookbook/specbundle/`,
|
||||
`docs/cookbook/base/`
|
||||
|
||||
That page carries the deployment we recommend to users — GPU count, parallelism
|
||||
degrees, quantization, and the flags that matter for that model — so it is the
|
||||
answer to "how should this be served", and the baseline any tuning starts from.
|
||||
|
||||
Configs under `test/` are a different thing: pinned, reproducible CI setups for
|
||||
correctness and regression checks, not deployment advice. Reach for them when
|
||||
you need an exactly reproducible run, and say which cookbook flags you diverged
|
||||
from — a measurement taken on a configuration nobody deploys describes nothing.
|
||||
|
||||
Keep it true in both directions: when a change alters a model's recommended
|
||||
deployment, update that model's cookbook page in the same PR.
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
paths:
|
||||
- "**/bench_*.py"
|
||||
- "**/benchmark/**/*.py"
|
||||
- "benchmark/kernels/**"
|
||||
---
|
||||
|
||||
# Kernel benchmarks
|
||||
|
||||
How a benchmark is written is up to you. The requirement is only that the numbers are valid.
|
||||
|
||||
## Tools
|
||||
|
||||
In SGLang we recommend `marker.do_bench` (`python/sglang/kernels/jit/benchmark/marker.py`). It works for any callable, not only JIT kernels, and it measures with a cold L2 by default. It is not mandatory. CUPTI (e.g. CUDA 13 `cupti-python`), the torch profiler / `bench_kineto`, Triton's `do_bench` / proton, `flashinfer.testing`, Nsight Systems / Nsight Compute, or a hand-written loop are all fine. Each pitfall below must be handled, whichever tool you use.
|
||||
|
||||
## L2 cache reuse
|
||||
|
||||
Timing the same kernel on the same inputs in a loop measures an L2-hot kernel, not the one serving runs. `triton.testing.do_bench_cudagraph` with fixed arguments and bare `for` loops between two events both do this. Measure cold L2 by flushing between calls or by rotating enough input copies to exceed L2 several times over. Hot L2 is acceptable only when the working set far exceeds L2 or when hot L2 is the real serving pattern, and the reason must be stated.
|
||||
|
||||
Host-memory (zero-copy) loads do not go through device L2. Peer-GPU (NVLink) reads still see reuse (on remote GPU).
|
||||
@@ -6,5 +6,4 @@ Before modifying the following components, read the listed skill first.
|
||||
- **`Scheduler` / `TokenizerManager` / `ModelRunner` `__init__`** (`python/sglang/srt/managers/scheduler.py`, `python/sglang/srt/managers/tokenizer_manager.py`, `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Any edit to a frozen core file** (currently `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Environment variables** (adding, renaming, or reviewing any `SGLANG_*` env var, migrating a legacy `SGL_*` alias, or touching `python/sglang/srt/environ.py`) → [`env-var-conventions`](../skills/env-var-conventions/SKILL.md)
|
||||
- **Chat rendering in `rust/sglang-processor`** (adding or changing a model under `rust/sglang-processor/src/render/`, or its parity fixtures) → [`processor-model-parity`](../skills/processor-model-parity/SKILL.md)
|
||||
- **Scripted runtime** (anything related to the scripted runtime) → [`scripted-runtime-notes`](../skills/scripted-runtime-notes/SKILL.md)
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Accuracy evals: use sgl-eval
|
||||
|
||||
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
|
||||
model's accuracy.
|
||||
|
||||
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
|
||||
server's env, at the version pinned in the `test` extra of
|
||||
`python/pyproject.toml`.
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
|
||||
```
|
||||
|
||||
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
|
||||
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
|
||||
|
||||
- **Sampling:** use the model's recommended settings (temperature, top_p,
|
||||
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
|
||||
them with its own defaults, so pass them explicitly, or use
|
||||
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
|
||||
sets the `thinking` chat-template key; if the model's template reads a
|
||||
different one (commonly `enable_thinking`), pass that key with
|
||||
`--chat-template-kwarg` instead. When comparing against a CI test, use that
|
||||
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
|
||||
|
||||
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
|
||||
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
|
||||
base models, since sgl-eval only sends chat requests.
|
||||
|
||||
Only compare scores from the same harness and settings, and say which ones you
|
||||
used when you report a number.
|
||||
@@ -1 +0,0 @@
|
||||
../.agents/skills
|
||||
@@ -1,14 +1,10 @@
|
||||
---
|
||||
name: add-jit-kernel
|
||||
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang.kernels JIT infrastructure and public operator groups
|
||||
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
|
||||
---
|
||||
|
||||
# Tutorial: Adding a New JIT Kernel to SGLang
|
||||
|
||||
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||
The implementation tutorial below does not replace that API contract.
|
||||
|
||||
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||
|
||||
## Goal
|
||||
@@ -501,32 +497,6 @@ def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) ->
|
||||
|
||||
---
|
||||
|
||||
## Register the public entry point (required)
|
||||
|
||||
Add metadata in `python/sglang/kernels/ops/elementwise/__init__.py` so the
|
||||
operator appears in the registry without importing its implementation:
|
||||
|
||||
```python
|
||||
from sglang.kernels.registry import register_kernel
|
||||
from sglang.kernels.spec import CapabilityRequirement, KernelBackend, KernelSpec
|
||||
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="elementwise.scale",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.elementwise.scale:scale",
|
||||
capabilities=frozenset({CapabilityRequirement.CUDA}),
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
Record the devices the implementation actually supports; the CUDA-only scale
|
||||
example above does not imply that every JIT kernel is CUDA-only. Runtime and
|
||||
correctness tests import `scale` from `sglang.kernels.ops.elementwise.scale`.
|
||||
Do not import the implementation eagerly in the group initializer. When an
|
||||
existing `BaseFusedOp` owns the operation, add the implementation there instead
|
||||
of registering a second conflicting op/backend pair.
|
||||
|
||||
## Step 3 (optional): Tune JIT build flags
|
||||
|
||||
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
|
||||
@@ -641,8 +611,6 @@ if __name__ == "__main__":
|
||||
|
||||
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
|
||||
|
||||
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
|
||||
|
||||
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
|
||||
|
||||
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
|
||||
@@ -651,7 +619,7 @@ Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/j
|
||||
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
|
||||
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
|
||||
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
|
||||
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"`; if you narrow it, keep every read, in-place, and output tensor.
|
||||
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning).
|
||||
- `use_cuda_graph=False` for kernels that can't be captured.
|
||||
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
|
||||
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
|
||||
@@ -756,7 +724,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh` — minimal runnable reference
|
||||
- `python/sglang/kernels/jit/csrc/add_constant.cuh` — minimal runnable reference
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
|
||||
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
|
||||
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
|
||||
@@ -5,10 +5,6 @@ description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel
|
||||
|
||||
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
|
||||
|
||||
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
|
||||
operator namespace, logical grouping, lazy registry metadata, and test placement.
|
||||
The implementation tutorial below does not replace that API contract.
|
||||
|
||||
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
|
||||
|
||||
## Goal
|
||||
@@ -29,7 +25,7 @@ Add a new operation that scales each element of a tensor by a scalar factor:
|
||||
In addition, every new kernel must ship with:
|
||||
|
||||
- **Tests** (pytest)
|
||||
- **A benchmark script** (`marker.do_bench` from `sglang.kernels.jit.benchmark` — it works for any callable, not only JIT kernels)
|
||||
- **A benchmark script** (triton.testing)
|
||||
|
||||
---
|
||||
|
||||
@@ -198,21 +194,6 @@ Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
|
||||
|
||||
---
|
||||
|
||||
## SGLang integration (required for runtime use)
|
||||
|
||||
After exposing the AOT wheel symbol, add a lazy wrapper and a `KernelSpec` under
|
||||
`python/sglang/kernels/ops/<group>/`. Set `backend=KernelBackend.AOT`, record
|
||||
actual device/architecture support, and keep `sgl_kernel` imports inside the
|
||||
implementation path. SGLang runtime and integration tests import that wrapper.
|
||||
For this example, use `sglang.kernels.ops.elementwise.scale`.
|
||||
|
||||
The wheel-level tests below validate its standalone API/build. They do not
|
||||
replace CI-registered SGLang correctness tests in
|
||||
`test/registered/kernels/ops/elementwise/test_scale.py` and benchmarks in
|
||||
`test/registered/kernels/benchmark/elementwise/bench_scale.py`. Follow
|
||||
`write-sglang-test` for their registration and CI budget; do not register tests
|
||||
under the shipped `python/sglang/` package.
|
||||
|
||||
## Step 6: Write tests (required)
|
||||
|
||||
Create `python/sglang/kernels/aot/tests/test_scale.py`:
|
||||
@@ -260,46 +241,62 @@ if __name__ == "__main__":
|
||||
|
||||
## Step 7: Add a benchmark (required)
|
||||
|
||||
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
|
||||
|
||||
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
|
||||
|
||||
```python
|
||||
import itertools
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.testing
|
||||
|
||||
import sgl_kernel
|
||||
from sglang.kernels.jit.benchmark import marker
|
||||
from sglang.utils import is_in_ci
|
||||
|
||||
IS_CI = is_in_ci()
|
||||
|
||||
dtypes = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
|
||||
sizes = [4096] if IS_CI else [2**n for n in range(10, 20)] # 1K … 512K
|
||||
factors = [2.0]
|
||||
|
||||
configs = list(itertools.product(dtypes, sizes))
|
||||
|
||||
|
||||
def sglang_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
|
||||
sgl_kernel.scale(input, factor, out=out)
|
||||
def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
|
||||
return input * factor
|
||||
|
||||
|
||||
def torch_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
|
||||
torch.mul(input, factor, out=out)
|
||||
|
||||
|
||||
FN_MAP = {"sglang": sglang_scale, "torch": torch_scale}
|
||||
|
||||
|
||||
@marker.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32], [torch.float16])
|
||||
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096]) # 1K .. 512K
|
||||
@marker.benchmark("provider", ["sglang", "torch"])
|
||||
def benchmark(dtype: torch.dtype, size: int, provider: str):
|
||||
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(input)
|
||||
return marker.do_bench(
|
||||
FN_MAP[provider],
|
||||
# Pass every tensor through input_args (not a closure) so marker rotates
|
||||
# them across CUDA-graph calls to defeat L2 reuse.
|
||||
input_args=(input, 2.0, out),
|
||||
# Bandwidth = bytes(input) + bytes(out), both already in input_args.
|
||||
memory_output=None,
|
||||
@triton.testing.perf_report(
|
||||
triton.testing.Benchmark(
|
||||
x_names=["dtype", "size"],
|
||||
x_vals=configs,
|
||||
line_arg="provider",
|
||||
line_vals=["sglang", "torch"],
|
||||
line_names=["SGL Kernel", "PyTorch"],
|
||||
styles=[("green", "-"), ("red", "--")],
|
||||
ylabel="µs (median)",
|
||||
plot_name="scale-performance",
|
||||
args={},
|
||||
)
|
||||
)
|
||||
def benchmark(dtype, size, provider):
|
||||
input = torch.randn(size, dtype=dtype, device="cuda")
|
||||
out = torch.empty_like(input)
|
||||
factor = 2.0
|
||||
|
||||
if provider == "sglang":
|
||||
fn = lambda: sgl_kernel.scale(input, factor, out=out)
|
||||
else:
|
||||
fn = lambda: torch_scale(input, factor)
|
||||
|
||||
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
|
||||
fn, quantiles=[0.5, 0.2, 0.8]
|
||||
)
|
||||
return 1000 * ms, 1000 * max_ms, 1000 * min_ms
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
benchmark.run()
|
||||
benchmark.run(print_data=True)
|
||||
```
|
||||
|
||||
---
|
||||
+3
-3
@@ -167,7 +167,7 @@ describes what each one does.
|
||||
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
|
||||
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
|
||||
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
|
||||
|
||||
**Job specs example** (base-b):
|
||||
```json
|
||||
@@ -378,7 +378,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
|---------|-------------|---------------|
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
|
||||
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
|
||||
@@ -396,7 +396,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
|
||||
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
|
||||
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||
+1
-1
@@ -264,5 +264,5 @@ git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs/docs.json docs/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
|
||||
gh pr create --title "Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
+3
-3
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||
then parallelism (`--tp`, `--attn-dp-size`), then MoE
|
||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
knobs/values that are disabled under the live facts, so stale picks never
|
||||
emit a blocked combination. Interleave prefill-CP + DP-Attention is
|
||||
deliberately NOT grayed (combined support is planned upstream even though
|
||||
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
|
||||
current releases assert `dp_size == 1` for interleave) — the engine shows
|
||||
a warning hint below the command box instead.
|
||||
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
|
||||
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
|
||||
+5
-3
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
|
||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
@@ -357,7 +357,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--attn-dp-size 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
@@ -402,7 +403,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 16",
|
||||
"--attn-dp-size 16",
|
||||
"--dp 16",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
+2
-2
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
|
||||
|
||||
## 1. Quick start
|
||||
|
||||
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
|
||||
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
|
||||
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
|
||||
a verified recipe below.
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
+1
-1
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
|
||||
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||
even when its flags are uncoupled (`--attn-dp-size N` is a pure
|
||||
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
|
||||
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||
Playground axis with the flags baked into cells when the legacy default was
|
||||
ON — EXCEPT parsers:
|
||||
+2
-2
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
|
||||
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
||||
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
|
||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
|
||||
Caveats discovered in the pilot:
|
||||
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
||||
+2
-2
@@ -136,8 +136,8 @@ than restating.
|
||||
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
|
||||
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||
column must be produced by the SAME harness — flag a page that, say, measures one
|
||||
platform's GSM8K with a historical raw completion harness and another's with
|
||||
`sgl-eval run gsm8k --num-examples 1319` and shows both as one "GSM8K %"
|
||||
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
|
||||
`run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
|
||||
(the scores aren't comparable). Either standardize on one harness (matching
|
||||
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
|
||||
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
|
||||
+1
-1
@@ -162,7 +162,7 @@ Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only f
|
||||
For a rename where the old key must keep working with a warning:
|
||||
|
||||
```python
|
||||
SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME")
|
||||
SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK")
|
||||
```
|
||||
|
||||
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
|
||||
@@ -11,7 +11,6 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
|
||||
|
||||
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||
- At least one available CUDA GPU
|
||||
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
|
||||
|
||||
## Step-by-step Workflow
|
||||
|
||||
@@ -47,11 +46,13 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
|
||||
### Step 3: Validate accuracy (sanity check)
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
|
||||
```
|
||||
|
||||
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
|
||||
- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
### Step 4: Generate the profile
|
||||
@@ -109,8 +110,8 @@ for i in $(seq 1 120); do
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
|
||||
# Expected: score > 80%
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
|
||||
# Expected: Accuracy > 0.8
|
||||
|
||||
# 4. Profile
|
||||
python3 -m sglang.test.send_one --profile
|
||||
+7
-7
@@ -46,8 +46,8 @@ in-flight row as shipped.
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||
@@ -70,8 +70,8 @@ in-flight row as shipped.
|
||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||
@@ -148,8 +148,8 @@ Stable entries should be folded into the mainline family rows above.
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
|
||||
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
|
||||
|
||||
@@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package.
|
||||
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
|
||||
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
|
||||
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
|
||||
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
||||
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
||||
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
|
||||
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
|
||||
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
|
||||
+3
-3
@@ -391,7 +391,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
|
||||
"<br>python/sglang/srt/layers/layernorm.py"
|
||||
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||
"<br>python/sglang/srt/layers/communicator.py"
|
||||
),
|
||||
active_keywords=(
|
||||
"allreduce_fusion",
|
||||
@@ -419,7 +419,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
pattern="AITER allreduce fusion",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/distributed/communication_op.py"
|
||||
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||
"<br>python/sglang/srt/layers/communicator.py"
|
||||
"<br>python/sglang/srt/layers/layernorm.py"
|
||||
),
|
||||
active_keywords=(
|
||||
@@ -721,7 +721,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
FusionPatternSpec(
|
||||
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
|
||||
"python/sglang/srt/layers/communicator.py"
|
||||
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
||||
"forward_mla.py"
|
||||
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user