Compare commits
44
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4e60127fb4 | ||
|
|
e9fdf3ad0a | ||
|
|
2087760ea8 | ||
|
|
ef1a5955f3 | ||
|
|
7e17dee8b7 | ||
|
|
0090df5d64 | ||
|
|
4938eea2dc | ||
|
|
8c6cbd8c40 | ||
|
|
c82aed9fef | ||
|
|
e6f6420275 | ||
|
|
566176d0cf | ||
|
|
4f52678224 | ||
|
|
068acff09d | ||
|
|
f9056f0fcd | ||
|
|
2aae0cb4ee | ||
|
|
ea5fc832c1 | ||
|
|
9894dc789c | ||
|
|
77e7555ae3 | ||
|
|
319ee3c056 | ||
|
|
e59014be87 | ||
|
|
fd86688428 | ||
|
|
da343ba5d1 | ||
|
|
9ea02a7201 | ||
|
|
6c1c54b514 | ||
|
|
51624199b4 | ||
|
|
31b3758c99 | ||
|
|
740372bef0 | ||
|
|
19fe20baec | ||
|
|
383df91fcc | ||
|
|
534842ac08 | ||
|
|
4dd95a8817 | ||
|
|
656e00dcd3 | ||
|
|
a4b1363bf4 | ||
|
|
ad2acaa2df | ||
|
|
ecaaa37cb0 | ||
|
|
1b70fc192c | ||
|
|
3ebbb2d536 | ||
|
|
5d52cccbee | ||
|
|
74d03143e2 | ||
|
|
2b192aade6 | ||
|
|
9247c97919 | ||
|
|
5e7190b87a | ||
|
|
e00930c548 | ||
|
|
d7ff74b984 |
@@ -1,249 +0,0 @@
|
||||
---
|
||||
name: processor-model-parity
|
||||
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
|
||||
---
|
||||
|
||||
# Processor model parity
|
||||
|
||||
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
|
||||
server, sgl-router, the renderer). A model is supported only when, for the same
|
||||
request body, the processor produces **the same prompt text and the same prompt
|
||||
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
|
||||
the implementation, and SGLang code exists only where the two differ.
|
||||
|
||||
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
|
||||
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
|
||||
Multimodal preprocessing is not covered here: `render_request` fails on media
|
||||
content parts, so hosts send those chats to the engine. The fixture generator
|
||||
serves every checkpoint text-only, so a media case records that rejection.
|
||||
|
||||
## Done means
|
||||
|
||||
- **All text should work**: every text chat the engine accepts renders and
|
||||
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
|
||||
the processor cannot reproduce goes to the engine; list it in the PR.
|
||||
- **Chat and completions go through the engine's `/generate`**, with the router
|
||||
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
|
||||
engine's OpenAI route is only the `Unsupported` fallback. That needs the
|
||||
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
|
||||
against a live engine.
|
||||
|
||||
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
|
||||
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
|
||||
to `rust/sglang-processor/` unless they start with `python/`.
|
||||
|
||||
## Where a model's code goes
|
||||
|
||||
| Piece | File | Needed when |
|
||||
|---|---|---|
|
||||
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
|
||||
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
|
||||
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
|
||||
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
|
||||
| README row | the `render` table in `README.md` | always |
|
||||
|
||||
No new test code: `tests/parity.rs` checks every fixture in the directory.
|
||||
|
||||
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
|
||||
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
|
||||
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
|
||||
renders the model:
|
||||
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
|
||||
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
|
||||
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
|
||||
`PromptFormatter`, replace that branch.
|
||||
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
|
||||
variant holding Dynamo's Jinja `PromptFormatter`, picked in
|
||||
`ChatFormatter::from_tokenizer_config` from the template text (as
|
||||
`parser/template_detection.py` picks it, where it has a detector), and build the
|
||||
template context that `_process_messages` and `_apply_jinja_template` build, like
|
||||
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
|
||||
rewrites shapes its load-time probes find the template rejects (a later system
|
||||
turn, for one); SGLang raises on them, so check those before rendering.
|
||||
|
||||
## Method
|
||||
|
||||
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
|
||||
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
|
||||
Note whether its tokenizer ships a `chat_template`; that can change the spec.
|
||||
1. **Trace the Python path for the model.** Write down every transformation
|
||||
between the HTTP body and `prompt_ids`, in order:
|
||||
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
|
||||
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
|
||||
Reuse `render/reasoning.rs` for this rather than porting it again.
|
||||
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
|
||||
replaces the request effort; server `--default-chat-template-kwargs` merge in.
|
||||
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
|
||||
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
|
||||
It also reads the server's `--tool-call-parser` and whether the tokenizer has
|
||||
a chat template. If the deployment depends on the parser, set
|
||||
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
|
||||
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
|
||||
effort profile reads `encoding/encoding_dsv4.py`).
|
||||
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
|
||||
message dumping, content flattening, `_handle_last_assistant_message`, system
|
||||
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
|
||||
call, and how the prompt is tokenized.
|
||||
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
|
||||
rules live there: dropped turns, `</think>` on empty reasoning, argument
|
||||
formatting, and the errors it raises.
|
||||
|
||||
This list is the **divergence inventory**.
|
||||
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
|
||||
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
|
||||
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
|
||||
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
|
||||
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
|
||||
which often differ from SGLang's. Diff the low-level encoder against Python's
|
||||
for the rules in step 1.
|
||||
3. **Write one request case per inventory item** in the fixture (`name` +
|
||||
`request` only), then generate the expected outputs from Python (below). Cover
|
||||
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
|
||||
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
|
||||
each failing case, add the smallest SGLang step that fixes it, with a one-line
|
||||
doc comment naming the Python source it mirrors. Never patch the fixture to
|
||||
match Rust.
|
||||
5. **Verify** text offline, then token ids with the checkpoint cached (commands
|
||||
below). If sgl-router or the renderer uses the formatter, run their suites too.
|
||||
|
||||
## Checklist (DeepSeek-V4 case that covers each)
|
||||
|
||||
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
|
||||
reduced to role and content; null content becomes `""`; history tool calls keep
|
||||
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
|
||||
- **Content parts**: which part types survive and the join separator. (`parts`)
|
||||
- **Tools**: all request tools or only those `tool_choice` allows; field order
|
||||
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
|
||||
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
|
||||
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
|
||||
`tool_schema_type_aliases`)
|
||||
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
|
||||
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
|
||||
- **Pydantic coercion**: typed request fields are coerced before rendering. For
|
||||
example, `strict: 1` and `defer_loading: "false"` become booleans, and
|
||||
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
|
||||
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
|
||||
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
|
||||
JSON string; other encoders format them their own way. Keep key order
|
||||
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
|
||||
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
|
||||
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
|
||||
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
|
||||
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
|
||||
truthiness (`1` is on); strings use Python's yes-word set.
|
||||
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
|
||||
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
|
||||
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
|
||||
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
|
||||
a separately tokenized prefix. Check whether content is flattened before the
|
||||
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
|
||||
assistant turn's tool calls are discarded, so object checks on their arguments
|
||||
belong after the split. Hosts reach the model through `render_prompt` too, so
|
||||
check their continuation handling as well.
|
||||
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
|
||||
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
|
||||
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
|
||||
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
|
||||
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
|
||||
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
|
||||
continuation prefix is encoded alone and loses a leading BOS
|
||||
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
|
||||
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
|
||||
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
|
||||
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
|
||||
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
|
||||
includes pydantic rejections and Python crashes (a 500 is still a rejection).
|
||||
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
|
||||
PR that hosts fall back to the engine for it.
|
||||
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
|
||||
reach, fix it upstream and mark the case `known_gap` with the reason. The test
|
||||
reports the case without failing, and fails once it matches so the marker gets
|
||||
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
|
||||
|
||||
## Generating fixtures
|
||||
|
||||
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
|
||||
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
|
||||
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
|
||||
renders it through the real `_process_messages`, under the case's
|
||||
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
|
||||
writes:
|
||||
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
|
||||
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
|
||||
are kept as written). An `AttributeError` aborts instead: the stub server lacks
|
||||
something, so extend it;
|
||||
- the fixture-level `config` (the `config.json` fields the processor reads, plus
|
||||
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
|
||||
checkpoint its `chat_template`, so the test renders offline.
|
||||
|
||||
```sh
|
||||
cd rust/sglang-processor
|
||||
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
|
||||
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
|
||||
```
|
||||
|
||||
When a new spec needs more, extend `serving_chat()` in the script (one place):
|
||||
- **More server state:** set the extra attribute that `__init__` resolves, for
|
||||
example `_dsv41_default_reasoning_effort`.
|
||||
|
||||
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
|
||||
recorded by the template's text output. Its `render_request` returns the
|
||||
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
|
||||
|
||||
Pin `revision` to a commit hash and regenerate only on purpose.
|
||||
|
||||
## Verify
|
||||
|
||||
```sh
|
||||
cd rust
|
||||
cargo test -p sglang-processor --locked # text, offline
|
||||
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
|
||||
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
|
||||
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
|
||||
```
|
||||
|
||||
The token check loads the pinned commit's snapshot from the cache. When the
|
||||
snapshot is missing, the test prints `token ids not checked` (visible with
|
||||
`-- --nocapture`), so confirm that line is absent before claiming token parity.
|
||||
With the snapshot cached, the test also checks that the checkpoint resolves the
|
||||
fixture's recorded DeepSeek-V4 profile.
|
||||
|
||||
## OpenAI layer
|
||||
|
||||
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
|
||||
once its render parity holds. Two more fixtures then cover the rest of Python's
|
||||
OpenAI layer:
|
||||
- `tests/fixtures/reasoning_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
|
||||
chunked outputs. A parser is served once `src/think/models/` ports it.
|
||||
- `tests/fixtures/tool_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
|
||||
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
|
||||
parser is served once `src/tool_call/models/` ports it.
|
||||
- `tests/fixtures/openai_parity/<model-id>.json`, from
|
||||
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
|
||||
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
|
||||
output for it, and the response Python builds from that output. Launch the
|
||||
engine with the model's cookbook flags, including its parsers. Without the
|
||||
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
|
||||
fixture does.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
|
||||
`dynamo-renderer` or `dynamo-tokenizers`.
|
||||
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
|
||||
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
|
||||
sent, so such requests stay on the engine's route. Fixtures don't show this; a
|
||||
live engine does.
|
||||
- Python mutates the request before rendering (`_validate_request` rewrites tool
|
||||
schemas), so fixture generators drive the server's own entry points.
|
||||
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
|
||||
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
|
||||
generator pins them and `tests/parity.rs` clears them, so fixtures do not
|
||||
depend on the machine.
|
||||
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
|
||||
such as `task` and message-level `tools`. Parity is defined on `render_request`;
|
||||
report host-adapter gaps separately rather than bending the model code.
|
||||
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
|
||||
beyond the dispatch arm.
|
||||
@@ -6,5 +6,4 @@ Before modifying the following components, read the listed skill first.
|
||||
- **`Scheduler` / `TokenizerManager` / `ModelRunner` `__init__`** (`python/sglang/srt/managers/scheduler.py`, `python/sglang/srt/managers/tokenizer_manager.py`, `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Any edit to a frozen core file** (currently `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Environment variables** (adding, renaming, or reviewing any `SGLANG_*` env var, migrating a legacy `SGL_*` alias, or touching `python/sglang/srt/environ.py`) → [`env-var-conventions`](../skills/env-var-conventions/SKILL.md)
|
||||
- **Chat rendering in `rust/sglang-processor`** (adding or changing a model under `rust/sglang-processor/src/render/`, or its parity fixtures) → [`processor-model-parity`](../skills/processor-model-parity/SKILL.md)
|
||||
- **Scripted runtime** (anything related to the scripted runtime) → [`scripted-runtime-notes`](../skills/scripted-runtime-notes/SKILL.md)
|
||||
|
||||
@@ -36,4 +36,4 @@ a real `AttributeError` if the field is ever renamed:
|
||||
revision=getattr(server_args, "revision", None), # BAD
|
||||
revision=server_args.revision, # GOOD
|
||||
```
|
||||
(see `python/sglang/srt/parser/template_detection.py`)
|
||||
(see `python/sglang/srt/managers/template_detection.py`)
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Accuracy evals: use sgl-eval
|
||||
|
||||
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
|
||||
model's accuracy.
|
||||
|
||||
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
|
||||
server's env, at the version pinned in the `test` extra of
|
||||
`python/pyproject.toml`.
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
|
||||
```
|
||||
|
||||
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
|
||||
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
|
||||
|
||||
- **Sampling:** use the model's recommended settings (temperature, top_p,
|
||||
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
|
||||
them with its own defaults, so pass them explicitly, or use
|
||||
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
|
||||
sets the `thinking` chat-template key; if the model's template reads a
|
||||
different one (commonly `enable_thinking`), pass that key with
|
||||
`--chat-template-kwarg` instead. When comparing against a CI test, use that
|
||||
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
|
||||
|
||||
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
|
||||
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
|
||||
base models, since sgl-eval only sends chat requests.
|
||||
|
||||
Only compare scores from the same harness and settings, and say which ones you
|
||||
used when you report a number.
|
||||
@@ -27,8 +27,8 @@ A new unit test case must fall into one of these categories:
|
||||
3. **Critical-path bookkeeping.** Defends conventions that are easy to break by
|
||||
forgetting to sync -- registry completeness, field lifecycle, serialization
|
||||
compatibility. Enumerating assertions are fine here; the guarded failure
|
||||
mode is "someone extended X without updating Y". Example:
|
||||
`test/registered/unit/test_global_config_read_ratchet.py`.
|
||||
mode is "someone extended X without updating Y". Example: the ratchet tests
|
||||
(`test/registered/unit/test_module_state_ratchet.py`).
|
||||
|
||||
Not admissible:
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
../.agents/skills
|
||||
@@ -742,7 +742,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
|
||||
- `docs/docs/developer_guide/development_jit_kernel_guide.mdx`
|
||||
- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
|
||||
- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
|
||||
- `python/sglang/kernels/jit/utils/compile/` — `load_jit` (`loader.py`), `make_cpp_args` (`cpp_args.py`)
|
||||
- `python/sglang/kernels/jit/utils/compile.py` — `load_jit`, `make_cpp_args`
|
||||
- `python/sglang/kernels/jit/utils/common.py` — `cache_once`, `should_run_full_tests`, `get_ci_test_range`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`, `is_type`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/ffi.h` — `ffi::empty`, `ffi::empty_like`, `ffi::from_blob`
|
||||
+3
-4
@@ -167,7 +167,7 @@ describes what each one does.
|
||||
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
|
||||
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
|
||||
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
|
||||
|
||||
**Job specs example** (base-b):
|
||||
```json
|
||||
@@ -378,7 +378,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
|---------|-------------|---------------|
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
|
||||
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
|
||||
@@ -396,9 +396,8 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
|
||||
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
| `/run-cpu-ci` | Dispatches `run-cpu-ci.yml` on main: only the `base-a-test-cpu` suite, at the PR head SHA, no caches saved. Same gate as `/rerun-test`. Alias: `/rerun-cpu-ci` |
|
||||
|
||||
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||
|
||||
+1
-1
@@ -16,7 +16,7 @@ These are investigation hints, not a list of changes to apply.
|
||||
| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. |
|
||||
| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. |
|
||||
| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. |
|
||||
| Repeated template detection messages | `parser/template_detection.py` and `parser/template_manager.py` | Check whether a summary already contains the same information. |
|
||||
| Repeated template detection messages | `managers/template_detection.py` and `managers/template_manager.py` | Check whether a summary already contains the same information. |
|
||||
| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. |
|
||||
| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. |
|
||||
| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. |
|
||||
+1
-1
@@ -264,5 +264,5 @@ git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs/docs.json docs/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
|
||||
gh pr create --title "Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
+3
-3
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||
then parallelism (`--tp`, `--attn-dp-size`), then MoE
|
||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
knobs/values that are disabled under the live facts, so stale picks never
|
||||
emit a blocked combination. Interleave prefill-CP + DP-Attention is
|
||||
deliberately NOT grayed (combined support is planned upstream even though
|
||||
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
|
||||
current releases assert `dp_size == 1` for interleave) — the engine shows
|
||||
a warning hint below the command box instead.
|
||||
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
|
||||
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
|
||||
+5
-3
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
|
||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
@@ -357,7 +357,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--attn-dp-size 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
@@ -402,7 +403,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 16",
|
||||
"--attn-dp-size 16",
|
||||
"--dp 16",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
+2
-2
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
|
||||
|
||||
## 1. Quick start
|
||||
|
||||
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
|
||||
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
|
||||
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
|
||||
a verified recipe below.
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
+1
-1
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
|
||||
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||
even when its flags are uncoupled (`--attn-dp-size N` is a pure
|
||||
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
|
||||
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||
Playground axis with the flags baked into cells when the legacy default was
|
||||
ON — EXCEPT parsers:
|
||||
+2
-2
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
|
||||
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
||||
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
|
||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
|
||||
Caveats discovered in the pilot:
|
||||
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
||||
+1
-1
@@ -205,7 +205,7 @@ than restating.
|
||||
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
|
||||
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
|
||||
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
|
||||
(`python/sglang/srt/parser/template_detection.py`) actually resolve THIS model's
|
||||
(`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's
|
||||
chat template to the right parser — no rule match means auto silently disables the
|
||||
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
|
||||
|
||||
@@ -11,7 +11,6 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
|
||||
|
||||
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||
- At least one available CUDA GPU
|
||||
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
|
||||
|
||||
## Step-by-step Workflow
|
||||
|
||||
@@ -50,7 +49,7 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
|
||||
```
|
||||
|
||||
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
@@ -109,8 +108,8 @@ for i in $(seq 1 120); do
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
|
||||
# Expected: score > 80%
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
|
||||
# Expected: Accuracy > 0.8
|
||||
|
||||
# 4. Profile
|
||||
python3 -m sglang.test.send_one --profile
|
||||
+13
-36
@@ -17,47 +17,25 @@ logical grouping and remove the old `_jit_` filename prefix.
|
||||
The `sgl_kernel` wheel retains its own public API and packaging tests.
|
||||
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
|
||||
routing in `moe`, attention index selection in `attention`, sampling in
|
||||
`sampling`, normalization in `layernorm`.
|
||||
- Classify a fused operator by its complete contract; do not split a fused
|
||||
kernel into separate launches just to separate its parts. Split unrelated
|
||||
public entry points that happen to share a source file. Decided rules:
|
||||
- Any cross-rank collective or peer-memory access (all-reduce, all-gather,
|
||||
reduce-scatter, a2a, multicast/NVLS) puts the op in `communication`,
|
||||
whatever compute it fuses.
|
||||
- Producer + output quantization stays with the producer (norm + quant in
|
||||
`layernorm`, activation + quant in `activation`, GEMM with a quant epilogue
|
||||
in `gemm`, MoE dispatch/permute + quant in `moe`). `quantization` holds only
|
||||
pure format conversion: quant, dequant, scale-layout conversion, repack.
|
||||
- RoPE stays in `attention` (standalone RoPE, position ids, QK-norm + RoPE).
|
||||
- `kvcache` owns ops that read/write KV-cache contents; `attention` owns the
|
||||
index/shape/schedule tables (page tables, `cu_seqlens`, `kv_indices`, split
|
||||
metadata) and index selection. A fused compute + store stays with its
|
||||
compute group.
|
||||
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
|
||||
quantization alone belongs in `quantization`.
|
||||
- Model-specific files/subpackages and tuning data are allowed **inside** a
|
||||
logical group. Do not add a model bundle or an implementation file directly
|
||||
under `ops/`. Propose a new logical group only for a distinct responsibility,
|
||||
and update `ops._GROUPS`, the README group list, and tests together
|
||||
(`test_kernels_namespace.py` checks the README against `_GROUPS` and the
|
||||
README layout against the directories under `kernels/`).
|
||||
- Agent-generated implementations live in `kernels/kda_kernels/` (Kernel Design
|
||||
Agents, `KernelBackend.KDA`) and `kernels/cake_kernels/` (Cake adapters over
|
||||
FlashInfer). They are backends, not groups: register their public ops in the
|
||||
logical group with a `KernelSpec` whose target points into them, and do not
|
||||
reorganize those directories.
|
||||
- Keep shared build/runtime infrastructure (including the Helion runtime
|
||||
helper) in `kernels/jit`; operator wrappers call it from their logical group.
|
||||
Keep process groups, communicator state, workspace classes, model dispatch,
|
||||
and buffer ownership in `srt` (`srt/layers/communication/` for the K3 and Muse
|
||||
Spark adapters); kernels must not import srt runtime modules to inspect
|
||||
model objects, so pass flags in. Adapters there need not pretend to be generic
|
||||
interfaces.
|
||||
and update `ops._GROUPS`, documentation, and tests together.
|
||||
- Classify a fused operator by its complete contract. Do not split a fused
|
||||
kernel into separate launches just to separate norm, RoPE, or quantization.
|
||||
Split unrelated public entry points that happen to share a source file.
|
||||
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
|
||||
wrappers call it from their logical group. Keep process groups, communicator
|
||||
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
|
||||
`srt/layers/communication/` need not pretend to be generic interfaces.
|
||||
|
||||
## Register the public entry point
|
||||
|
||||
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
|
||||
existing `BaseFusedOp` registration when the operation has interchangeable
|
||||
backends (a `BaseFusedOp` built per layer registers `module:Class.forward_<backend>`
|
||||
targets). Group imports must not import GPU implementations or compile kernels.
|
||||
backends. Group imports must not import GPU implementations or compile kernels.
|
||||
|
||||
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
|
||||
existing backend dispatch and describe actual device/architecture restrictions
|
||||
@@ -75,9 +53,8 @@ factories, reference helpers, and runtime classes are not separate kernel APIs.
|
||||
runtime module. A CUDA allocation does not make a runtime test a kernel test.
|
||||
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
|
||||
standalone test/benchmark entry points in production operator modules.
|
||||
- Shared test helpers and reference oracles: `sglang.test.kernels`. Preserve
|
||||
vendored upstream trees and AOT wheel packaging boundaries instead of
|
||||
reorganizing them incidentally.
|
||||
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
|
||||
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
|
||||
|
||||
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
|
||||
execution entry points, and CI stages/runners. Apportion existing time estimates
|
||||
+8
-8
@@ -49,15 +49,15 @@ in-flight row as shipped.
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/kernels/ops/layernorm/fused_rmsnorm.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
||||
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
||||
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
||||
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
||||
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
||||
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
||||
@@ -69,18 +69,18 @@ in-flight row as shipped.
|
||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/kernels/ops/attention/linear/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
|
||||
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
|
||||
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/kernels/ops/attention/linear/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/kernels/ops/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
|
||||
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
|
||||
|
||||
@@ -146,7 +146,7 @@ Stable entries should be folded into the mainline family rows above.
|
||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
+4
-1
@@ -18,7 +18,7 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
||||
|
||||
## Scheduler-side trace writing
|
||||
|
||||
- `python/sglang/srt/managers/scheduler_components/profiler_manager.py`
|
||||
- `python/sglang/srt/managers/scheduler_profiler_mixin.py`
|
||||
- actual trace start/stop behavior
|
||||
- filename pattern for `TP/DP/PP/EP` and optional stage suffixes
|
||||
- `CUDA_PROFILER` and torch profiler handling
|
||||
@@ -37,3 +37,6 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
||||
|
||||
- `test/registered/profiling/test_start_profile.py`
|
||||
- validates `/start_profile` behavior, including `CUDA_PROFILER`
|
||||
|
||||
- `test/registered/profiling/test_profile_v2.py`
|
||||
- validates stage-scoped trace outputs under `SGLANG_PROFILE_V2`
|
||||
+4
-6
@@ -482,9 +482,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="Fused QK RoPE reshape + KV cache write",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py"
|
||||
),
|
||||
candidate_path="python/sglang/srt/layers/attention/utils.py",
|
||||
active_keywords=("fused_qk_rope_reshape_and_cache",),
|
||||
split_groups=(
|
||||
("rotary", "rope", "mrope"),
|
||||
@@ -535,7 +533,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused metadata copy for graph replay",
|
||||
candidate_path="python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py",
|
||||
candidate_path="python/sglang/kernels/ops/attention/fused_metadata_copy.py",
|
||||
active_keywords=(
|
||||
"fused_metadata_copy",
|
||||
"fused_metadata_copy_multi",
|
||||
@@ -702,7 +700,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
pattern="Fused MoE activation + quant / re-quant",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
|
||||
"<br>python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
|
||||
),
|
||||
active_keywords=(
|
||||
@@ -763,7 +761,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused quantize + indexed K-cache store",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py"
|
||||
"python/sglang/kernels/ops/attention/fused_store_index_cache.py"
|
||||
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
||||
),
|
||||
active_keywords=("fused_store_index_k_cache",),
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user