Compare commits
44
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4e60127fb4 | ||
|
|
e9fdf3ad0a | ||
|
|
2087760ea8 | ||
|
|
ef1a5955f3 | ||
|
|
7e17dee8b7 | ||
|
|
0090df5d64 | ||
|
|
4938eea2dc | ||
|
|
8c6cbd8c40 | ||
|
|
c82aed9fef | ||
|
|
e6f6420275 | ||
|
|
566176d0cf | ||
|
|
4f52678224 | ||
|
|
068acff09d | ||
|
|
f9056f0fcd | ||
|
|
2aae0cb4ee | ||
|
|
ea5fc832c1 | ||
|
|
9894dc789c | ||
|
|
77e7555ae3 | ||
|
|
319ee3c056 | ||
|
|
e59014be87 | ||
|
|
fd86688428 | ||
|
|
da343ba5d1 | ||
|
|
9ea02a7201 | ||
|
|
6c1c54b514 | ||
|
|
51624199b4 | ||
|
|
31b3758c99 | ||
|
|
740372bef0 | ||
|
|
19fe20baec | ||
|
|
383df91fcc | ||
|
|
534842ac08 | ||
|
|
4dd95a8817 | ||
|
|
656e00dcd3 | ||
|
|
a4b1363bf4 | ||
|
|
ad2acaa2df | ||
|
|
ecaaa37cb0 | ||
|
|
1b70fc192c | ||
|
|
3ebbb2d536 | ||
|
|
5d52cccbee | ||
|
|
74d03143e2 | ||
|
|
2b192aade6 | ||
|
|
9247c97919 | ||
|
|
5e7190b87a | ||
|
|
e00930c548 | ||
|
|
d7ff74b984 |
@@ -1,249 +0,0 @@
|
||||
---
|
||||
name: processor-model-parity
|
||||
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
|
||||
---
|
||||
|
||||
# Processor model parity
|
||||
|
||||
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
|
||||
server, sgl-router, the renderer). A model is supported only when, for the same
|
||||
request body, the processor produces **the same prompt text and the same prompt
|
||||
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
|
||||
the implementation, and SGLang code exists only where the two differ.
|
||||
|
||||
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
|
||||
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
|
||||
Multimodal preprocessing is not covered here: `render_request` fails on media
|
||||
content parts, so hosts send those chats to the engine. The fixture generator
|
||||
serves every checkpoint text-only, so a media case records that rejection.
|
||||
|
||||
## Done means
|
||||
|
||||
- **All text should work**: every text chat the engine accepts renders and
|
||||
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
|
||||
the processor cannot reproduce goes to the engine; list it in the PR.
|
||||
- **Chat and completions go through the engine's `/generate`**, with the router
|
||||
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
|
||||
engine's OpenAI route is only the `Unsupported` fallback. That needs the
|
||||
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
|
||||
against a live engine.
|
||||
|
||||
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
|
||||
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
|
||||
to `rust/sglang-processor/` unless they start with `python/`.
|
||||
|
||||
## Where a model's code goes
|
||||
|
||||
| Piece | File | Needed when |
|
||||
|---|---|---|
|
||||
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
|
||||
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
|
||||
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
|
||||
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
|
||||
| README row | the `render` table in `README.md` | always |
|
||||
|
||||
No new test code: `tests/parity.rs` checks every fixture in the directory.
|
||||
|
||||
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
|
||||
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
|
||||
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
|
||||
renders the model:
|
||||
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
|
||||
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
|
||||
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
|
||||
`PromptFormatter`, replace that branch.
|
||||
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
|
||||
variant holding Dynamo's Jinja `PromptFormatter`, picked in
|
||||
`ChatFormatter::from_tokenizer_config` from the template text (as
|
||||
`parser/template_detection.py` picks it, where it has a detector), and build the
|
||||
template context that `_process_messages` and `_apply_jinja_template` build, like
|
||||
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
|
||||
rewrites shapes its load-time probes find the template rejects (a later system
|
||||
turn, for one); SGLang raises on them, so check those before rendering.
|
||||
|
||||
## Method
|
||||
|
||||
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
|
||||
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
|
||||
Note whether its tokenizer ships a `chat_template`; that can change the spec.
|
||||
1. **Trace the Python path for the model.** Write down every transformation
|
||||
between the HTTP body and `prompt_ids`, in order:
|
||||
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
|
||||
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
|
||||
Reuse `render/reasoning.rs` for this rather than porting it again.
|
||||
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
|
||||
replaces the request effort; server `--default-chat-template-kwargs` merge in.
|
||||
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
|
||||
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
|
||||
It also reads the server's `--tool-call-parser` and whether the tokenizer has
|
||||
a chat template. If the deployment depends on the parser, set
|
||||
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
|
||||
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
|
||||
effort profile reads `encoding/encoding_dsv4.py`).
|
||||
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
|
||||
message dumping, content flattening, `_handle_last_assistant_message`, system
|
||||
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
|
||||
call, and how the prompt is tokenized.
|
||||
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
|
||||
rules live there: dropped turns, `</think>` on empty reasoning, argument
|
||||
formatting, and the errors it raises.
|
||||
|
||||
This list is the **divergence inventory**.
|
||||
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
|
||||
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
|
||||
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
|
||||
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
|
||||
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
|
||||
which often differ from SGLang's. Diff the low-level encoder against Python's
|
||||
for the rules in step 1.
|
||||
3. **Write one request case per inventory item** in the fixture (`name` +
|
||||
`request` only), then generate the expected outputs from Python (below). Cover
|
||||
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
|
||||
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
|
||||
each failing case, add the smallest SGLang step that fixes it, with a one-line
|
||||
doc comment naming the Python source it mirrors. Never patch the fixture to
|
||||
match Rust.
|
||||
5. **Verify** text offline, then token ids with the checkpoint cached (commands
|
||||
below). If sgl-router or the renderer uses the formatter, run their suites too.
|
||||
|
||||
## Checklist (DeepSeek-V4 case that covers each)
|
||||
|
||||
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
|
||||
reduced to role and content; null content becomes `""`; history tool calls keep
|
||||
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
|
||||
- **Content parts**: which part types survive and the join separator. (`parts`)
|
||||
- **Tools**: all request tools or only those `tool_choice` allows; field order
|
||||
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
|
||||
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
|
||||
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
|
||||
`tool_schema_type_aliases`)
|
||||
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
|
||||
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
|
||||
- **Pydantic coercion**: typed request fields are coerced before rendering. For
|
||||
example, `strict: 1` and `defer_loading: "false"` become booleans, and
|
||||
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
|
||||
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
|
||||
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
|
||||
JSON string; other encoders format them their own way. Keep key order
|
||||
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
|
||||
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
|
||||
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
|
||||
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
|
||||
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
|
||||
truthiness (`1` is on); strings use Python's yes-word set.
|
||||
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
|
||||
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
|
||||
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
|
||||
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
|
||||
a separately tokenized prefix. Check whether content is flattened before the
|
||||
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
|
||||
assistant turn's tool calls are discarded, so object checks on their arguments
|
||||
belong after the split. Hosts reach the model through `render_prompt` too, so
|
||||
check their continuation handling as well.
|
||||
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
|
||||
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
|
||||
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
|
||||
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
|
||||
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
|
||||
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
|
||||
continuation prefix is encoded alone and loses a leading BOS
|
||||
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
|
||||
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
|
||||
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
|
||||
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
|
||||
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
|
||||
includes pydantic rejections and Python crashes (a 500 is still a rejection).
|
||||
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
|
||||
PR that hosts fall back to the engine for it.
|
||||
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
|
||||
reach, fix it upstream and mark the case `known_gap` with the reason. The test
|
||||
reports the case without failing, and fails once it matches so the marker gets
|
||||
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
|
||||
|
||||
## Generating fixtures
|
||||
|
||||
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
|
||||
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
|
||||
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
|
||||
renders it through the real `_process_messages`, under the case's
|
||||
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
|
||||
writes:
|
||||
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
|
||||
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
|
||||
are kept as written). An `AttributeError` aborts instead: the stub server lacks
|
||||
something, so extend it;
|
||||
- the fixture-level `config` (the `config.json` fields the processor reads, plus
|
||||
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
|
||||
checkpoint its `chat_template`, so the test renders offline.
|
||||
|
||||
```sh
|
||||
cd rust/sglang-processor
|
||||
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
|
||||
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
|
||||
```
|
||||
|
||||
When a new spec needs more, extend `serving_chat()` in the script (one place):
|
||||
- **More server state:** set the extra attribute that `__init__` resolves, for
|
||||
example `_dsv41_default_reasoning_effort`.
|
||||
|
||||
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
|
||||
recorded by the template's text output. Its `render_request` returns the
|
||||
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
|
||||
|
||||
Pin `revision` to a commit hash and regenerate only on purpose.
|
||||
|
||||
## Verify
|
||||
|
||||
```sh
|
||||
cd rust
|
||||
cargo test -p sglang-processor --locked # text, offline
|
||||
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
|
||||
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
|
||||
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
|
||||
```
|
||||
|
||||
The token check loads the pinned commit's snapshot from the cache. When the
|
||||
snapshot is missing, the test prints `token ids not checked` (visible with
|
||||
`-- --nocapture`), so confirm that line is absent before claiming token parity.
|
||||
With the snapshot cached, the test also checks that the checkpoint resolves the
|
||||
fixture's recorded DeepSeek-V4 profile.
|
||||
|
||||
## OpenAI layer
|
||||
|
||||
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
|
||||
once its render parity holds. Two more fixtures then cover the rest of Python's
|
||||
OpenAI layer:
|
||||
- `tests/fixtures/reasoning_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
|
||||
chunked outputs. A parser is served once `src/think/models/` ports it.
|
||||
- `tests/fixtures/tool_parity/<parser>.json`, from
|
||||
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
|
||||
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
|
||||
parser is served once `src/tool_call/models/` ports it.
|
||||
- `tests/fixtures/openai_parity/<model-id>.json`, from
|
||||
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
|
||||
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
|
||||
output for it, and the response Python builds from that output. Launch the
|
||||
engine with the model's cookbook flags, including its parsers. Without the
|
||||
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
|
||||
fixture does.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
|
||||
`dynamo-renderer` or `dynamo-tokenizers`.
|
||||
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
|
||||
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
|
||||
sent, so such requests stay on the engine's route. Fixtures don't show this; a
|
||||
live engine does.
|
||||
- Python mutates the request before rendering (`_validate_request` rewrites tool
|
||||
schemas), so fixture generators drive the server's own entry points.
|
||||
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
|
||||
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
|
||||
generator pins them and `tests/parity.rs` clears them, so fixtures do not
|
||||
depend on the machine.
|
||||
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
|
||||
such as `task` and message-level `tools`. Parity is defined on `render_request`;
|
||||
report host-adapter gaps separately rather than bending the model code.
|
||||
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
|
||||
beyond the dispatch arm.
|
||||
@@ -6,5 +6,4 @@ Before modifying the following components, read the listed skill first.
|
||||
- **`Scheduler` / `TokenizerManager` / `ModelRunner` `__init__`** (`python/sglang/srt/managers/scheduler.py`, `python/sglang/srt/managers/tokenizer_manager.py`, `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Any edit to a frozen core file** (currently `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
|
||||
- **Environment variables** (adding, renaming, or reviewing any `SGLANG_*` env var, migrating a legacy `SGL_*` alias, or touching `python/sglang/srt/environ.py`) → [`env-var-conventions`](../skills/env-var-conventions/SKILL.md)
|
||||
- **Chat rendering in `rust/sglang-processor`** (adding or changing a model under `rust/sglang-processor/src/render/`, or its parity fixtures) → [`processor-model-parity`](../skills/processor-model-parity/SKILL.md)
|
||||
- **Scripted runtime** (anything related to the scripted runtime) → [`scripted-runtime-notes`](../skills/scripted-runtime-notes/SKILL.md)
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Accuracy evals: use sgl-eval
|
||||
|
||||
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
|
||||
model's accuracy.
|
||||
|
||||
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
|
||||
server's env, at the version pinned in the `test` extra of
|
||||
`python/pyproject.toml`.
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
|
||||
```
|
||||
|
||||
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
|
||||
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
|
||||
|
||||
- **Sampling:** use the model's recommended settings (temperature, top_p,
|
||||
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
|
||||
them with its own defaults, so pass them explicitly, or use
|
||||
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
|
||||
sets the `thinking` chat-template key; if the model's template reads a
|
||||
different one (commonly `enable_thinking`), pass that key with
|
||||
`--chat-template-kwarg` instead. When comparing against a CI test, use that
|
||||
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
|
||||
|
||||
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
|
||||
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
|
||||
base models, since sgl-eval only sends chat requests.
|
||||
|
||||
Only compare scores from the same harness and settings, and say which ones you
|
||||
used when you report a number.
|
||||
@@ -1 +0,0 @@
|
||||
../.agents/skills
|
||||
+3
-3
@@ -167,7 +167,7 @@ describes what each one does.
|
||||
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
|
||||
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
|
||||
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
|
||||
|
||||
**Job specs example** (base-b):
|
||||
```json
|
||||
@@ -378,7 +378,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
|---------|-------------|---------------|
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
|
||||
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
|
||||
@@ -396,7 +396,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
|
||||
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
|
||||
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||
+1
-1
@@ -264,5 +264,5 @@ git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs/docs.json docs/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
|
||||
gh pr create --title "Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
+3
-3
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||
then parallelism (`--tp`, `--attn-dp-size`), then MoE
|
||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
knobs/values that are disabled under the live facts, so stale picks never
|
||||
emit a blocked combination. Interleave prefill-CP + DP-Attention is
|
||||
deliberately NOT grayed (combined support is planned upstream even though
|
||||
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
|
||||
current releases assert `dp_size == 1` for interleave) — the engine shows
|
||||
a warning hint below the command box instead.
|
||||
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
|
||||
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
|
||||
+5
-3
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
|
||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
@@ -357,7 +357,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--attn-dp-size 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
@@ -402,7 +403,8 @@ sgl-eval run gsm8k \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 16",
|
||||
"--attn-dp-size 16",
|
||||
"--dp 16",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
+2
-2
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
|
||||
|
||||
## 1. Quick start
|
||||
|
||||
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
|
||||
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
|
||||
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
|
||||
a verified recipe below.
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
+1
-1
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
|
||||
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||
even when its flags are uncoupled (`--attn-dp-size N` is a pure
|
||||
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
|
||||
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||
Playground axis with the flags baked into cells when the legacy default was
|
||||
ON — EXCEPT parsers:
|
||||
+2
-2
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
|
||||
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
||||
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
|
||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
|
||||
Caveats discovered in the pilot:
|
||||
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
||||
@@ -11,7 +11,6 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
|
||||
|
||||
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||
- At least one available CUDA GPU
|
||||
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
|
||||
|
||||
## Step-by-step Workflow
|
||||
|
||||
@@ -50,7 +49,7 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
|
||||
```
|
||||
|
||||
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
@@ -109,8 +108,8 @@ for i in $(seq 1 120); do
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
|
||||
# Expected: score > 80%
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
|
||||
# Expected: Accuracy > 0.8
|
||||
|
||||
# 4. Profile
|
||||
python3 -m sglang.test.send_one --profile
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user