Compare commits

..
44 Commits
Author SHA1 Message Date
BingjiaWang 4e60127fb4 [DSV4.1][PD][1/N] Support prefill node use cp parallel strategy (#40323) 2026-10-20 18:27:25 +08:00
Siyuan Chen e9fdf3ad0a [DSV4.1][PD][5/N] Decode node support DP attention (#40177) 2026-10-20 18:23:37 +08:00
minke.yu 2087760ea8 fix(v41): materialize CP request rows in select_compact_prefill
PR #42922 covered prefill_requests but the compact-candidate consume path
(DSpark draft consuming target candidates) still subscripted req_rows
directly, crashing with TypeError under CP8+DSpark5 prefill when
out_page_indices is set. Route through materialize_prefill_request_rows;
non-CP behavior unchanged (returns req_rows as-is).
2026-10-20 12:10:08 +08:00
minke.yu ef1a5955f3 PR #39306: H20-3e block-FP8 GEMM configs + adapt tables to NVIDIA H20
Net diff of PR #39306 vs upstream/main (kernel path already landed via #39657).
Also duplicated the six #41098 quantization tables and the six #39306 gemm
tables under device_name=NVIDIA_H20: config lookup is exact-filename and our
GPUs report 'NVIDIA H20' (96GB), not H20-3e. H20-3e shares the SM90 compute
configuration, so the tuned tables transfer.
2026-10-20 10:26:58 +08:00
Yuhao Yang 7e17dee8b7 Keep wo_a small-batch on its validated G=2 shape 2026-10-20 10:24:26 +08:00
Yuhao Yang 0090df5d64 Gate the G=1 wo_a GEMV to sm90 2026-10-20 10:24:26 +08:00
Yuhao Yang 4938eea2dc [DSV4.1] Enable Hopper decode optimizations for TP8 2026-10-20 10:24:25 +08:00
冯骐 8c6cbd8c40 Support DSpark decoder bounded replay with DSV4.1 vision CP
Use tail-aware CP gathers and logits metadata, preserve original CP ownership of canonical routing IDs, and pass global token indices to DSpark. Adapt mHC state slicing and register CPU regression tests. Based on PR40219; tail handling informed by PR40161.
2026-10-20 10:20:57 +08:00
冯骐 c82aed9fef Fix DSV4.1 Torch prefill indexer request-pool rows under CP
Materialize actual request-pool IDs from per-rank real row counts. Preserve zero-row requests, bounded-replay tails and the non-CP fast path. Add CPU regression coverage.
2026-10-20 10:16:54 +08:00
minke.yu e6f6420275 fix(v41): import Backend/Phase/with_phase in vision prefill CP validator 2026-10-20 00:43:11 +08:00
minke.yu 566176d0cf deps: pin sglang-kernel back to 0.4.7 for torch 2.13 ABI
da343ba5d1 (#42604 cherry-pick) bumped sglang-kernel to 0.4.9, whose
wheels are compiled against torch 2.14 (requires_dist torch==2.14.1 and
references c10::cuda::CUDAErrorLogCapture, absent in torch 2.13).
Import died with undefined symbol on this torch-2.13 branch.
sgl-deep-gemm 0.2.2 has no torch pin and stays.
2026-10-19 23:02:11 +08:00
minke.yu 4f52678224 fix(mm_schedule): _collect_image_span_requests emit 4-tuple overlapping
The vision-CP cherry-pick stored (idx, item, token_count) 3-tuples into
PerImageRequestInfo.overlapping, but the field and its consumers
(_assemble_per_image_chunk, phase-1a) use (idx, item, start, end) with
cache key (hash, end-start+1). First owner-mode image request died with
'not enough values to unpack (expected 4, got 3)'.
2026-10-19 21:47:57 +08:00
minke.yu 068acff09d fix(mm_schedule): port _item_overlap from upstream for vision-CP span collect
fd86688428 (vision CP single-owner encoding) cherry-pick used
_item_overlap in _collect_image_span_requests without the definition;
first multimodal prefill request killed every scheduler with NameError.
2026-10-19 21:21:33 +08:00
minke.yu f9056f0fcd fix(dsv4 fp4_indexer): 128B-align SM90 paged score rows for deep_select
fp4_index_logits_paged padded rows to 8 floats (32B) and
fp4_index_logits_candidates did not pad at all, but the tvm-ffi
deep_select topk kernel enforces kInputAlignmentBytes=128 on the input
row stride. bs>1 decode/CUDA-graph capture died with
'Tensor stride for dimension 0 is not aligned to 128 bytes'.
Candidates kernel now takes a separate OUT_STRIDE constexpr.
2026-10-19 20:55:01 +08:00
minke.yu 2aae0cb4ee fix(deep_gemm_wrapper): import DEEPGEMM_NEED_TMA_ALIGNED_SCALES/DEEPGEMM_SCALE_UE8M0 in compile_utils
Cherry-pick da343ba5d1 (#42604) lost the configurer import hunk; runtime
NameError killed every scheduler right after weight load on SM90.
2026-10-19 20:16:46 +08:00
minke.yu ea5fc832c1 scheduler: CP-symmetric idle check for health-check admission
Health-check admit/skip used is_fully_idle(), which includes rank-local
hicache drain queues; ranks diverge right after activity, so one rank
dispatched the health-check generate while others piggyback-skipped,
deadlocking CP (hicache drain all_reduce vs CP request broadcast).
Seen on cp2/cp4 + hicache L3 after router health checks.

Recovered from b300 /data/ymk/sglang working copy (uncommitted WIP).

(cherry picked from commit b91137ab98)
2026-10-19 10:47:57 +08:00
minke.yu 9894dc789c G9 P0 fixes: v2 pool_transfers for storage hit query, disagg drain hook, query gate
- unified_radix_cache.query_storage_hit_length: build PREFETCH-phase
  pool_transfers (comp prepare_prefetch + build_hicache_transfers +
  sidecars) so DeepSeek V4's logical-anchor KV goes through
  batch_exists_v2 side-pool existence instead of v1 batch_exists which
  always misses (no physical KV objects are stored for V4).
- scheduler.handle_query_cache_hit: allow L3 query when hicache storage
  backend is attached (not only decode-hicache mode; SWA models forbid
  decode radix+hicache so P nodes could never query L3); construct
  RadixKey from array('q') to match tree node key types.
- disaggregation/prefill: record_prefill_drain on final chunk in the
  disagg-prefill result path (it bypasses batch_result_processor, so
  drain_rate_tokens_per_s stayed 0 on P nodes).
- deploy: 3p1d l3test compose against the real mooncake store stack.

(cherry picked from commit e7dc175e94)
2026-10-19 10:47:07 +08:00
minke.yu 77e7555ae3 G9 P0 fix: /query_cache_hit 绕过 _tokenize_one_request 的 ReqState 依赖
_tokenize_one_request._create_tokenized_object 会读 rid_to_state[obj.rid]
(正常生成路径在 tokenize 前注册 ReqState),纯查询无 rid → KeyError: None。
改为:dsv41 chat encoding 直接渲染的 prompt_ids 直接用;否则对渲染后
text 调 _tokenize_texts(与真实路径同一分词入口)。tok8 环境实测 200。

(cherry picked from commit 71c6d8bc55)
2026-10-19 10:46:23 +08:00
minke.yu 319ee3c056 G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
  记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
  set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费

B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
  + query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
  _tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
  v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit

(cherry picked from commit 4eabf5e228)
2026-10-19 10:46:23 +08:00
minke.yu e59014be87 gateway: import g9i sgl-model-gateway wholesale from 26.10.18 (c8d06424fa)
Replaces the v0.5.21 gateway with our g9i admission/routing gateway:
est_TTFT shadow mode, AIMD congestion window, D-ledger booking, fast-P
probe routing, cache-aware admission gates, gateway metrics (T1-T25b).
Note: upstream cleanup f337f01b80 (dead eval modules) is superseded.
2026-10-19 10:45:36 +08:00
minke.yu fd86688428 PR #40219: DeepSeek V4.1 vision CP and single-owner image encoding
Source: https://github.com/sgl-project/sglang/pull/40219 (applied via GitHub .diff, 3way)
Adaptations for v0.5.21 base:
- attn_dp_enabled_of(cfg) -> cfg.enable_dp_attention (#41818 not in base)
- dropped _validate_joint_schema_request call (#42721 Clef, not in base)
- moved Backend import before the TRT-LLM prefill-graph check
2026-10-19 10:45:10 +08:00
Vladislav Nosivskoy da343ba5d1 [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 (#42604)
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2ff3e6dce1)
2026-10-19 10:38:40 +08:00
minke.yu 9ea02a7201 PR #40421: reuse shared compressed KV dequantization in PD prefill
Source: https://github.com/sgl-project/sglang/pull/40421 (applied via GitHub .diff, 3way)
2026-10-19 10:37:44 +08:00
minke.yu 6c1c54b514 fixup: import is_gfx95_supported for #41294 and #40860 ROCm branches
dsv41_sparse.py and graph_variants.py referenced is_gfx95_supported without
importing it (latent NameError on the ROCm path).
2026-10-19 10:37:08 +08:00
minke.yu 51624199b4 PR #41294: use the fused low-ratio compressor on Hopper (sm90 gate)
Source: https://github.com/sgl-project/sglang/pull/41294 (applied via GitHub .diff, 3way)
Skipped: test_dsv41_fused_compress.py (file not in v0.5.21 base).
2026-10-19 10:35:29 +08:00
minke.yu 31b3758c99 PR #42365: fuse Hopper prefill indexer scoring into one Triton kernel
Source: https://github.com/sgl-project/sglang/pull/42365 (applied via GitHub .diff, 3way)
Conflicts resolved: unioned env flags with #40945 SM90 flags; merged fused
scoring loop with #40860 candidate_blocks path (candidate > fused > torch priority).
2026-10-19 10:33:13 +08:00
minke.yu 740372bef0 PR #40945: SM90 static verification indexing (grouped/length-aware indexer)
Source: https://github.com/sgl-project/sglang/pull/40945 (applied via GitHub .diff, 3way)
Adaptation: PR base predates #41659 which removed the Selection struct;
ported sm90_decode.py and call sites to the inputs.out_page_indices interface
(raw_indices=None on decode paths, reset via inputs.reset_outputs()).
2026-10-19 10:30:41 +08:00
minke.yu 19fe20baec PR #40860: SM90 candidate indexers (compact candidate dispatch)
Source: https://github.com/sgl-project/sglang/pull/40860 (applied via GitHub .diff, 3way)
Conflicts resolved: kept #41658 padded-return trim in fp4_index_logits_paged,
took PR side for candidate_blocks branches, SM90 graph gate, deep_select test param.
Skipped: test_dsv41_candidate_graph_variants.py (file not in v0.5.21 base).
2026-10-19 10:22:09 +08:00
Po-Han Huang (NVIDIA) 383df91fcc [Test] dsv4: Fix Hopper indexer replay test after DeepSelect decode (#41561) (#44193)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d5f80aa3c3)
2026-10-19 10:16:18 +08:00
yuyu5333 534842ac08 [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. (#41561)
Co-authored-by: Claude Code <noreply@anthropic.com>
(cherry picked from commit b50fdeb0ef)
2026-10-19 10:16:18 +08:00
Yuwei An 4dd95a8817 [DSV4.1] Reduce eager Engram extend preparation (#42159)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d434ea444d)
2026-10-19 10:16:08 +08:00
Yuwei An 656e00dcd3 [DSv4.1] mxfp8 dispatch cache (#42156)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
(cherry picked from commit 2d38934493)
2026-10-19 10:15:46 +08:00
DarkSharpness a4b1363bf4 [DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs (#41659)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 520d6d0229)
2026-10-19 10:15:46 +08:00
DarkSharpness ad2acaa2df [DSv4.1] Fold q_rope_store into fused_q_norm_rope (#41657)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 379ec90fb2)
2026-10-19 10:15:45 +08:00
Yuwei An ecaaa37cb0 [Fix][DSV4.1] SWA page size with bounded replay (#42128)
(cherry picked from commit ef867fa40d)
2026-10-19 10:14:30 +08:00
DarkSharpness 1b70fc192c [DSv4.1] Trim prefill model-entry overhead (#41704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2f6e0ec45d)
2026-10-19 10:14:29 +08:00
DarkSharpness 3ebbb2d536 [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 5916999afb)
2026-10-19 10:14:29 +08:00
DarkSharpness 5d52cccbee [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit d75d5b33f7)
2026-10-19 10:14:28 +08:00
minke.yu 74d03143e2 PR #41098: H20 block-FP8 tuned GEMM configs
Source: https://github.com/sgl-project/sglang/pull/41098 (applied via GitHub .diff, 3way)
2026-10-19 10:13:59 +08:00
Xiaoyu Zhang 2b192aade6 [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (#41251)
(cherry picked from commit 391e665dab)
2026-10-19 10:13:35 +08:00
minke.yu 9247c97919 PR #40919: fix dspark folded result buffer reuse race
Source: https://github.com/sgl-project/sglang/pull/40919 (applied via GitHub .diff, 3way)
2026-10-19 10:12:23 +08:00
minke.yu 5e7190b87a PR #40758: escape literal DSV4.1 image placeholder in user text
Source: https://github.com/sgl-project/sglang/pull/40758 (applied via GitHub .diff, 3way)
2026-10-19 10:12:16 +08:00
e00930c548 [Cherry-pick to release/v0.5.21] [Test] Demote PD test RDMA openability check to a warning (#41681) (#41802)
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 19:44:12 -07:00
Kangyan-ZhouandBaizhou Zhang d7ff74b984 [Cherry-pick to release/v0.5.21] [Fix] Restore deferred layer dumps and pin SentencePiece for InternVL (#41777) (#41788)
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
2026-09-29 17:30:38 -07:00
4390 changed files with 113048 additions and 513955 deletions
@@ -1,249 +0,0 @@
---
name: processor-model-parity
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
---
# Processor model parity
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
server, sgl-router, the renderer). A model is supported only when, for the same
request body, the processor produces **the same prompt text and the same prompt
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
the implementation, and SGLang code exists only where the two differ.
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
Multimodal preprocessing is not covered here: `render_request` fails on media
content parts, so hosts send those chats to the engine. The fixture generator
serves every checkpoint text-only, so a media case records that rejection.
## Done means
- **All text should work**: every text chat the engine accepts renders and
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
the processor cannot reproduce goes to the engine; list it in the PR.
- **Chat and completions go through the engine's `/generate`**, with the router
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
engine's OpenAI route is only the `Unsupported` fallback. That needs the
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
against a live engine.
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
to `rust/sglang-processor/` unless they start with `python/`.
## Where a model's code goes
| Piece | File | Needed when |
|---|---|---|
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
| README row | the `render` table in `README.md` | always |
No new test code: `tests/parity.rs` checks every fixture in the directory.
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
renders the model:
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
`PromptFormatter`, replace that branch.
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
variant holding Dynamo's Jinja `PromptFormatter`, picked in
`ChatFormatter::from_tokenizer_config` from the template text (as
`parser/template_detection.py` picks it, where it has a detector), and build the
template context that `_process_messages` and `_apply_jinja_template` build, like
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
rewrites shapes its load-time probes find the template rejects (a later system
turn, for one); SGLang raises on them, so check those before rendering.
## Method
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
Note whether its tokenizer ships a `chat_template`; that can change the spec.
1. **Trace the Python path for the model.** Write down every transformation
between the HTTP body and `prompt_ids`, in order:
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
Reuse `render/reasoning.rs` for this rather than porting it again.
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
replaces the request effort; server `--default-chat-template-kwargs` merge in.
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
It also reads the server's `--tool-call-parser` and whether the tokenizer has
a chat template. If the deployment depends on the parser, set
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
effort profile reads `encoding/encoding_dsv4.py`).
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
message dumping, content flattening, `_handle_last_assistant_message`, system
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
call, and how the prompt is tokenized.
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
rules live there: dropped turns, `</think>` on empty reasoning, argument
formatting, and the errors it raises.
This list is the **divergence inventory**.
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
which often differ from SGLang's. Diff the low-level encoder against Python's
for the rules in step 1.
3. **Write one request case per inventory item** in the fixture (`name` +
`request` only), then generate the expected outputs from Python (below). Cover
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
each failing case, add the smallest SGLang step that fixes it, with a one-line
doc comment naming the Python source it mirrors. Never patch the fixture to
match Rust.
5. **Verify** text offline, then token ids with the checkpoint cached (commands
below). If sgl-router or the renderer uses the formatter, run their suites too.
## Checklist (DeepSeek-V4 case that covers each)
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
reduced to role and content; null content becomes `""`; history tool calls keep
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
- **Content parts**: which part types survive and the join separator. (`parts`)
- **Tools**: all request tools or only those `tool_choice` allows; field order
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
`tool_schema_type_aliases`)
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
- **Pydantic coercion**: typed request fields are coerced before rendering. For
example, `strict: 1` and `defer_loading: "false"` become booleans, and
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
JSON string; other encoders format them their own way. Keep key order
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
truthiness (`1` is on); strings use Python's yes-word set.
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
a separately tokenized prefix. Check whether content is flattened before the
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
assistant turn's tool calls are discarded, so object checks on their arguments
belong after the split. Hosts reach the model through `render_prompt` too, so
check their continuation handling as well.
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
continuation prefix is encoded alone and loses a leading BOS
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
includes pydantic rejections and Python crashes (a 500 is still a rejection).
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
PR that hosts fall back to the engine for it.
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
reach, fix it upstream and mark the case `known_gap` with the reason. The test
reports the case without failing, and fails once it matches so the marker gets
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
## Generating fixtures
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
renders it through the real `_process_messages`, under the case's
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
writes:
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
are kept as written). An `AttributeError` aborts instead: the stub server lacks
something, so extend it;
- the fixture-level `config` (the `config.json` fields the processor reads, plus
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
checkpoint its `chat_template`, so the test renders offline.
```sh
cd rust/sglang-processor
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
```
When a new spec needs more, extend `serving_chat()` in the script (one place):
- **More server state:** set the extra attribute that `__init__` resolves, for
example `_dsv41_default_reasoning_effort`.
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
recorded by the template's text output. Its `render_request` returns the
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
Pin `revision` to a commit hash and regenerate only on purpose.
## Verify
```sh
cd rust
cargo test -p sglang-processor --locked # text, offline
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
```
The token check loads the pinned commit's snapshot from the cache. When the
snapshot is missing, the test prints `token ids not checked` (visible with
`-- --nocapture`), so confirm that line is absent before claiming token parity.
With the snapshot cached, the test also checks that the checkpoint resolves the
fixture's recorded DeepSeek-V4 profile.
## OpenAI layer
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
once its render parity holds. Two more fixtures then cover the rest of Python's
OpenAI layer:
- `tests/fixtures/reasoning_parity/<parser>.json`, from
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
chunked outputs. A parser is served once `src/think/models/` ports it.
- `tests/fixtures/tool_parity/<parser>.json`, from
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
parser is served once `src/tool_call/models/` ports it.
- `tests/fixtures/openai_parity/<model-id>.json`, from
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
output for it, and the response Python builds from that output. Launch the
engine with the model's cookbook flags, including its parsers. Without the
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
fixture does.
## Pitfalls
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
`dynamo-renderer` or `dynamo-tokenizers`.
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
sent, so such requests stay on the engine's route. Fixtures don't show this; a
live engine does.
- Python mutates the request before rendering (`_validate_request` rewrites tool
schemas), so fixture generators drive the server's own entry points.
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
generator pins them and `tests/parity.rs` clears them, so fixtures do not
depend on the machine.
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
such as `task` and message-level `tools`. Parity is defined on `render_request`;
report host-adapter gaps separately rather than bending the model code.
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
beyond the dispatch arm.
@@ -6,5 +6,4 @@ Before modifying the following components, read the listed skill first.
- **`Scheduler` / `TokenizerManager` / `ModelRunner` `__init__`** (`python/sglang/srt/managers/scheduler.py`, `python/sglang/srt/managers/tokenizer_manager.py`, `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
- **Any edit to a frozen core file** (currently `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
- **Environment variables** (adding, renaming, or reviewing any `SGLANG_*` env var, migrating a legacy `SGL_*` alias, or touching `python/sglang/srt/environ.py`) → [`env-var-conventions`](../skills/env-var-conventions/SKILL.md)
- **Chat rendering in `rust/sglang-processor`** (adding or changing a model under `rust/sglang-processor/src/render/`, or its parity fixtures) → [`processor-model-parity`](../skills/processor-model-parity/SKILL.md)
- **Scripted runtime** (anything related to the scripted runtime) → [`scripted-runtime-notes`](../skills/scripted-runtime-notes/SKILL.md)
-31
View File
@@ -1,31 +0,0 @@
# Accuracy evals: use sgl-eval
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
model's accuracy.
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
server's env, at the version pinned in the `test` extra of
`python/pyproject.toml`.
```bash
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
```
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
- **Sampling:** use the model's recommended settings (temperature, top_p,
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
them with its own defaults, so pass them explicitly, or use
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
sets the `thinking` chat-template key; if the model's template reads a
different one (commonly `enable_thinking`), pass that key with
`--chat-template-kwarg` instead. When comparing against a CI test, use that
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
base models, since sgl-eval only sends chat requests.
Only compare scores from the same harness and settings, and say which ones you
used when you report a number.
-1
View File
@@ -1 +0,0 @@
../.agents/skills
@@ -167,7 +167,7 @@ describes what each one does.
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
4. If all matched jobs are completed and count matches `expected_count` → success
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
**Job specs example** (base-b):
```json
@@ -378,7 +378,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|---------|-------------|---------------|
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
@@ -396,7 +396,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
@@ -264,5 +264,5 @@ git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
docs/docs.json docs/cookbook/<category>/intro.mdx
git commit -m "Add <Display-Name> cookbook"
git push -u origin add-<model>-cookbook
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
gh pr create --title "Add <Display-Name> cookbook" --body "..."
```
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
`{{PORT}}` placeholders so users can override through the Env modal.
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
then parallelism (`--tp`, `--attn-dp-size`), then MoE
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
flags, then tuning knobs, with `--host` / `--port` last. The playground
engine assumes this ordering when inserting overrides (its anchors target
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
| Axis key | Widget | Use when |
|---|---|---|
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
knobs/values that are disabled under the live facts, so stale picks never
emit a blocked combination. Interleave prefill-CP + DP-Attention is
deliberately NOT grayed (combined support is planned upstream even though
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
current releases assert `dp_size == 1` for interleave) — the engine shows
a warning hint below the command box instead.
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
playgroundFeatures: {
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
attention: {
knobs: [
{ id: "tp", label: "TP", values: [
@@ -357,7 +357,8 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--attn-dp-size 4",
"--dp 4",
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -402,7 +403,8 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--attn-dp-size 16",
"--dp 16",
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
## 1. Quick start
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
a verified recipe below.
<Deployment config={config} />
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
can't do coupled changes), and so does a toggle the legacy page itself labels
with operating-point words — e.g. a `dpattention` radio whose options are
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
even when its flags are uncoupled (`--attn-dp-size N` is a pure
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
flag add). Any other toggle that only adds/removes its own flags becomes a
Playground axis with the flags baked into cells when the legacy default was
ON — EXCEPT parsers:
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
Caveats discovered in the pilot:
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
@@ -11,7 +11,6 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
- A working SGLang installation (`pip install -e .` or equivalent)
- At least one available CUDA GPU
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
## Step-by-step Workflow
@@ -50,7 +49,7 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
```
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- This is a quick sanity check, not a rigorous benchmark.
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
@@ -109,8 +108,8 @@ for i in $(seq 1 120); do
done
# 3. Accuracy check
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
# Expected: score > 80%
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
# Expected: Accuracy > 0.8
# 4. Profile
python3 -m sglang.test.send_one --profile

Some files were not shown because too many files have changed in this diff Show More