minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[do not merge] add new cookbooks (#34658)

Browse Source 
Co-authored-by: Jianfei Wang <jianfei.wangg@outlook.com>
Co-authored-by: Jianfei Wang <905787410@qq.com>
This commit is contained in:
Yuhao Yang
2026-08-13 13:35:15 +00:00
committed by GitHub
co-authored by Jianfei Wang Jianfei Wang
parent 74c0322342
commit c1142677a8
3 changed files with 384 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/RedNote/Dots3-Note.mdx
+213
View File
@@ -0,0 +1,213 @@
---
title: Dots3-Note
description: "Deploy RedNote dots3.note with SGLang — a native multimodal omni model (MoE ViT + Whisper-derived audio encoder + native video flattening) on the dots3 hybrid MLA/SWA language model, with DSA and full-sharing MTP speculative decoding."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
dots3.note support is in [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829). Until that PR is included in a tagged SGLang release, install from a build that contains the PR.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate
git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/33829/head && git checkout FETCH_HEAD
uv pip install -e python
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:dev-dots3-note
```
This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself.
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick the checkpoint precision — the only deployment choice. The recipe runs on a single 8-GPU H200 node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet.
**Precision** — selects the MoE path, not just the weights. The BF16 cells pin `--moe-runner-backend deep_gemm` with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via `SGLANG_ENABLE_JIT_DEEPGEMM=1`). The FP8 cells leave both at `auto` and let SGLang resolve the runner from the checkpoint's quantization config.
**Spec Decode** — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path.
<Note>
Every cell in the Deploy panel above is currently **unverified**: the recipe runs, but no serving round on public weights has landed (the checkpoint is not yet released). Treat the cells as starting points and re-measure throughput and accuracy on your workload.
</Note>
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/rednote/dots3-note.jsx";
<Deployment config={config} />
## 1. Model Introduction
dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input.
- **Native multimodality** — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor.
- **Native video pipeline** — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path.
- **Hybrid attention** — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers.
- **MTP speculative decoding** — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head.
<Note>
The dots3.note checkpoint is **not yet publicly released**. The recipes on this page were validated against [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829); a Hugging Face repository will be linked here at launch.
</Note>
**Resources:** [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829)
{/* TODO: Add the Hugging Face link once the checkpoint is released. */}
## 2. Configuration Tips
**Hybrid KV pool.** dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with `--swa-full-tokens-ratio 0.03` setting the ratio of SWA-layer KV tokens to full-layer KV tokens (`swa_tokens ≈ full_tokens × ratio`). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck.
**MoE runner.** Leave the runner at the cell default: `deep_gemm` for BF16 checkpoints, `auto` for quantized ones. DeepEP is the all-to-all transport in every cell (`--moe-a2a-backend deepep`, dispatch tokens per rank tuned via `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128`).
**Attention backend.** FA3 across the board: prefill, decode, and draft (`--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3`) with `--page-size 64`. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode.
**DSA.** DSA indexing on full-attention layers is on by default. To disable it, add `--json-model-override-args '{"index_topk":null}'`.
**CUDA graphs.** The cells enable decode-side CUDA graphs only (`--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled`, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to `--cuda-graph-backend-decode disabled` (and expect `--deepep-mode normal` to be the better fit).
**Context length.** `--context-length 524288` is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool.
**Language-only mode.** Add `--language-only` to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see [EPD](#epd-disaggregation) below.
## 3. Advanced Usage
### 3.1 Native video input
dots3.note accepts a native `video_url`. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question.
<Accordion title="Video Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="dots3.note",
messages=[
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {"url": "https://example.com/sample.mp4"},
},
{"type": "text", "text": "Summarize what happens in this video."},
],
}
],
extra_body={
"seq": 131072,
"audio_cap": 0.5,
"audio_sr": 16000,
"k_mode": "eval_ek",
},
)
print(response.choices[0].message.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update...
```
</Accordion>
Per-request video preprocessing controls (all optional, passed via `extra_body`):
| Field | Default | Purpose |
|-------|---------|---------|
| `seq` | `131072` | Total sequence budget used by the video flattener. |
| `audio_cap` | `1.0` | Maximum fraction of the input budget assigned to audio; `0` disables audio processing. |
| `audio_sr` | `16000` | Audio sample rate. |
| `k_mode` | `eval_ek` | Deterministic evaluation/sampling mode of the flattener. |
These controls are request-scoped so that evaluation jobs with different context budgets can share one server. The flattener reserves room for `max_new_tokens` inside the budget and falls back to visual-only processing if audio would exceed the configured token budget.
<Warning>
Native video currently supports one video per request, and a native video cannot be mixed with separate image or audio inputs in the same request.
</Warning>
### 3.2 Image and audio input
Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format and SGLang's multimodal serving (`--enable-multimodal` is in every cell). The vision and audio towers run in-process, so no extra server is needed.
### 3.3 Tool Calling
The cells launch with `--tool-call-parser dots`, so structured tool calls surface via `message.tool_calls` out of the box.
<Accordion title="Tool Calling Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
resp = client.chat.completions.create(
model="dots3.note",
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
tools=tools,
)
print(resp.choices[0].message.tool_calls)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update...
```
</Accordion>
<a id="epd-disaggregation" />
### 3.4 Encoder/LLM Disaggregation (EPD)
`Dot3NoteForCausalLM` supports both roles of an encoder/LLM-disaggregated deployment:
- **Encoder role** — serve with `--encoder-only`; the instance runs only the vision and audio towers.
- **Language role** — serve with `--language-only`; the instance skips tower construction, leaving the memory to the language model.
See the [EPD guide](../../../docs/advanced_features/epd_disaggregation) for how to wire the roles together.
docs/docs.json
+6
View File
@@ -1279,6 +1279,12 @@
"cookbook/autoregressive/Meituan/LongCat-2.0"
]
},
{
"group": "RedNote",
"pages": [
"cookbook/autoregressive/RedNote/Dots3-Note"
]
},
{
"group": "Google",
"pages": [
docs/src/snippets/configs/rednote/dots3-note.jsx
+165
View File
@@ -0,0 +1,165 @@
// Dots3-Note cookbook config. Consumed by _deployment.jsx + _playground.jsx.
// Single `export const config` literal - no spreads/calls/IIFE (Mintlify re-evals at hydration).
export const config = {
modelName: "Dots3-Note",
// No Playground on this page — the only extra knob (the dots tool-call parser)
// is already baked into the cells.
showPlaygroundLink: false,
// Hopper only for now — no Blackwell support.
supportedHardware: ["h200"],
// One model and one node shape — only the checkpoint precision is a real choice.
matchDims: [
{
id: "quant",
title: "Checkpoint Precision",
options: [
{ id: "bf16", label: "BF16" },
{ id: "fp8", label: "FP8" },
],
},
],
modelNames: {
// TODO: replace with the public repo id once the checkpoint is released.
default: "<dots-note-checkpoint>",
},
placeholders: {
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
PORT: { target: "command", label: "Bind port", default: "30000" },
HF_TOKEN: {
target: "command",
label: "HF token (Docker)",
default: "<your-hf-token>",
},
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
},
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
-H 'Content-Type: application/json' \\
-d '{
"model": "{{MODEL_NAME}}",
"messages": [{
"role": "user",
"content": [
{"type": "video_url", "video_url": {"url": "https://example.com/sample.mp4"}},
{"type": "text", "text": "Summarize what happens in this video."}
]
}]
}'`,
dockerImages: {
h200: "lmsysorg/sglang:dev",
},
cells: [
{
match: { hw: "h200", quant: "bf16" },
nnodes: 1,
verified: false,
env: [
"SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1",
"SGLANG_ENABLE_JIT_DEEPGEMM=1",
"SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD=8192",
"SGLANG_MAX_KV_CHUNK_CAPACITY=8192",
"SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128",
"SGLANG_WARMUP_TIMEOUT=1800",
],
flags: [
"--model-path {{MODEL_NAME}}",
"--context-length 524288",
"--enable-dp-attention",
"--dp-size 8",
"--tp-size 8",
"--ep-size 8",
"--mem-fraction-static 0.87",
"--max-running-requests 256",
"--chunked-prefill-size 16384",
"--trust-remote-code",
"--swa-full-tokens-ratio 0.03",
"--prefill-attention-backend fa3",
"--decode-attention-backend fa3",
"--page-size 64",
"--moe-dense-tp-size 1",
"--cuda-graph-backend-decode full",
"--cuda-graph-backend-prefill disabled",
"--cuda-graph-max-bs-decode 32",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--speculative-draft-model-path {{MODEL_NAME}}",
"--speculative-draft-attention-backend fa3",
"--moe-a2a-backend deepep",
"--moe-runner-backend deep_gemm",
"--deepep-dispatcher-output-dtype bf16",
"--deepep-mode auto",
"--enable-nccl-nvls",
"--enable-multimodal",
"--enable-metrics",
"--tool-call-parser dots",
"--reasoning-parser qwen3",
"--watchdog-timeout 1800",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "h200", quant: "fp8" },
nnodes: 1,
verified: false,
env: [
"SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1",
"SGLANG_ENABLE_JIT_DEEPGEMM=1",
"SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD=8192",
"SGLANG_MAX_KV_CHUNK_CAPACITY=8192",
"SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128",
"SGLANG_WARMUP_TIMEOUT=1800",
],
flags: [
"--model-path {{MODEL_NAME}}",
"--context-length 524288",
"--enable-dp-attention",
"--dp-size 8",
"--tp-size 8",
"--ep-size 8",
"--mem-fraction-static 0.87",
"--max-running-requests 256",
"--chunked-prefill-size 16384",
"--trust-remote-code",
"--swa-full-tokens-ratio 0.03",
"--prefill-attention-backend fa3",
"--decode-attention-backend fa3",
"--page-size 64",
"--moe-dense-tp-size 1",
"--cuda-graph-backend-decode full",
"--cuda-graph-backend-prefill disabled",
"--cuda-graph-max-bs-decode 32",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--speculative-draft-model-path {{MODEL_NAME}}",
"--speculative-draft-attention-backend fa3",
"--moe-a2a-backend deepep",
"--moe-runner-backend auto",
"--deepep-dispatcher-output-dtype auto",
"--deepep-mode auto",
"--enable-nccl-nvls",
"--enable-multimodal",
"--enable-metrics",
"--reasoning-parser qwen3",
"--tool-call-parser dots",
"--watchdog-timeout 1800",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
};