diff --git a/docs/cookbook/autoregressive/RedNote/Dots3-Note.mdx b/docs/cookbook/autoregressive/RedNote/Dots3-Note.mdx new file mode 100644 index 000000000..254aa7609 --- /dev/null +++ b/docs/cookbook/autoregressive/RedNote/Dots3-Note.mdx @@ -0,0 +1,213 @@ +--- +title: Dots3-Note +description: "Deploy RedNote dots3.note with SGLang — a native multimodal omni model (MoE ViT + Whisper-derived audio encoder + native video flattening) on the dots3 hybrid MLA/SWA language model, with DSA and full-sharing MTP speculative decoding." +tag: NEW +--- + +## Deployment + + + + + +dots3.note support is in [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829). Until that PR is included in a tagged SGLang release, install from a build that contains the PR. + + + + + +```bash Command +pip install -U uv +uv venv --python 3.12 && source .venv/bin/activate + +git clone https://github.com/sgl-project/sglang.git +cd sglang +git fetch origin pull/33829/head && git checkout FETCH_HEAD +uv pip install -e python +``` + +Then run the **Python** output of the command panel below in that environment. + + + + + +```bash Command +docker pull lmsysorg/sglang:dev-dots3-note +``` + +This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself. + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick the checkpoint precision — the only deployment choice. The recipe runs on a single 8-GPU H200 node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet. + +**Precision** — selects the MoE path, not just the weights. The BF16 cells pin `--moe-runner-backend deep_gemm` with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via `SGLANG_ENABLE_JIT_DEEPGEMM=1`). The FP8 cells leave both at `auto` and let SGLang resolve the runner from the checkpoint's quantization config. + +**Spec Decode** — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path. + + +Every cell in the Deploy panel above is currently **unverified**: the recipe runs, but no serving round on public weights has landed (the checkpoint is not yet released). Treat the cells as starting points and re-measure throughput and accuracy on your workload. + + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/rednote/dots3-note.jsx"; + + + +## 1. Model Introduction + +dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input. + +- **Native multimodality** — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor. +- **Native video pipeline** — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path. +- **Hybrid attention** — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers. +- **MTP speculative decoding** — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head. + + +The dots3.note checkpoint is **not yet publicly released**. The recipes on this page were validated against [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829); a Hugging Face repository will be linked here at launch. + + +**Resources:** [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829) + +{/* TODO: Add the Hugging Face link once the checkpoint is released. */} + +## 2. Configuration Tips + +**Hybrid KV pool.** dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with `--swa-full-tokens-ratio 0.03` setting the ratio of SWA-layer KV tokens to full-layer KV tokens (`swa_tokens ≈ full_tokens × ratio`). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck. + +**MoE runner.** Leave the runner at the cell default: `deep_gemm` for BF16 checkpoints, `auto` for quantized ones. DeepEP is the all-to-all transport in every cell (`--moe-a2a-backend deepep`, dispatch tokens per rank tuned via `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128`). + +**Attention backend.** FA3 across the board: prefill, decode, and draft (`--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3`) with `--page-size 64`. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode. + +**DSA.** DSA indexing on full-attention layers is on by default. To disable it, add `--json-model-override-args '{"index_topk":null}'`. + +**CUDA graphs.** The cells enable decode-side CUDA graphs only (`--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled`, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to `--cuda-graph-backend-decode disabled` (and expect `--deepep-mode normal` to be the better fit). + +**Context length.** `--context-length 524288` is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool. + +**Language-only mode.** Add `--language-only` to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see [EPD](#epd-disaggregation) below. + +## 3. Advanced Usage + +### 3.1 Native video input + +dots3.note accepts a native `video_url`. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="dots3.note", + messages=[ + { + "role": "user", + "content": [ + { + "type": "video_url", + "video_url": {"url": "https://example.com/sample.mp4"}, + }, + {"type": "text", "text": "Summarize what happens in this video."}, + ], + } + ], + extra_body={ + "seq": 131072, + "audio_cap": 0.5, + "audio_sr": 16000, + "k_mode": "eval_ek", + }, +) + +print(response.choices[0].message.content) +``` + + + + + +```text Output +Pending update... +``` + + + +Per-request video preprocessing controls (all optional, passed via `extra_body`): + +| Field | Default | Purpose | +|-------|---------|---------| +| `seq` | `131072` | Total sequence budget used by the video flattener. | +| `audio_cap` | `1.0` | Maximum fraction of the input budget assigned to audio; `0` disables audio processing. | +| `audio_sr` | `16000` | Audio sample rate. | +| `k_mode` | `eval_ek` | Deterministic evaluation/sampling mode of the flattener. | + +These controls are request-scoped so that evaluation jobs with different context budgets can share one server. The flattener reserves room for `max_new_tokens` inside the budget and falls back to visual-only processing if audio would exceed the configured token budget. + + +Native video currently supports one video per request, and a native video cannot be mixed with separate image or audio inputs in the same request. + + +### 3.2 Image and audio input + +Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format and SGLang's multimodal serving (`--enable-multimodal` is in every cell). The vision and audio towers run in-process, so no extra server is needed. + +### 3.3 Tool Calling + +The cells launch with `--tool-call-parser dots`, so structured tool calls surface via `message.tool_calls` out of the box. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, +}] +resp = client.chat.completions.create( + model="dots3.note", + messages=[{"role": "user", "content": "What's the weather in Beijing?"}], + tools=tools, +) +print(resp.choices[0].message.tool_calls) +``` + + + + + +```text Output +Pending update... +``` + + + + + +### 3.4 Encoder/LLM Disaggregation (EPD) + +`Dot3NoteForCausalLM` supports both roles of an encoder/LLM-disaggregated deployment: + +- **Encoder role** — serve with `--encoder-only`; the instance runs only the vision and audio towers. +- **Language role** — serve with `--language-only`; the instance skips tower construction, leaving the memory to the language model. + +See the [EPD guide](../../../docs/advanced_features/epd_disaggregation) for how to wire the roles together. diff --git a/docs/docs.json b/docs/docs.json index 5f75a9573..95a9cdc05 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1279,6 +1279,12 @@ "cookbook/autoregressive/Meituan/LongCat-2.0" ] }, + { + "group": "RedNote", + "pages": [ + "cookbook/autoregressive/RedNote/Dots3-Note" + ] + }, { "group": "Google", "pages": [ diff --git a/docs/src/snippets/configs/rednote/dots3-note.jsx b/docs/src/snippets/configs/rednote/dots3-note.jsx new file mode 100644 index 000000000..912490355 --- /dev/null +++ b/docs/src/snippets/configs/rednote/dots3-note.jsx @@ -0,0 +1,165 @@ +// Dots3-Note cookbook config. Consumed by _deployment.jsx + _playground.jsx. +// Single `export const config` literal - no spreads/calls/IIFE (Mintlify re-evals at hydration). + +export const config = { + modelName: "Dots3-Note", + + // No Playground on this page — the only extra knob (the dots tool-call parser) + // is already baked into the cells. + showPlaygroundLink: false, + + // Hopper only for now — no Blackwell support. + supportedHardware: ["h200"], + + // One model and one node shape — only the checkpoint precision is a real choice. + matchDims: [ + { + id: "quant", + title: "Checkpoint Precision", + options: [ + { id: "bf16", label: "BF16" }, + { id: "fp8", label: "FP8" }, + ], + }, + ], + + modelNames: { + // TODO: replace with the public repo id once the checkpoint is released. + default: "", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + HF_TOKEN: { + target: "command", + label: "HF token (Docker)", + default: "", + }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ + "model": "{{MODEL_NAME}}", + "messages": [{ + "role": "user", + "content": [ + {"type": "video_url", "video_url": {"url": "https://example.com/sample.mp4"}}, + {"type": "text", "text": "Summarize what happens in this video."} + ] + }] +}'`, + + dockerImages: { + h200: "lmsysorg/sglang:dev", + }, + + + cells: [ + { + match: { hw: "h200", quant: "bf16" }, + nnodes: 1, + verified: false, + env: [ + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", + "SGLANG_ENABLE_JIT_DEEPGEMM=1", + "SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD=8192", + "SGLANG_MAX_KV_CHUNK_CAPACITY=8192", + "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128", + "SGLANG_WARMUP_TIMEOUT=1800", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--context-length 524288", + "--enable-dp-attention", + "--dp-size 8", + "--tp-size 8", + "--ep-size 8", + "--mem-fraction-static 0.87", + "--max-running-requests 256", + "--chunked-prefill-size 16384", + "--trust-remote-code", + "--swa-full-tokens-ratio 0.03", + "--prefill-attention-backend fa3", + "--decode-attention-backend fa3", + "--page-size 64", + "--moe-dense-tp-size 1", + "--cuda-graph-backend-decode full", + "--cuda-graph-backend-prefill disabled", + "--cuda-graph-max-bs-decode 32", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--speculative-draft-model-path {{MODEL_NAME}}", + "--speculative-draft-attention-backend fa3", + "--moe-a2a-backend deepep", + "--moe-runner-backend deep_gemm", + "--deepep-dispatcher-output-dtype bf16", + "--deepep-mode auto", + "--enable-nccl-nvls", + "--enable-multimodal", + "--enable-metrics", + "--tool-call-parser dots", + "--reasoning-parser qwen3", + "--watchdog-timeout 1800", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", quant: "fp8" }, + nnodes: 1, + verified: false, + env: [ + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", + "SGLANG_ENABLE_JIT_DEEPGEMM=1", + "SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD=8192", + "SGLANG_MAX_KV_CHUNK_CAPACITY=8192", + "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128", + "SGLANG_WARMUP_TIMEOUT=1800", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--context-length 524288", + "--enable-dp-attention", + "--dp-size 8", + "--tp-size 8", + "--ep-size 8", + "--mem-fraction-static 0.87", + "--max-running-requests 256", + "--chunked-prefill-size 16384", + "--trust-remote-code", + "--swa-full-tokens-ratio 0.03", + "--prefill-attention-backend fa3", + "--decode-attention-backend fa3", + "--page-size 64", + "--moe-dense-tp-size 1", + "--cuda-graph-backend-decode full", + "--cuda-graph-backend-prefill disabled", + "--cuda-graph-max-bs-decode 32", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--speculative-draft-model-path {{MODEL_NAME}}", + "--speculative-draft-attention-backend fa3", + "--moe-a2a-backend deepep", + "--moe-runner-backend auto", + "--deepep-dispatcher-output-dtype auto", + "--deepep-mode auto", + "--enable-nccl-nvls", + "--enable-multimodal", + "--enable-metrics", + "--reasoning-parser qwen3", + "--tool-call-parser dots", + "--watchdog-timeout 1800", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};