diff --git a/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx new file mode 100644 index 000000000..706b9d251 --- /dev/null +++ b/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -0,0 +1,453 @@ +--- +title: DeepSeek-V4 +metatags: + description: "Deploy DeepSeek-V4 with SGLang — a next-generation MoE model from DeepSeek. Blackwell deployments use the FP4 checkpoint; Hopper deployments use the FP8 checkpoint." +tag: NEW +--- + +## 1. Model Introduction + +**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. It ships as two Instruct repos (one per variant) plus matching Base repos: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VariantTotal paramsActive (MoE)Use
DeepSeek-V4-Flash284B13Bsingle-node serving: B200 / GB300 / H200 on 4 GPUs
DeepSeek-V4-Pro1.6T49Bhigh-capacity: B200 8 GPU / GB300 4 GPU / H200 16 GPU (2 nodes)
+ +The Instruct repos ship **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers all GPUs that support FP4). The Base (pre-trained only) variants — `DeepSeek-V4-Flash-Base`, `DeepSeek-V4-Pro-Base` — ship pure FP8 mixed and are **not** for chat / tool calling. + +**Key Features** (per the official model card): + +- **Hybrid Attention Architecture** — combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) for long-context efficiency. At 1M-token context, DeepSeek-V4-Pro uses only ~27% of per-token inference FLOPs and ~10% of KV cache compared with DeepSeek-V3.2. +- **Manifold-Constrained Hyper-Connections (mHC)** — strengthens residual connections, improving signal-propagation stability across layers while preserving expressivity. +- **Muon optimizer** — faster convergence and greater training stability. +- **Context length: 1M tokens**; pre-trained on 32T+ diverse, high-quality tokens. +- **Three reasoning modes**: *Non-think* (fast, intuitive responses), *Think High* (conscious logical analysis, slower but more accurate), *Think Max* (push reasoning to its fullest extent). Recommend a ≥ 384K context window when running Think Max. +- Ships with a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar (`<｜DSML｜tool_calls>` / `<｜DSML｜invoke>` / `<｜DSML｜parameter>`). + +**Recommended Generation Parameters:** `temperature=1.0`, `top_p=1.0` (per the official model card). + +**License:** MIT. + +**Resources:** + +- HuggingFace: [DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash), [DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) +- ModelScope: [DeepSeek-V4-Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash), [DeepSeek-V4-Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro) + +## 2. SGLang Installation + +SGLang offers multiple installation methods. Choose based on your hardware platform. + +Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. + +**Docker Images by Hardware Platform:** + + + + + + + + + + + + + + + + + + + + + + + + + + +
Hardware PlatformDocker Image
NVIDIA B200lmsysorg/sglang:deepseek-v4-blackwell
NVIDIA GB300lmsysorg/sglang:deepseek-v4-grace-blackwell
NVIDIA H200lmsysorg/sglang:deepseek-v4-hopper
+ +## 3. Model Deployment + +SGLang supports three main serving recipes for DeepSeek-V4 with different latency/throughput trade-offs (`low-latency`, `balanced`, `max-throughput`), plus specialized recipes for long-context (`cp`, prefill context-parallel) and prefill/decode disaggregation (`pd-disagg`). The interactive generator below emits the exact launch command for any `(hardware, variant, recipe)` combination. + +### 3.1 Basic Configuration + +**Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware + recipe combination. + +import { DeepSeekV4Deployment } from "/src/snippets/autoregressive/deepseek-v4-deployment.jsx"; + + + +### 3.2 Configuration Tips + +{/* TODO: expand this section as more recipes are validated end-to-end. */} + +**Concurrency & DeepEP dispatch buffer** + +Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs`, `--max-running-requests`, and the env together. + +The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised. + +**MTP (Multi-Token Prediction, EAGLE)** + +- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. +- `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch. +- `max-throughput`: MTP disabled — at saturation the verify step costs more than it saves. +- MTP currently requires `SGLANG_ENABLE_SPEC_V2=1`. + + + +**Hopper (H200) note** + +The H200 image and checkpoint are currently being uploaded — public path coming shortly. + +## 4. Model Invocation + +### 4.1 Basic Usage + +For basic API usage and request examples, see: + +- [Basic API Usage](../../../docs/basic_usage/send_request) + +Once the server is running (for example via the command generator above), send a request: + +```shell Command +curl http://localhost:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-ai/DeepSeek-V4-Flash", + "messages": [{"role": "user", "content": "What is 15% of 240?"}] + }' +``` + +> **PD-Disagg note**: if you deployed with the `pd-disagg` recipe from the generator above, the prefill server is on port `30000`, the decode server on `30001`, and the **router** on port `8000` — client traffic should target `http://localhost:8000`, not `:30000`. + +### 4.2 Advanced Usage + +#### 4.2.1 Reasoning Parser + +Enable the `deepseek-v4` reasoning parser (check the box in the [command panel above](#3-model-deployment)) to separate thinking from the final answer into `reasoning_content` vs `content`. + +**Streaming with Thinking Process:** + +```python Example +from openai import OpenAI + +client = OpenAI( + base_url="http://localhost:30000/v1", + api_key="EMPTY" +) + +response = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V4-Flash", + messages=[ + {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} + ], + max_tokens=2048, + extra_body={"chat_template_kwargs": {"thinking": True}}, + stream=True, +) + +thinking_started = False +has_thinking = False +has_answer = False + +for chunk in response: + if not chunk.choices: + continue + delta = chunk.choices[0].delta + + if getattr(delta, "reasoning_content", None): + if not thinking_started: + print("=============== Thinking =================", flush=True) + thinking_started = True + has_thinking = True + print(delta.reasoning_content, end="", flush=True) + + if delta.content: + if has_thinking and not has_answer: + print("\n=============== Content =================", flush=True) + has_answer = True + print(delta.content, end="", flush=True) + +print() +``` + +**Output Example:** + +```text Output +Pending update — replace with real server output after deployment. +``` + +#### 4.2.2 Tool Calling + +Enable the `deepseekv4` tool-call parser (check the box in the [command panel above](#3-model-deployment)) to surface structured tool calls via `message.tool_calls`. + +**Python Example (with Thinking Process):** + +```python Example +from openai import OpenAI + +client = OpenAI( + base_url="http://localhost:30000/v1", + api_key="EMPTY" +) + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string", "description": "The city name"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, + }, + "required": ["location"], + }, + }, + } +] + +response = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V4-Flash", + messages=[{"role": "user", "content": "What's the weather in Beijing?"}], + tools=tools, + extra_body={"chat_template_kwargs": {"thinking": True}}, + stream=True, +) + +thinking_started = False +has_thinking = False +tool_calls_accumulator = {} + +for chunk in response: + if not chunk.choices: + continue + delta = chunk.choices[0].delta + + if getattr(delta, "reasoning_content", None): + if not thinking_started: + print("=============== Thinking =================", flush=True) + thinking_started = True + has_thinking = True + print(delta.reasoning_content, end="", flush=True) + + if getattr(delta, "tool_calls", None): + if has_thinking and thinking_started: + print("\n=============== Content =================\n", flush=True) + thinking_started = False + for tool_call in delta.tool_calls: + index = tool_call.index + if index not in tool_calls_accumulator: + tool_calls_accumulator[index] = {"name": None, "arguments": ""} + if tool_call.function: + if tool_call.function.name: + tool_calls_accumulator[index]["name"] = tool_call.function.name + if tool_call.function.arguments: + tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments + + if delta.content: + print(delta.content, end="", flush=True) + +for index, tool_call in sorted(tool_calls_accumulator.items()): + print(f"Tool Call: {tool_call['name']}") + print(f" Arguments: {tool_call['arguments']}") + +print() +``` + +**Output Example:** + +```text Output +Pending update — replace with real server output after deployment. +``` + +## 5. Benchmark + +### 5.1 Speed Benchmark on Blackwell + +**Test Environment:** + +- Hardware: NVIDIA B200 GPU (4x) +- Model: DeepSeek-V4-Flash (FP4) +- Tensor Parallelism: 4 +- sglang version: Pending update + +We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. + +#### 5.1.1 Latency-Sensitive Benchmark + +- **Model Deployment Command:** see the [command panel above](#3-model-deployment). + +- Benchmark Command: + +```shell Command +python3 -m sglang.bench_serving \ + --backend sglang \ + --host 127.0.0.1 \ + --port 30000 \ + --model deepseek-ai/DeepSeek-V4-Flash \ + --random-input-len 1024 \ + --random-output-len 1024 \ + --num-prompts 10 \ + --max-concurrency 1 +``` + +- **Test Results:** + +```text Output +Pending update — replace with real bench_serving output after the latency run. +``` + +#### 5.1.2 Throughput-Sensitive Benchmark + +- **Model Deployment Command:** see the [command panel above](#3-model-deployment). + +- Benchmark Command: + +```shell Command +python3 -m sglang.bench_serving \ + --backend sglang \ + --host 127.0.0.1 \ + --port 30000 \ + --model deepseek-ai/DeepSeek-V4-Flash \ + --random-input-len 1024 \ + --random-output-len 1024 \ + --num-prompts 1000 \ + --max-concurrency 100 +``` + +- **Test Results:** + +```text Output +Pending update — replace with real bench_serving output after the throughput run. +``` + +### 5.2 Accuracy Benchmark + +#### 5.2.1 GSM8K Benchmark + +- **Benchmark Command:** + +```shell Command +python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000 +``` + +- **Test Results:** + - DeepSeek-V4-Flash (FP4, Blackwell) + ``` + Pending update + ``` + - DeepSeek-V4-Flash (FP8, Hopper) + ``` + Pending update + ``` + +#### 5.2.2 MMLU Benchmark + +- **Benchmark Command:** + +```shell Command +cd sglang +bash benchmark/mmlu/download_data.sh +python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 30000 +``` + +- **Test Results:** + - DeepSeek-V4-Flash (FP4, Blackwell) + ``` + Pending update + ``` + - DeepSeek-V4-Flash (FP8, Hopper) + ``` + Pending update + ``` + +### 5.3 Speed Benchmark on Hopper + +**Test Environment:** + +- Hardware: NVIDIA H200 GPU (4x) +- Model: DeepSeek-V4-Flash (FP8) +- Tensor Parallelism: 4 +- sglang version: Pending update + +#### 5.3.1 Latency-Sensitive Benchmark + +- **Model Deployment Command:** see the [command panel above](#3-model-deployment). + +- Benchmark Command: + +```shell Command +python3 -m sglang.bench_serving \ + --backend sglang \ + --host 127.0.0.1 \ + --port 30000 \ + --model deepseek-ai/DeepSeek-V4-Flash \ + --random-input-len 1024 \ + --random-output-len 1024 \ + --num-prompts 10 \ + --max-concurrency 1 +``` + +- **Test Results:** + +```text Output +Pending update — replace with real bench_serving output after the latency run. +``` + +#### 5.3.2 Throughput-Sensitive Benchmark + +- **Model Deployment Command:** see the [command panel above](#3-model-deployment). + +- Benchmark Command: + +```shell Command +python3 -m sglang.bench_serving \ + --backend sglang \ + --host 127.0.0.1 \ + --port 30000 \ + --model deepseek-ai/DeepSeek-V4-Flash \ + --random-input-len 1024 \ + --random-output-len 1024 \ + --num-prompts 1000 \ + --max-concurrency 100 +``` + +- **Test Results:** + +```text Output +Pending update — replace with real bench_serving output after the throughput run. +``` diff --git a/docs_new/cookbook/autoregressive/intro.mdx b/docs_new/cookbook/autoregressive/intro.mdx index 513c30367..966c21985 100644 --- a/docs_new/cookbook/autoregressive/intro.mdx +++ b/docs_new/cookbook/autoregressive/intro.mdx @@ -16,7 +16,7 @@ metatags: { + // DeepSeek-V4 deployment matrix (small / real checkpoint): + // Hardware × Recipe → concrete launch command. + // + // Hardware (quantization determined by GPU generation): + // B200 → FP4 weights, Flash TP=4 / Pro TP=8 single-node + // GB300 → FP4 weights, Flash TP=4 / Pro TP=4 single-node + // H200 → FP8 weights, Flash TP=4 / Pro TP=16 2-node + // Model variant → HF slug: + // Flash (285B) → deepseek-ai/DeepSeek-V4-Flash + // Pro (1.6T) → deepseek-ai/DeepSeek-V4-Pro + // + // Recipe: + // low-latency → TP(+DP on H200 no, Blackwell no), MTP 3/4 + // balanced → DP-attn + DeepEP + MTP 1/2 + // max-throughput → DP-attn + DeepEP, no MTP + // cp → TP + DeepEP + context-parallel flags, no MTP + // pd-disagg → 1P1D (prefill + decode + router), separate commands shown together + // + // HF slugs, parser names, and `sglang serve` flag parity are all confirmed — + // see cookbook_v2/DISCUSSION.md ("人类提供的事实" and 设计决定 §3). + + const options = { + hardware: { + name: "hardware", + title: "Hardware Platform", + items: [ + { id: "b200", label: "B200 (FP4)", default: true }, + { id: "gb300", label: "GB300 (FP4)", default: false }, + { id: "h200", label: "H200 (FP8)", default: false }, + ], + }, + modelSize: { + name: "modelSize", + title: "Model Variant", + items: [ + { id: "small", label: "Flash", default: true, subtitle: "285B" }, + { id: "big", label: "Pro", default: false, subtitle: "1.6T" }, + ], + }, + recipe: { + name: "recipe", + title: "Recipe", + items: [ + { id: "low-latency", label: "Low-Latency", default: true, subtitle: "MTP 3/4" }, + { id: "balanced", label: "Balanced", default: false, subtitle: "MTP 1/2 + DeepEP" }, + { id: "max-throughput", label: "Max-Throughput", default: false, subtitle: "DP + DeepEP" }, + { id: "cp", label: "Context-Parallel", default: false, subtitle: "long prompts" }, + { id: "pd-disagg", label: "PD-Disagg", default: false, subtitle: "1P + 1D + router" }, + ], + }, + reasoningParser: { + name: "reasoningParser", + title: "Reasoning Parser", + items: [ + { id: "disabled", label: "Disabled", default: true }, + { id: "enabled", label: "Enabled", default: false, subtitle: "deepseek-v4" }, + ], + }, + toolcall: { + name: "toolcall", + title: "Tool Call Parser", + items: [ + { id: "disabled", label: "Disabled", default: true }, + { id: "enabled", label: "Enabled", default: false, subtitle: "deepseekv4" }, + ], + }, + }; + + const resolveItems = (option) => option.items; + + const getInitialState = () => { + const initialState = {}; + for (const [key, option] of Object.entries(options)) { + const items = resolveItems(option); + const def = items.find((i) => i.default && !i.disabled) || items.find((i) => !i.disabled) || items[0]; + initialState[key] = def.id; + } + return initialState; + }; + + const [values, setValues] = useState(getInitialState); + const [isDark, setIsDark] = useState(false); + + useEffect(() => { + const checkDarkMode = () => { + const html = document.documentElement; + const isDarkMode = + html.classList.contains("dark") || + html.getAttribute("data-theme") === "dark" || + html.style.colorScheme === "dark"; + setIsDark(isDarkMode); + }; + checkDarkMode(); + const observer = new MutationObserver(checkDarkMode); + observer.observe(document.documentElement, { + attributes: true, + attributeFilter: ["class", "data-theme", "style"], + }); + return () => observer.disconnect(); + }, []); + + const handleRadioChange = (optionName, value) => { + setValues((prev) => ({ ...prev, [optionName]: value })); + }; + + // ============================================================================ + // generateCommand — strict mirror of sunrise_allinone.py LAUNCH_COMMANDS + // for BOTH small and big (1.6T) real-checkpoint rows. + // + // SOURCE OF TRUTH: sunrise_final/sunrise_allinone.py LAUNCH_COMMANDS dict. + // Allowed deviations are documented in cookbook_v2/DISCUSSION.md + // → "Human-approved diffs from allinone": + // 1. NVSHMEM env (B200) removed — personal hardware NIC mapping + // 2. Model path uses HF slug instead of allinone's local paths + // 3. `sglang serve` instead of `python3 -m sglang.launch_server` + // 4. (retired — big is now a real ckpt and exposed) + // 5. GB300 PD MNNVL topology envs (MC_FORCE_MNNVL / NCCL_*) removed; + // SGLANG_MOONCAKE_CUSTOM_MEM_POOL kept. + // + // Any other diff vs allinone is a bug — fix the JSX, not the whitelist. + // ============================================================================ + + // === SHARED BEGIN === + // Constants reachable by both generateCommand and buildPDDisaggCommand. + // verify_commands.mjs also scrapes this block between the SHARED markers and + // prepends it to the extracted function bodies (since `new Function(body)` + // loses closure scope). Don't rename the markers. + + // Per (hardware, modelSize) spec derived from allinone _MODEL_SPEC. + // "small" (JSX id) = DeepSeek-V4-Flash (285B); "big" = DeepSeek-V4-Pro (1.6T). + // The internal ids match allinone's model="small" / model="big" keys so the + // verify_commands.py diff is mechanical. One HF repo per variant holds both + // FP8 and FP4 weights (quantization picked by hardware, not by repo suffix). + const HW_SIZE_SPEC = { + "b200|small": { slug: "deepseek-ai/DeepSeek-V4-Flash", tp: 4, multinode: false }, + "b200|big": { slug: "deepseek-ai/DeepSeek-V4-Pro", tp: 8, multinode: false }, + "gb300|small": { slug: "deepseek-ai/DeepSeek-V4-Flash", tp: 4, multinode: false }, + "gb300|big": { slug: "deepseek-ai/DeepSeek-V4-Pro", tp: 4, multinode: false }, + // H200 needs a separate FP8-only Instruct ckpt (Flash / Pro public repos + // ship FP4-mixed weights). That ckpt is still being uploaded, so we emit a + // placeholder that fails loudly on copy-paste instead of silently pulling + // the wrong weights. Replace with the real slug once Hopper ckpts are public. + "h200|small": { slug: "", tp: 4, multinode: false }, + "h200|big": { slug: "", tp: 16, multinode: true, nnodes: 2 }, + }; + // Per (hardware, modelSize) PD role TP (from allinone _PD_SPEC). + const PD_TP_SPEC = { + "b200|small": { tp: 2, multinode: false }, + "b200|big": { tp: 8, multinode: false }, + "gb300|small": { tp: 4, multinode: false }, + "gb300|big": { tp: 4, multinode: false }, + "h200|small": { tp: 4, multinode: false }, + "h200|big": { tp: 16, multinode: true, nnodes: 2 }, + }; + // Recipes that have been end-to-end verified on the latest (Flash/Pro) HF + // checkpoints. Every cell NOT listed here is emitted with its entire body + // commented out (every line prefixed with `# `) plus a "being verified" + // banner on top — so copy-pasting an unverified command is a no-op in shell. + // To mark a cell verified, add its "hardware|modelSize|recipe" string here + // and the cell renders as a normal, runnable command. + // pd-disagg is verified as a single unit (both prefill and decode together). + const VERIFIED_RECIPES = new Set([ + "b200|small|low-latency", + "b200|big|low-latency", + ]); + const BEING_VERIFIED_NOTE = + "# NOTE: this recipe is being verified on the latest checkpoint"; + + // Prefix every line with "# " so the whole command becomes a shell no-op. + const commentOutCommand = (cmd) => + cmd + .split("\n") + .map((line) => (line.length ? `# ${line}` : "#")) + .join("\n"); + + // DeepEP large SMS flag (allinone _DEEPEP_LARGE_SMS_FLAG). + const DEEPEP_LARGE_SMS_FLAG = + ` --deepep-config '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'`; + + // Multi-node flags (renders with / placeholders; + // allinone template uses {node0_ip} / {node_rank} that verify_commands.py formats + // with the same placeholder strings so dynamic-diff stays exact). + const multiNodeFlags = (nnodes) => [ + ` --nnodes ${nnodes}`, + ` --node-rank `, + ` --dist-init-addr :20000`, + ]; + + const prependMultiNodeNote = (cmd, nnodes) => + `# Multi-node (${nnodes} nodes). Run the same command on every node with:\n` + + `# = 0 on the head node, 1..${nnodes - 1} on the others\n` + + `# = IP of the head node (reachable from all others)\n` + + `${cmd}`; + // === SHARED END === + + const generateCommand = () => { + const { hardware, modelSize, recipe, reasoningParser, toolcall } = values; + const specKey = `${hardware}|${modelSize}`; + const spec = HW_SIZE_SPEC[specKey]; + const { slug, tp, multinode, nnodes } = spec; + const isBig = modelSize === "big"; + + if (recipe === "pd-disagg") { + return buildPDDisaggCommand(hardware, modelSize); + } + + // ---- env ---- + // _LAUNCH_HEAD always prepends these: + const COMMON_ENV = ["SGLANG_JIT_DEEPGEMM_PRECOMPILE=0"]; + // Per-hardware env (whitelist #1: NVSHMEM removed for B200). + const HW_ENV = { + h200: ["SGLANG_DSV4_FP4_EXPERTS=0"], // allinone _ENV_H200 + b200: [], // _ENV_B200 minus NVSHMEM + gb300: [], // _ENV_GB300 + }[hardware]; + + // Recipe-specific env (matches allinone exactly, taking size into account). + const recipeEnv = []; + if (recipe === "low-latency") { + // H200 big low-latency has extra dispatch-token cap (allinone line 233). + if (hardware === "h200" && isBig) { + recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128"); + } + } else if (recipe === "balanced") { + if (hardware === "h200") { + recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"); + } else { + // Blackwell: small=1024, big=256 (allinone ternary). + recipeEnv.push(isBig + ? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256" + : "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"); + } + } else if (recipe === "max-throughput") { + if (hardware === "h200") { + recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"); + } else { + recipeEnv.push(isBig + ? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256" + : "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"); + } + } else if (recipe === "cp") { + recipeEnv.push("SGLANG_OPT_USE_JIT_INDEXER_METADATA=1"); + if (hardware === "h200") { + recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"); + } else { + // Blackwell cp: small=1024, big=256 (allinone ternary). + recipeEnv.push(isBig + ? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256" + : "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"); + } + } + // SGLANG_ENABLE_SPEC_V2=1 was in allinone's _ENV_MTP for low-latency / balanced + // recipes, but V4 auto-enables spec-v2 when MTP is detected — human confirmed + // the env is redundant on the public cookbook path. Kept as a no-op reference + // in allinone for legacy runs. + + // ---- flags ---- + const flags = []; + flags.push(" --trust-remote-code"); // _LAUNCH_HEAD + flags.push(` --model-path ${slug}`); + + if (recipe === "low-latency") { + // allinone: + // H200 small: pure TP + MTP_314 + // H200 big: DP-attn + DeepEP + MTP_314 + cg=32 max-run=64 + multi-node + mem-frac 0.82 + // Blackwell: TP + flashinfer_mxfp4 + MTP_314 + chunked-prefill-size 4096 + autotune-fix + // Big Blackwell additionally: mem-frac 0.82 + flags.push(` --tp ${tp}`); + if (hardware === "h200" && isBig) { + flags.push(` --dp ${tp}`); + flags.push(" --enable-dp-attention"); + } + if (multinode) flags.push(...multiNodeFlags(nnodes)); + if (hardware === "h200" && isBig) { + flags.push(" --moe-a2a-backend deepep"); + } + if (hardware !== "h200") { + flags.push(" --moe-runner-backend flashinfer_mxfp4"); + } + if (hardware === "h200" && isBig) { + flags.push(" --cuda-graph-max-bs 32"); + flags.push(" --max-running-requests 64"); + } + // MTP 3/4 + flags.push(" --speculative-algo EAGLE"); + flags.push(" --speculative-num-steps 3"); + flags.push(" --speculative-eagle-topk 1"); + flags.push(" --speculative-num-draft-tokens 4"); + if (hardware !== "h200") { + flags.push(" --chunked-prefill-size 4096"); + flags.push(" --disable-flashinfer-autotune"); + } + if (isBig) flags.push(" --mem-fraction-static 0.82"); + } else if (recipe === "balanced") { + // allinone balanced: TP + DP + DP-attn + DeepEP + MTP_112. + // H200 small: cg=128 max-run=128 | H200 big: cg=128 max-run=128 (same) + // B200 small: no cg/max-run | B200 big: cg=64 max-run=128 + // GB300 small: no cg/max-run | GB300 big: cg=128 max-run=256 + flags.push(` --tp ${tp}`); + flags.push(` --dp ${tp}`); + flags.push(" --enable-dp-attention"); + if (multinode) flags.push(...multiNodeFlags(nnodes)); + flags.push(" --moe-a2a-backend deepep"); + flags.push(" --speculative-algo EAGLE"); + flags.push(" --speculative-num-steps 1"); + flags.push(" --speculative-eagle-topk 1"); + flags.push(" --speculative-num-draft-tokens 2"); + if (isBig) flags.push(" --mem-fraction-static 0.82"); + if (hardware === "h200") { + flags.push(" --cuda-graph-max-bs 128"); + flags.push(" --max-running-requests 128"); + } else if (isBig && hardware === "b200") { + flags.push(" --cuda-graph-max-bs 64"); + flags.push(" --max-running-requests 128"); + } else if (isBig && hardware === "gb300") { + flags.push(" --cuda-graph-max-bs 128"); + flags.push(" --max-running-requests 256"); + } + // allinone H200 gates DEEPEP_LARGE_SMS_FLAG on !multinode — only H200 big + // is multi-node; all Blackwell cells get the flag unconditionally. + if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG); + } else if (recipe === "max-throughput") { + // allinone max-throughput: TP + DP + DP-attn + DeepEP (NO MTP). + // H200 small: cg=128 max-run=256 | H200 big: cg=128 max-run=256 (same) + // B200 small: no cg/max-run | B200 big: cg=64 max-run=256 + // GB300 small: no cg/max-run | GB300 big: cg=128 max-run=256 + flags.push(` --tp ${tp}`); + flags.push(` --dp ${tp}`); + flags.push(" --enable-dp-attention"); + if (multinode) flags.push(...multiNodeFlags(nnodes)); + flags.push(" --moe-a2a-backend deepep"); + if (isBig) flags.push(" --mem-fraction-static 0.82"); + if (hardware === "h200") { + flags.push(" --cuda-graph-max-bs 128"); + flags.push(" --max-running-requests 256"); + } else if (isBig && hardware === "b200") { + flags.push(" --cuda-graph-max-bs 64"); + flags.push(" --max-running-requests 256"); + } else if (isBig && hardware === "gb300") { + flags.push(" --cuda-graph-max-bs 128"); + flags.push(" --max-running-requests 256"); + } + if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG); + } else if (recipe === "cp") { + // allinone cp: TP (NO --dp) + DeepEP + _CP_FLAGS (mem-frac 0.78, max-run 1024). + // Blackwell big additionally: mem-frac 0.70 (overrides), cg=256, max-run=256. + // No flashinfer_mxfp4 even on Blackwell (allinone omits). + flags.push(` --tp ${tp}`); + if (multinode) flags.push(...multiNodeFlags(nnodes)); + flags.push(" --moe-a2a-backend deepep"); + flags.push(" --enable-nsa-prefill-context-parallel"); + flags.push(" --nsa-prefill-cp-mode round-robin-split"); + flags.push(" --chunked-prefill-size 16384"); + flags.push(" --mem-fraction-static 0.78"); + flags.push(" --max-running-requests 1024"); + if (isBig && hardware !== "h200") { + // Blackwell big cp: extra overrides. allinone emits these AFTER _CP_FLAGS, + // so two --mem-fraction-static appear — argparse last-wins (0.70 beats 0.78). + flags.push(" --mem-fraction-static 0.70"); + flags.push(" --cuda-graph-max-bs 256"); + flags.push(" --max-running-requests 256"); + } + // H200 CP gates DEEPEP_LARGE_SMS_FLAG on !multinode; Blackwell always gets it. + if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG); + } + + // Optional parsers (cookbook UI extension; not in allinone — opt-in toggles only). + if (toolcall === "enabled") flags.push(" --tool-call-parser deepseekv4"); + if (reasoningParser === "enabled") flags.push(" --reasoning-parser deepseek-v4"); + + flags.push(" --host 0.0.0.0"); + flags.push(" --port 30000"); + + // Assemble: [HW env] [recipe env] [common env] \ sglang serve \ flags... + const envAll = [...HW_ENV, ...recipeEnv, ...COMMON_ENV]; + const envBlock = envAll.length ? envAll.join(" \\\n") + " \\\n" : ""; + const base = `${envBlock}sglang serve \\\n${flags.join(" \\\n")}`; + const withMultinode = multinode ? prependMultiNodeNote(base, nnodes) : base; + const verifyKey = `${hardware}|${modelSize}|${recipe}`; + return VERIFIED_RECIPES.has(verifyKey) + ? withMultinode + : `${BEING_VERIFIED_NOTE}\n${commentOutCommand(withMultinode)}`; + }; + + // ============================================================================ + // buildPDDisaggCommand — mirror of allinone pd-p / pd-d for small AND big. + // + // _PD_SPEC[(hw, size)] → tp (and whether multinode). + // H200-fp8 small: tp=4 single-node, ib=mlx5_0 + // H200-fp8 big: tp=16 2-node, ib=mlx5_0 + // B200 small: tp=2 single-node, ib=mlx5_7 + // B200 big: tp=8 single-node, ib=mlx5_7 + // GB300 small/big: tp=4 single-node, ib="" (uses MNNVL, no IB device) + // + // deepep flag only on Blackwell PD; H200 PD does NOT use deepep. + // cap_env (SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024) only on B200 decode. + // SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True only on GB300. + // --dist-init-addr for disagg wiring only on non-GB300. + // --max-running-requests 256 only on decode (PD decode can't retract). + // No flashinfer_mxfp4 / autotune-fix / MTP / mem-fraction-static on PD (allinone omits). + // ============================================================================ + const buildPDDisaggCommand = (hardware, modelSize) => { + const specKey = `${hardware}|${modelSize}`; + const { tp: pdTp, multinode, nnodes } = PD_TP_SPEC[specKey]; + const slug = HW_SIZE_SPEC[specKey].slug; + const ibDevice = { h200: "mlx5_0", b200: "mlx5_7", gb300: "" }[hardware]; + const isGB300 = hardware === "gb300"; + const isBlackwell = hardware === "b200" || isGB300; + + const HW_ENV = { + h200: ["SGLANG_DSV4_FP4_EXPERTS=0"], + b200: [], + gb300: [], + }[hardware]; + // Whitelist #5: only SGLANG_MOONCAKE_CUSTOM_MEM_POOL kept; MC_FORCE_MNNVL / + // NCCL_MNNVL_ENABLE / NCCL_CUMEM_ENABLE stripped (personal-cluster topology). + const MNNVL_ENV = isGB300 ? ["SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True"] : []; + const COMMON_ENV = ["SGLANG_JIT_DEEPGEMM_PRECOMPILE=0"]; + + const buildRole = (mode, port, distPort) => { + const roleEnv = []; + if (hardware === "b200" && mode === "decode") { + roleEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"); + } + const envAll = [...HW_ENV, ...roleEnv, ...MNNVL_ENV, ...COMMON_ENV]; + const envBlock = envAll.length ? envAll.join(" \\\n") + " \\\n" : ""; + + const flags = []; + flags.push(" --trust-remote-code"); + flags.push(` --model-path ${slug}`); + flags.push(` --tp ${pdTp}`); + flags.push(` --dp ${pdTp}`); + flags.push(" --enable-dp-attention"); + if (multinode) flags.push(...multiNodeFlags(nnodes)); + if (isBlackwell) flags.push(" --moe-a2a-backend deepep"); + flags.push(` --disaggregation-mode ${mode}`); + flags.push(" --disaggregation-transfer-backend mooncake"); + if (ibDevice) flags.push(` --disaggregation-ib-device ${ibDevice}`); + if (!isGB300) flags.push(` --dist-init-addr 127.0.0.1:${distPort}`); + if (mode === "decode") flags.push(" --max-running-requests 256"); + flags.push(" --host 0.0.0.0"); + flags.push(` --port ${port}`); + + return `${envBlock}sglang serve \\\n${flags.join(" \\\n")}`; + }; + + const prefillHeader = multinode + ? `# --- Prefill role (port 30000) — multi-node, run on each of ${nnodes} nodes ---` + : "# --- Prefill role (port 30000) ---"; + const decodeHeader = multinode + ? `# --- Decode role (port 30001) — multi-node, run on each of ${nnodes} nodes ---` + : "# --- Decode role (port 30001) ---"; + + const prefill = `${prefillHeader}\n${buildRole("prefill", 30000, 30335)}`; + const decode = `${decodeHeader}\n${buildRole("decode", 30001, 30435)}`; + const router = `# --- Router (port 8000) --- +python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://127.0.0.1:30000 \\ + --decode http://127.0.0.1:30001 \\ + --host 0.0.0.0 --port 8000 \\ + --disable-circuit-breaker \\ + --health-check-interval-secs 999999`; + + const full = `${prefill}\n\n${decode}\n\n${router}`; + const verifyKey = `${hardware}|${modelSize}|pd-disagg`; + return VERIFIED_RECIPES.has(verifyKey) + ? full + : `${BEING_VERIFIED_NOTE}\n${commentOutCommand(full)}`; + }; + + // ---- styles ---- + const containerStyle = { maxWidth: "900px", margin: "0 auto", display: "flex", flexDirection: "column", gap: "4px" }; + const cardStyle = { + padding: "8px 12px", + border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, + borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, + borderRadius: "4px", + display: "flex", + alignItems: "center", + gap: "12px", + background: isDark ? "#1f2937" : "#fff", + }; + const titleStyle = { fontSize: "13px", fontWeight: "600", minWidth: "140px", flexShrink: 0, color: isDark ? "#e5e7eb" : "inherit" }; + const itemsStyle = { display: "flex", rowGap: "2px", columnGap: "6px", flexWrap: "wrap", alignItems: "center", flex: 1 }; + const labelBaseStyle = { + padding: "4px 10px", + border: `1px solid ${isDark ? "#9ca3af" : "#d1d5db"}`, + borderRadius: "3px", + cursor: "pointer", + display: "inline-flex", + flexDirection: "column", + alignItems: "center", + justifyContent: "center", + fontWeight: "500", + fontSize: "13px", + transition: "all 0.2s", + userSelect: "none", + minWidth: "45px", + textAlign: "center", + flex: 1, + background: isDark ? "#374151" : "#fff", + color: isDark ? "#e5e7eb" : "inherit", + }; + const checkedStyle = { background: "#D45D44", color: "white", borderColor: "#D45D44" }; + const disabledStyle = { cursor: "not-allowed", opacity: 0.4 }; + const subtitleStyle = { display: "block", fontSize: "9px", marginTop: "1px", lineHeight: "1.1", opacity: 0.7 }; + const commandDisplayStyle = { + flex: 1, + padding: "12px 16px", + background: isDark ? "#111827" : "#f5f5f5", + borderRadius: "6px", + fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", + fontSize: "12px", + lineHeight: "1.5", + color: isDark ? "#e5e7eb" : "#374151", + whiteSpace: "pre-wrap", + overflowX: "auto", + margin: 0, + border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, + }; + + return ( +
+ {Object.entries(options).map(([key, option]) => { + const items = resolveItems(option); + return ( +
+
{option.title}
+
+ {items.map((item) => { + const isChecked = values[option.name] === item.id; + const isDisabled = !!item.disabled; + return ( + + ); + })} +
+
+ ); + })} +
+
Run this Command:
+ 
{generateCommand()}
+
+
+ ); +};