Add DeepSeek V4 cookbook (#23605)
Co-authored-by: fzyzcjy <ch271828n@outlook.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
fzyzcjy
zijiexia
fzyzcjy
zijiexia
parent
30909cbeeb
commit
492883c8ca
@@ -0,0 +1,453 @@
|
||||
---
|
||||
title: DeepSeek-V4
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-V4 with SGLang — a next-generation MoE model from DeepSeek. Blackwell deployments use the FP4 checkpoint; Hopper deployments use the FP8 checkpoint."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. It ships as two Instruct repos (one per variant) plus matching Base repos:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "30%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "40%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Total params</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Active (MoE)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash">DeepSeek-V4-Flash</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>284B</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving: B200 / GB300 / H200 on 4 GPUs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.6T</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>49B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>high-capacity: B200 8 GPU / GB300 4 GPU / H200 16 GPU (2 nodes)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The Instruct repos ship **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers all GPUs that support FP4). The Base (pre-trained only) variants — `DeepSeek-V4-Flash-Base`, `DeepSeek-V4-Pro-Base` — ship pure FP8 mixed and are **not** for chat / tool calling.
|
||||
|
||||
**Key Features** (per the official model card):
|
||||
|
||||
- **Hybrid Attention Architecture** — combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) for long-context efficiency. At 1M-token context, DeepSeek-V4-Pro uses only ~27% of per-token inference FLOPs and ~10% of KV cache compared with DeepSeek-V3.2.
|
||||
- **Manifold-Constrained Hyper-Connections (mHC)** — strengthens residual connections, improving signal-propagation stability across layers while preserving expressivity.
|
||||
- **Muon optimizer** — faster convergence and greater training stability.
|
||||
- **Context length: 1M tokens**; pre-trained on 32T+ diverse, high-quality tokens.
|
||||
- **Three reasoning modes**: *Non-think* (fast, intuitive responses), *Think High* (conscious logical analysis, slower but more accurate), *Think Max* (push reasoning to its fullest extent). Recommend a ≥ 384K context window when running Think Max.
|
||||
- Ships with a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar (`<｜DSML｜tool_calls>` / `<｜DSML｜invoke>` / `<｜DSML｜parameter>`).
|
||||
|
||||
**Recommended Generation Parameters:** `temperature=1.0`, `top_p=1.0` (per the official model card).
|
||||
|
||||
**License:** MIT.
|
||||
|
||||
**Resources:**
|
||||
|
||||
- HuggingFace: [DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash), [DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)
|
||||
- ModelScope: [DeepSeek-V4-Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash), [DeepSeek-V4-Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. Choose based on your hardware platform.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "55%"}} />
|
||||
<col style={{width: "45%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>lmsysorg/sglang:deepseek-v4-blackwell</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>lmsysorg/sglang:deepseek-v4-grace-blackwell</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>lmsysorg/sglang:deepseek-v4-hopper</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
SGLang supports three main serving recipes for DeepSeek-V4 with different latency/throughput trade-offs (`low-latency`, `balanced`, `max-throughput`), plus specialized recipes for long-context (`cp`, prefill context-parallel) and prefill/decode disaggregation (`pd-disagg`). The interactive generator below emits the exact launch command for any `(hardware, variant, recipe)` combination.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware + recipe combination.
|
||||
|
||||
import { DeepSeekV4Deployment } from "/src/snippets/autoregressive/deepseek-v4-deployment.jsx";
|
||||
|
||||
<DeepSeekV4Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
{/* TODO: expand this section as more recipes are validated end-to-end. */}
|
||||
|
||||
**Concurrency & DeepEP dispatch buffer**
|
||||
|
||||
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs`, `--max-running-requests`, and the env together.
|
||||
|
||||
The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.
|
||||
|
||||
**MTP (Multi-Token Prediction, EAGLE)**
|
||||
|
||||
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
|
||||
- `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
|
||||
- `max-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
|
||||
- MTP currently requires `SGLANG_ENABLE_SPEC_V2=1`.
|
||||
|
||||
<a id="hopper-note" />
|
||||
|
||||
**Hopper (H200) note**
|
||||
|
||||
The H200 image and checkpoint are currently being uploaded — public path coming shortly.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, see:
|
||||
|
||||
- [Basic API Usage](../../../docs/basic_usage/send_request)
|
||||
|
||||
Once the server is running (for example via the command generator above), send a request:
|
||||
|
||||
```shell Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"messages": [{"role": "user", "content": "What is 15% of 240?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
> **PD-Disagg note**: if you deployed with the `pd-disagg` recipe from the generator above, the prefill server is on port `30000`, the decode server on `30001`, and the **router** on port `8000` — client traffic should target `http://localhost:8000`, not `:30000`.
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
Enable the `deepseek-v4` reasoning parser (check the box in the [command panel above](#3-model-deployment)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Pending update — replace with real server output after deployment.
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Enable the `deepseekv4` tool-call parser (check the box in the [command panel above](#3-model-deployment)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if getattr(delta, "tool_calls", None):
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {"name": None, "arguments": ""}
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]["name"] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Pending update — replace with real server output after deployment.
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark on Blackwell
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (4x)
|
||||
- Model: DeepSeek-V4-Flash (FP4)
|
||||
- Tensor Parallelism: 4
|
||||
- sglang version: Pending update
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** see the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
Pending update — replace with real bench_serving output after the latency run.
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** see the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
Pending update — replace with real bench_serving output after the throughput run.
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
- DeepSeek-V4-Flash (FP4, Blackwell)
|
||||
```
|
||||
Pending update
|
||||
```
|
||||
- DeepSeek-V4-Flash (FP8, Hopper)
|
||||
```
|
||||
Pending update
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
cd sglang
|
||||
bash benchmark/mmlu/download_data.sh
|
||||
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 30000
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
- DeepSeek-V4-Flash (FP4, Blackwell)
|
||||
```
|
||||
Pending update
|
||||
```
|
||||
- DeepSeek-V4-Flash (FP8, Hopper)
|
||||
```
|
||||
Pending update
|
||||
```
|
||||
|
||||
### 5.3 Speed Benchmark on Hopper
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 GPU (4x)
|
||||
- Model: DeepSeek-V4-Flash (FP8)
|
||||
- Tensor Parallelism: 4
|
||||
- sglang version: Pending update
|
||||
|
||||
#### 5.3.1 Latency-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** see the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
Pending update — replace with real bench_serving output after the latency run.
|
||||
```
|
||||
|
||||
#### 5.3.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** see the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
Pending update — replace with real bench_serving output after the throughput run.
|
||||
```
|
||||
@@ -16,7 +16,7 @@ metatags:
|
||||
<Card
|
||||
title="DeepSeek"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2"
|
||||
href="/cookbook/autoregressive/DeepSeek/DeepSeek-V4"
|
||||
img="/cards/logos/deepseek.png"
|
||||
/>
|
||||
<Card
|
||||
|
||||
@@ -940,6 +940,7 @@
|
||||
{
|
||||
"group": "DeepSeek",
|
||||
"pages": [
|
||||
"cookbook/autoregressive/DeepSeek/DeepSeek-V4",
|
||||
"cookbook/autoregressive/DeepSeek/DeepSeek-V3_2",
|
||||
"cookbook/autoregressive/DeepSeek/DeepSeek-V3_1",
|
||||
"cookbook/autoregressive/DeepSeek/DeepSeek-V3",
|
||||
|
||||
@@ -0,0 +1,569 @@
|
||||
export const DeepSeekV4Deployment = () => {
|
||||
// DeepSeek-V4 deployment matrix (small / real checkpoint):
|
||||
// Hardware × Recipe → concrete launch command.
|
||||
//
|
||||
// Hardware (quantization determined by GPU generation):
|
||||
// B200 → FP4 weights, Flash TP=4 / Pro TP=8 single-node
|
||||
// GB300 → FP4 weights, Flash TP=4 / Pro TP=4 single-node
|
||||
// H200 → FP8 weights, Flash TP=4 / Pro TP=16 2-node
|
||||
// Model variant → HF slug:
|
||||
// Flash (285B) → deepseek-ai/DeepSeek-V4-Flash
|
||||
// Pro (1.6T) → deepseek-ai/DeepSeek-V4-Pro
|
||||
//
|
||||
// Recipe:
|
||||
// low-latency → TP(+DP on H200 no, Blackwell no), MTP 3/4
|
||||
// balanced → DP-attn + DeepEP + MTP 1/2
|
||||
// max-throughput → DP-attn + DeepEP, no MTP
|
||||
// cp → TP + DeepEP + context-parallel flags, no MTP
|
||||
// pd-disagg → 1P1D (prefill + decode + router), separate commands shown together
|
||||
//
|
||||
// HF slugs, parser names, and `sglang serve` flag parity are all confirmed —
|
||||
// see cookbook_v2/DISCUSSION.md ("人类提供的事实" and 设计决定 §3).
|
||||
|
||||
const options = {
|
||||
hardware: {
|
||||
name: "hardware",
|
||||
title: "Hardware Platform",
|
||||
items: [
|
||||
{ id: "b200", label: "B200 (FP4)", default: true },
|
||||
{ id: "gb300", label: "GB300 (FP4)", default: false },
|
||||
{ id: "h200", label: "H200 (FP8)", default: false },
|
||||
],
|
||||
},
|
||||
modelSize: {
|
||||
name: "modelSize",
|
||||
title: "Model Variant",
|
||||
items: [
|
||||
{ id: "small", label: "Flash", default: true, subtitle: "285B" },
|
||||
{ id: "big", label: "Pro", default: false, subtitle: "1.6T" },
|
||||
],
|
||||
},
|
||||
recipe: {
|
||||
name: "recipe",
|
||||
title: "Recipe",
|
||||
items: [
|
||||
{ id: "low-latency", label: "Low-Latency", default: true, subtitle: "MTP 3/4" },
|
||||
{ id: "balanced", label: "Balanced", default: false, subtitle: "MTP 1/2 + DeepEP" },
|
||||
{ id: "max-throughput", label: "Max-Throughput", default: false, subtitle: "DP + DeepEP" },
|
||||
{ id: "cp", label: "Context-Parallel", default: false, subtitle: "long prompts" },
|
||||
{ id: "pd-disagg", label: "PD-Disagg", default: false, subtitle: "1P + 1D + router" },
|
||||
],
|
||||
},
|
||||
reasoningParser: {
|
||||
name: "reasoningParser",
|
||||
title: "Reasoning Parser",
|
||||
items: [
|
||||
{ id: "disabled", label: "Disabled", default: true },
|
||||
{ id: "enabled", label: "Enabled", default: false, subtitle: "deepseek-v4" },
|
||||
],
|
||||
},
|
||||
toolcall: {
|
||||
name: "toolcall",
|
||||
title: "Tool Call Parser",
|
||||
items: [
|
||||
{ id: "disabled", label: "Disabled", default: true },
|
||||
{ id: "enabled", label: "Enabled", default: false, subtitle: "deepseekv4" },
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
const resolveItems = (option) => option.items;
|
||||
|
||||
const getInitialState = () => {
|
||||
const initialState = {};
|
||||
for (const [key, option] of Object.entries(options)) {
|
||||
const items = resolveItems(option);
|
||||
const def = items.find((i) => i.default && !i.disabled) || items.find((i) => !i.disabled) || items[0];
|
||||
initialState[key] = def.id;
|
||||
}
|
||||
return initialState;
|
||||
};
|
||||
|
||||
const [values, setValues] = useState(getInitialState);
|
||||
const [isDark, setIsDark] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
const checkDarkMode = () => {
|
||||
const html = document.documentElement;
|
||||
const isDarkMode =
|
||||
html.classList.contains("dark") ||
|
||||
html.getAttribute("data-theme") === "dark" ||
|
||||
html.style.colorScheme === "dark";
|
||||
setIsDark(isDarkMode);
|
||||
};
|
||||
checkDarkMode();
|
||||
const observer = new MutationObserver(checkDarkMode);
|
||||
observer.observe(document.documentElement, {
|
||||
attributes: true,
|
||||
attributeFilter: ["class", "data-theme", "style"],
|
||||
});
|
||||
return () => observer.disconnect();
|
||||
}, []);
|
||||
|
||||
const handleRadioChange = (optionName, value) => {
|
||||
setValues((prev) => ({ ...prev, [optionName]: value }));
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
// generateCommand — strict mirror of sunrise_allinone.py LAUNCH_COMMANDS
|
||||
// for BOTH small and big (1.6T) real-checkpoint rows.
|
||||
//
|
||||
// SOURCE OF TRUTH: sunrise_final/sunrise_allinone.py LAUNCH_COMMANDS dict.
|
||||
// Allowed deviations are documented in cookbook_v2/DISCUSSION.md
|
||||
// → "Human-approved diffs from allinone":
|
||||
// 1. NVSHMEM env (B200) removed — personal hardware NIC mapping
|
||||
// 2. Model path uses HF slug instead of allinone's local paths
|
||||
// 3. `sglang serve` instead of `python3 -m sglang.launch_server`
|
||||
// 4. (retired — big is now a real ckpt and exposed)
|
||||
// 5. GB300 PD MNNVL topology envs (MC_FORCE_MNNVL / NCCL_*) removed;
|
||||
// SGLANG_MOONCAKE_CUSTOM_MEM_POOL kept.
|
||||
//
|
||||
// Any other diff vs allinone is a bug — fix the JSX, not the whitelist.
|
||||
// ============================================================================
|
||||
|
||||
// === SHARED BEGIN ===
|
||||
// Constants reachable by both generateCommand and buildPDDisaggCommand.
|
||||
// verify_commands.mjs also scrapes this block between the SHARED markers and
|
||||
// prepends it to the extracted function bodies (since `new Function(body)`
|
||||
// loses closure scope). Don't rename the markers.
|
||||
|
||||
// Per (hardware, modelSize) spec derived from allinone _MODEL_SPEC.
|
||||
// "small" (JSX id) = DeepSeek-V4-Flash (285B); "big" = DeepSeek-V4-Pro (1.6T).
|
||||
// The internal ids match allinone's model="small" / model="big" keys so the
|
||||
// verify_commands.py diff is mechanical. One HF repo per variant holds both
|
||||
// FP8 and FP4 weights (quantization picked by hardware, not by repo suffix).
|
||||
const HW_SIZE_SPEC = {
|
||||
"b200|small": { slug: "deepseek-ai/DeepSeek-V4-Flash", tp: 4, multinode: false },
|
||||
"b200|big": { slug: "deepseek-ai/DeepSeek-V4-Pro", tp: 8, multinode: false },
|
||||
"gb300|small": { slug: "deepseek-ai/DeepSeek-V4-Flash", tp: 4, multinode: false },
|
||||
"gb300|big": { slug: "deepseek-ai/DeepSeek-V4-Pro", tp: 4, multinode: false },
|
||||
// H200 needs a separate FP8-only Instruct ckpt (Flash / Pro public repos
|
||||
// ship FP4-mixed weights). That ckpt is still being uploaded, so we emit a
|
||||
// placeholder that fails loudly on copy-paste instead of silently pulling
|
||||
// the wrong weights. Replace with the real slug once Hopper ckpts are public.
|
||||
"h200|small": { slug: "<TO_BE_UPLOADED_DeepSeek-V4-Flash-hopper>", tp: 4, multinode: false },
|
||||
"h200|big": { slug: "<TO_BE_UPLOADED_DeepSeek-V4-Pro-hopper>", tp: 16, multinode: true, nnodes: 2 },
|
||||
};
|
||||
// Per (hardware, modelSize) PD role TP (from allinone _PD_SPEC).
|
||||
const PD_TP_SPEC = {
|
||||
"b200|small": { tp: 2, multinode: false },
|
||||
"b200|big": { tp: 8, multinode: false },
|
||||
"gb300|small": { tp: 4, multinode: false },
|
||||
"gb300|big": { tp: 4, multinode: false },
|
||||
"h200|small": { tp: 4, multinode: false },
|
||||
"h200|big": { tp: 16, multinode: true, nnodes: 2 },
|
||||
};
|
||||
// Recipes that have been end-to-end verified on the latest (Flash/Pro) HF
|
||||
// checkpoints. Every cell NOT listed here is emitted with its entire body
|
||||
// commented out (every line prefixed with `# `) plus a "being verified"
|
||||
// banner on top — so copy-pasting an unverified command is a no-op in shell.
|
||||
// To mark a cell verified, add its "hardware|modelSize|recipe" string here
|
||||
// and the cell renders as a normal, runnable command.
|
||||
// pd-disagg is verified as a single unit (both prefill and decode together).
|
||||
const VERIFIED_RECIPES = new Set([
|
||||
"b200|small|low-latency",
|
||||
"b200|big|low-latency",
|
||||
]);
|
||||
const BEING_VERIFIED_NOTE =
|
||||
"# NOTE: this recipe is being verified on the latest checkpoint";
|
||||
|
||||
// Prefix every line with "# " so the whole command becomes a shell no-op.
|
||||
const commentOutCommand = (cmd) =>
|
||||
cmd
|
||||
.split("\n")
|
||||
.map((line) => (line.length ? `# ${line}` : "#"))
|
||||
.join("\n");
|
||||
|
||||
// DeepEP large SMS flag (allinone _DEEPEP_LARGE_SMS_FLAG).
|
||||
const DEEPEP_LARGE_SMS_FLAG =
|
||||
` --deepep-config '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'`;
|
||||
|
||||
// Multi-node flags (renders with <node0-ip> / <node-rank> placeholders;
|
||||
// allinone template uses {node0_ip} / {node_rank} that verify_commands.py formats
|
||||
// with the same placeholder strings so dynamic-diff stays exact).
|
||||
const multiNodeFlags = (nnodes) => [
|
||||
` --nnodes ${nnodes}`,
|
||||
` --node-rank <node-rank>`,
|
||||
` --dist-init-addr <node0-ip>:20000`,
|
||||
];
|
||||
|
||||
const prependMultiNodeNote = (cmd, nnodes) =>
|
||||
`# Multi-node (${nnodes} nodes). Run the same command on every node with:\n` +
|
||||
`# <node-rank> = 0 on the head node, 1..${nnodes - 1} on the others\n` +
|
||||
`# <node0-ip> = IP of the head node (reachable from all others)\n` +
|
||||
`${cmd}`;
|
||||
// === SHARED END ===
|
||||
|
||||
const generateCommand = () => {
|
||||
const { hardware, modelSize, recipe, reasoningParser, toolcall } = values;
|
||||
const specKey = `${hardware}|${modelSize}`;
|
||||
const spec = HW_SIZE_SPEC[specKey];
|
||||
const { slug, tp, multinode, nnodes } = spec;
|
||||
const isBig = modelSize === "big";
|
||||
|
||||
if (recipe === "pd-disagg") {
|
||||
return buildPDDisaggCommand(hardware, modelSize);
|
||||
}
|
||||
|
||||
// ---- env ----
|
||||
// _LAUNCH_HEAD always prepends these:
|
||||
const COMMON_ENV = ["SGLANG_JIT_DEEPGEMM_PRECOMPILE=0"];
|
||||
// Per-hardware env (whitelist #1: NVSHMEM removed for B200).
|
||||
const HW_ENV = {
|
||||
h200: ["SGLANG_DSV4_FP4_EXPERTS=0"], // allinone _ENV_H200
|
||||
b200: [], // _ENV_B200 minus NVSHMEM
|
||||
gb300: [], // _ENV_GB300
|
||||
}[hardware];
|
||||
|
||||
// Recipe-specific env (matches allinone exactly, taking size into account).
|
||||
const recipeEnv = [];
|
||||
if (recipe === "low-latency") {
|
||||
// H200 big low-latency has extra dispatch-token cap (allinone line 233).
|
||||
if (hardware === "h200" && isBig) {
|
||||
recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128");
|
||||
}
|
||||
} else if (recipe === "balanced") {
|
||||
if (hardware === "h200") {
|
||||
recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256");
|
||||
} else {
|
||||
// Blackwell: small=1024, big=256 (allinone ternary).
|
||||
recipeEnv.push(isBig
|
||||
? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"
|
||||
: "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024");
|
||||
}
|
||||
} else if (recipe === "max-throughput") {
|
||||
if (hardware === "h200") {
|
||||
recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256");
|
||||
} else {
|
||||
recipeEnv.push(isBig
|
||||
? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"
|
||||
: "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024");
|
||||
}
|
||||
} else if (recipe === "cp") {
|
||||
recipeEnv.push("SGLANG_OPT_USE_JIT_INDEXER_METADATA=1");
|
||||
if (hardware === "h200") {
|
||||
recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024");
|
||||
} else {
|
||||
// Blackwell cp: small=1024, big=256 (allinone ternary).
|
||||
recipeEnv.push(isBig
|
||||
? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"
|
||||
: "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024");
|
||||
}
|
||||
}
|
||||
// SGLANG_ENABLE_SPEC_V2=1 was in allinone's _ENV_MTP for low-latency / balanced
|
||||
// recipes, but V4 auto-enables spec-v2 when MTP is detected — human confirmed
|
||||
// the env is redundant on the public cookbook path. Kept as a no-op reference
|
||||
// in allinone for legacy runs.
|
||||
|
||||
// ---- flags ----
|
||||
const flags = [];
|
||||
flags.push(" --trust-remote-code"); // _LAUNCH_HEAD
|
||||
flags.push(` --model-path ${slug}`);
|
||||
|
||||
if (recipe === "low-latency") {
|
||||
// allinone:
|
||||
// H200 small: pure TP + MTP_314
|
||||
// H200 big: DP-attn + DeepEP + MTP_314 + cg=32 max-run=64 + multi-node + mem-frac 0.82
|
||||
// Blackwell: TP + flashinfer_mxfp4 + MTP_314 + chunked-prefill-size 4096 + autotune-fix
|
||||
// Big Blackwell additionally: mem-frac 0.82
|
||||
flags.push(` --tp ${tp}`);
|
||||
if (hardware === "h200" && isBig) {
|
||||
flags.push(` --dp ${tp}`);
|
||||
flags.push(" --enable-dp-attention");
|
||||
}
|
||||
if (multinode) flags.push(...multiNodeFlags(nnodes));
|
||||
if (hardware === "h200" && isBig) {
|
||||
flags.push(" --moe-a2a-backend deepep");
|
||||
}
|
||||
if (hardware !== "h200") {
|
||||
flags.push(" --moe-runner-backend flashinfer_mxfp4");
|
||||
}
|
||||
if (hardware === "h200" && isBig) {
|
||||
flags.push(" --cuda-graph-max-bs 32");
|
||||
flags.push(" --max-running-requests 64");
|
||||
}
|
||||
// MTP 3/4
|
||||
flags.push(" --speculative-algo EAGLE");
|
||||
flags.push(" --speculative-num-steps 3");
|
||||
flags.push(" --speculative-eagle-topk 1");
|
||||
flags.push(" --speculative-num-draft-tokens 4");
|
||||
if (hardware !== "h200") {
|
||||
flags.push(" --chunked-prefill-size 4096");
|
||||
flags.push(" --disable-flashinfer-autotune");
|
||||
}
|
||||
if (isBig) flags.push(" --mem-fraction-static 0.82");
|
||||
} else if (recipe === "balanced") {
|
||||
// allinone balanced: TP + DP + DP-attn + DeepEP + MTP_112.
|
||||
// H200 small: cg=128 max-run=128 | H200 big: cg=128 max-run=128 (same)
|
||||
// B200 small: no cg/max-run | B200 big: cg=64 max-run=128
|
||||
// GB300 small: no cg/max-run | GB300 big: cg=128 max-run=256
|
||||
flags.push(` --tp ${tp}`);
|
||||
flags.push(` --dp ${tp}`);
|
||||
flags.push(" --enable-dp-attention");
|
||||
if (multinode) flags.push(...multiNodeFlags(nnodes));
|
||||
flags.push(" --moe-a2a-backend deepep");
|
||||
flags.push(" --speculative-algo EAGLE");
|
||||
flags.push(" --speculative-num-steps 1");
|
||||
flags.push(" --speculative-eagle-topk 1");
|
||||
flags.push(" --speculative-num-draft-tokens 2");
|
||||
if (isBig) flags.push(" --mem-fraction-static 0.82");
|
||||
if (hardware === "h200") {
|
||||
flags.push(" --cuda-graph-max-bs 128");
|
||||
flags.push(" --max-running-requests 128");
|
||||
} else if (isBig && hardware === "b200") {
|
||||
flags.push(" --cuda-graph-max-bs 64");
|
||||
flags.push(" --max-running-requests 128");
|
||||
} else if (isBig && hardware === "gb300") {
|
||||
flags.push(" --cuda-graph-max-bs 128");
|
||||
flags.push(" --max-running-requests 256");
|
||||
}
|
||||
// allinone H200 gates DEEPEP_LARGE_SMS_FLAG on !multinode — only H200 big
|
||||
// is multi-node; all Blackwell cells get the flag unconditionally.
|
||||
if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG);
|
||||
} else if (recipe === "max-throughput") {
|
||||
// allinone max-throughput: TP + DP + DP-attn + DeepEP (NO MTP).
|
||||
// H200 small: cg=128 max-run=256 | H200 big: cg=128 max-run=256 (same)
|
||||
// B200 small: no cg/max-run | B200 big: cg=64 max-run=256
|
||||
// GB300 small: no cg/max-run | GB300 big: cg=128 max-run=256
|
||||
flags.push(` --tp ${tp}`);
|
||||
flags.push(` --dp ${tp}`);
|
||||
flags.push(" --enable-dp-attention");
|
||||
if (multinode) flags.push(...multiNodeFlags(nnodes));
|
||||
flags.push(" --moe-a2a-backend deepep");
|
||||
if (isBig) flags.push(" --mem-fraction-static 0.82");
|
||||
if (hardware === "h200") {
|
||||
flags.push(" --cuda-graph-max-bs 128");
|
||||
flags.push(" --max-running-requests 256");
|
||||
} else if (isBig && hardware === "b200") {
|
||||
flags.push(" --cuda-graph-max-bs 64");
|
||||
flags.push(" --max-running-requests 256");
|
||||
} else if (isBig && hardware === "gb300") {
|
||||
flags.push(" --cuda-graph-max-bs 128");
|
||||
flags.push(" --max-running-requests 256");
|
||||
}
|
||||
if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG);
|
||||
} else if (recipe === "cp") {
|
||||
// allinone cp: TP (NO --dp) + DeepEP + _CP_FLAGS (mem-frac 0.78, max-run 1024).
|
||||
// Blackwell big additionally: mem-frac 0.70 (overrides), cg=256, max-run=256.
|
||||
// No flashinfer_mxfp4 even on Blackwell (allinone omits).
|
||||
flags.push(` --tp ${tp}`);
|
||||
if (multinode) flags.push(...multiNodeFlags(nnodes));
|
||||
flags.push(" --moe-a2a-backend deepep");
|
||||
flags.push(" --enable-nsa-prefill-context-parallel");
|
||||
flags.push(" --nsa-prefill-cp-mode round-robin-split");
|
||||
flags.push(" --chunked-prefill-size 16384");
|
||||
flags.push(" --mem-fraction-static 0.78");
|
||||
flags.push(" --max-running-requests 1024");
|
||||
if (isBig && hardware !== "h200") {
|
||||
// Blackwell big cp: extra overrides. allinone emits these AFTER _CP_FLAGS,
|
||||
// so two --mem-fraction-static appear — argparse last-wins (0.70 beats 0.78).
|
||||
flags.push(" --mem-fraction-static 0.70");
|
||||
flags.push(" --cuda-graph-max-bs 256");
|
||||
flags.push(" --max-running-requests 256");
|
||||
}
|
||||
// H200 CP gates DEEPEP_LARGE_SMS_FLAG on !multinode; Blackwell always gets it.
|
||||
if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG);
|
||||
}
|
||||
|
||||
// Optional parsers (cookbook UI extension; not in allinone — opt-in toggles only).
|
||||
if (toolcall === "enabled") flags.push(" --tool-call-parser deepseekv4");
|
||||
if (reasoningParser === "enabled") flags.push(" --reasoning-parser deepseek-v4");
|
||||
|
||||
flags.push(" --host 0.0.0.0");
|
||||
flags.push(" --port 30000");
|
||||
|
||||
// Assemble: [HW env] [recipe env] [common env] \ sglang serve \ flags...
|
||||
const envAll = [...HW_ENV, ...recipeEnv, ...COMMON_ENV];
|
||||
const envBlock = envAll.length ? envAll.join(" \\\n") + " \\\n" : "";
|
||||
const base = `${envBlock}sglang serve \\\n${flags.join(" \\\n")}`;
|
||||
const withMultinode = multinode ? prependMultiNodeNote(base, nnodes) : base;
|
||||
const verifyKey = `${hardware}|${modelSize}|${recipe}`;
|
||||
return VERIFIED_RECIPES.has(verifyKey)
|
||||
? withMultinode
|
||||
: `${BEING_VERIFIED_NOTE}\n${commentOutCommand(withMultinode)}`;
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
// buildPDDisaggCommand — mirror of allinone pd-p / pd-d for small AND big.
|
||||
//
|
||||
// _PD_SPEC[(hw, size)] → tp (and whether multinode).
|
||||
// H200-fp8 small: tp=4 single-node, ib=mlx5_0
|
||||
// H200-fp8 big: tp=16 2-node, ib=mlx5_0
|
||||
// B200 small: tp=2 single-node, ib=mlx5_7
|
||||
// B200 big: tp=8 single-node, ib=mlx5_7
|
||||
// GB300 small/big: tp=4 single-node, ib="" (uses MNNVL, no IB device)
|
||||
//
|
||||
// deepep flag only on Blackwell PD; H200 PD does NOT use deepep.
|
||||
// cap_env (SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024) only on B200 decode.
|
||||
// SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True only on GB300.
|
||||
// --dist-init-addr for disagg wiring only on non-GB300.
|
||||
// --max-running-requests 256 only on decode (PD decode can't retract).
|
||||
// No flashinfer_mxfp4 / autotune-fix / MTP / mem-fraction-static on PD (allinone omits).
|
||||
// ============================================================================
|
||||
const buildPDDisaggCommand = (hardware, modelSize) => {
|
||||
const specKey = `${hardware}|${modelSize}`;
|
||||
const { tp: pdTp, multinode, nnodes } = PD_TP_SPEC[specKey];
|
||||
const slug = HW_SIZE_SPEC[specKey].slug;
|
||||
const ibDevice = { h200: "mlx5_0", b200: "mlx5_7", gb300: "" }[hardware];
|
||||
const isGB300 = hardware === "gb300";
|
||||
const isBlackwell = hardware === "b200" || isGB300;
|
||||
|
||||
const HW_ENV = {
|
||||
h200: ["SGLANG_DSV4_FP4_EXPERTS=0"],
|
||||
b200: [],
|
||||
gb300: [],
|
||||
}[hardware];
|
||||
// Whitelist #5: only SGLANG_MOONCAKE_CUSTOM_MEM_POOL kept; MC_FORCE_MNNVL /
|
||||
// NCCL_MNNVL_ENABLE / NCCL_CUMEM_ENABLE stripped (personal-cluster topology).
|
||||
const MNNVL_ENV = isGB300 ? ["SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True"] : [];
|
||||
const COMMON_ENV = ["SGLANG_JIT_DEEPGEMM_PRECOMPILE=0"];
|
||||
|
||||
const buildRole = (mode, port, distPort) => {
|
||||
const roleEnv = [];
|
||||
if (hardware === "b200" && mode === "decode") {
|
||||
roleEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024");
|
||||
}
|
||||
const envAll = [...HW_ENV, ...roleEnv, ...MNNVL_ENV, ...COMMON_ENV];
|
||||
const envBlock = envAll.length ? envAll.join(" \\\n") + " \\\n" : "";
|
||||
|
||||
const flags = [];
|
||||
flags.push(" --trust-remote-code");
|
||||
flags.push(` --model-path ${slug}`);
|
||||
flags.push(` --tp ${pdTp}`);
|
||||
flags.push(` --dp ${pdTp}`);
|
||||
flags.push(" --enable-dp-attention");
|
||||
if (multinode) flags.push(...multiNodeFlags(nnodes));
|
||||
if (isBlackwell) flags.push(" --moe-a2a-backend deepep");
|
||||
flags.push(` --disaggregation-mode ${mode}`);
|
||||
flags.push(" --disaggregation-transfer-backend mooncake");
|
||||
if (ibDevice) flags.push(` --disaggregation-ib-device ${ibDevice}`);
|
||||
if (!isGB300) flags.push(` --dist-init-addr 127.0.0.1:${distPort}`);
|
||||
if (mode === "decode") flags.push(" --max-running-requests 256");
|
||||
flags.push(" --host 0.0.0.0");
|
||||
flags.push(` --port ${port}`);
|
||||
|
||||
return `${envBlock}sglang serve \\\n${flags.join(" \\\n")}`;
|
||||
};
|
||||
|
||||
const prefillHeader = multinode
|
||||
? `# --- Prefill role (port 30000) — multi-node, run on each of ${nnodes} nodes ---`
|
||||
: "# --- Prefill role (port 30000) ---";
|
||||
const decodeHeader = multinode
|
||||
? `# --- Decode role (port 30001) — multi-node, run on each of ${nnodes} nodes ---`
|
||||
: "# --- Decode role (port 30001) ---";
|
||||
|
||||
const prefill = `${prefillHeader}\n${buildRole("prefill", 30000, 30335)}`;
|
||||
const decode = `${decodeHeader}\n${buildRole("decode", 30001, 30435)}`;
|
||||
const router = `# --- Router (port 8000) ---
|
||||
python3 -m sglang_router.launch_router \\
|
||||
--pd-disaggregation \\
|
||||
--prefill http://127.0.0.1:30000 \\
|
||||
--decode http://127.0.0.1:30001 \\
|
||||
--host 0.0.0.0 --port 8000 \\
|
||||
--disable-circuit-breaker \\
|
||||
--health-check-interval-secs 999999`;
|
||||
|
||||
const full = `${prefill}\n\n${decode}\n\n${router}`;
|
||||
const verifyKey = `${hardware}|${modelSize}|pd-disagg`;
|
||||
return VERIFIED_RECIPES.has(verifyKey)
|
||||
? full
|
||||
: `${BEING_VERIFIED_NOTE}\n${commentOutCommand(full)}`;
|
||||
};
|
||||
|
||||
// ---- styles ----
|
||||
const containerStyle = { maxWidth: "900px", margin: "0 auto", display: "flex", flexDirection: "column", gap: "4px" };
|
||||
const cardStyle = {
|
||||
padding: "8px 12px",
|
||||
border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`,
|
||||
borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`,
|
||||
borderRadius: "4px",
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "12px",
|
||||
background: isDark ? "#1f2937" : "#fff",
|
||||
};
|
||||
const titleStyle = { fontSize: "13px", fontWeight: "600", minWidth: "140px", flexShrink: 0, color: isDark ? "#e5e7eb" : "inherit" };
|
||||
const itemsStyle = { display: "flex", rowGap: "2px", columnGap: "6px", flexWrap: "wrap", alignItems: "center", flex: 1 };
|
||||
const labelBaseStyle = {
|
||||
padding: "4px 10px",
|
||||
border: `1px solid ${isDark ? "#9ca3af" : "#d1d5db"}`,
|
||||
borderRadius: "3px",
|
||||
cursor: "pointer",
|
||||
display: "inline-flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
fontWeight: "500",
|
||||
fontSize: "13px",
|
||||
transition: "all 0.2s",
|
||||
userSelect: "none",
|
||||
minWidth: "45px",
|
||||
textAlign: "center",
|
||||
flex: 1,
|
||||
background: isDark ? "#374151" : "#fff",
|
||||
color: isDark ? "#e5e7eb" : "inherit",
|
||||
};
|
||||
const checkedStyle = { background: "#D45D44", color: "white", borderColor: "#D45D44" };
|
||||
const disabledStyle = { cursor: "not-allowed", opacity: 0.4 };
|
||||
const subtitleStyle = { display: "block", fontSize: "9px", marginTop: "1px", lineHeight: "1.1", opacity: 0.7 };
|
||||
const commandDisplayStyle = {
|
||||
flex: 1,
|
||||
padding: "12px 16px",
|
||||
background: isDark ? "#111827" : "#f5f5f5",
|
||||
borderRadius: "6px",
|
||||
fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace",
|
||||
fontSize: "12px",
|
||||
lineHeight: "1.5",
|
||||
color: isDark ? "#e5e7eb" : "#374151",
|
||||
whiteSpace: "pre-wrap",
|
||||
overflowX: "auto",
|
||||
margin: 0,
|
||||
border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`,
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={containerStyle} className="not-prose">
|
||||
{Object.entries(options).map(([key, option]) => {
|
||||
const items = resolveItems(option);
|
||||
return (
|
||||
<div key={key} style={cardStyle}>
|
||||
<div style={titleStyle}>{option.title}</div>
|
||||
<div style={itemsStyle}>
|
||||
{items.map((item) => {
|
||||
const isChecked = values[option.name] === item.id;
|
||||
const isDisabled = !!item.disabled;
|
||||
return (
|
||||
<label
|
||||
key={item.id}
|
||||
style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}
|
||||
title={item.disabledReason || ""}
|
||||
>
|
||||
<input
|
||||
type="radio"
|
||||
name={option.name}
|
||||
value={item.id}
|
||||
checked={isChecked}
|
||||
disabled={isDisabled}
|
||||
onChange={() => !isDisabled && handleRadioChange(option.name, item.id)}
|
||||
style={{ display: "none" }}
|
||||
/>
|
||||
{item.label}
|
||||
{item.subtitle && (
|
||||
<small style={{ ...subtitleStyle, color: isChecked ? "rgba(255,255,255,0.85)" : "inherit" }}>
|
||||
{item.subtitle}
|
||||
</small>
|
||||
)}
|
||||
</label>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
<div style={cardStyle}>
|
||||
<div style={titleStyle}>Run this Command:</div>
|
||||
<pre style={commandDisplayStyle}>{generateCommand()}</pre>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
Reference in New Issue
Block a user