minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Doc][CPU]Update Cookbook with Xeon support info (#27248)

Browse Source 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Zaili Wang
2026-06-05 22:39:40 -07:00
committed by GitHub
co-authored by gemini-code-assist[bot]
parent 38ae22e08c
commit caeb449cd6
26 changed files with 443 additions and 121 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx
+4 -1
View File
@@ -30,13 +30,15 @@ For more details, please refer to the [official DeepSeek-OCR-2 repository](https
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, and AMD MI300X, MI355X, and MI325X GPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs.
<DeepSeekOCR2Deployment />
@@ -47,6 +49,7 @@ This section provides deployment configurations optimized for different hardware
- **Single GPU Deployment:** DeepSeek-OCR-2 (~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving.
- **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway).
- **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`.
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings.
## 4. Model Invocation
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-OCR.mdx
+2
View File
@@ -26,6 +26,8 @@ For more details, please refer to the [official DeepSeek-OCR repository](https:/
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx
+5 -1
View File
@@ -15,7 +15,7 @@ import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deeps
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, and AMD MI300X, MI325X and MI355X GPUs
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
@@ -37,6 +37,8 @@ For more details, please refer to the [official DeepSeek-R1 repository](https://
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -97,6 +99,8 @@ python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --tru
**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx
+4
View File
@@ -22,6 +22,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
@@ -77,6 +79,8 @@ DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGL
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx
+4
View File
@@ -18,6 +18,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
@@ -75,6 +77,8 @@ DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGL
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx
+9 -1
View File
@@ -1,7 +1,7 @@
---
title: MiniMax-M2.7
metatags:
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA and AMD GPUs — model self-evolution, professional software engineering, and native agent teams."
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs — model self-evolution, professional software engineering, and native agent teams."
tag: NEW
---
@@ -26,6 +26,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
**Docker Images by Hardware Platform:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
@@ -139,6 +141,12 @@ import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-d
- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
**Hardware Requirements: Intel Xeon CPU**
- It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server.
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx
+3
View File
@@ -26,6 +26,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.
## 3. Model Deployment
@@ -46,6 +48,7 @@ This section provides a progressive guide from quick deployment to performance o
- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder.mdx
+6 -1
View File
@@ -28,9 +28,11 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X and NVIDIA B200, GB200 hardware platforms.
This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms.
### 3.1 Configuration
@@ -54,6 +56,9 @@ This section provides deployment configurations verified on AMD MI300X, MI325X,
**General:**
* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.
**Intel Xeon CPU:**
* For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx
+4
View File
@@ -32,6 +32,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -56,6 +58,8 @@ The Qwen3-Next series comes in only one size but offers different thinking modes
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/Qwen/Qwen3-VL.mdx
+5 -1
View File
@@ -26,13 +26,15 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs. The recommended launch configurations vary by hardware and model size.
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
@@ -76,6 +78,8 @@ python -m sglang.launch_server \
--enable-metrics
```
* For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+4
View File
@@ -111,6 +111,8 @@ docker pull lmsysorg/sglang:v0.5.9-rocm720-mi35x
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -225,6 +227,8 @@ This section provides deployment configurations optimized for different hardware
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
</Warning>
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
**NVIDIA:**
docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx
+4
View File
@@ -79,6 +79,8 @@ docker pull lmsysorg/sglang:latest
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -141,6 +143,8 @@ All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU a
</table>
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over:
docs_new/cookbook/autoregressive/Tencent/Hunyuan3-Preview.mdx
+4
View File
@@ -78,6 +78,8 @@ Please refer to the [official SGLang installation guide](../../../docs/get-start
The `hy3-preview` tag bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -167,6 +169,8 @@ SGLANG_ENABLE_SPEC_V2=1 sglang serve \
Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
docs_new/src/snippets/autoregressive/deepseek-ocr-deployment.jsx
+29 -9
View File
@@ -7,7 +7,8 @@ export const DeepSeekOCRDeployment = () => {
items: [
{ id: 'mi300x', label: 'MI300X', default: true },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
quantization: {
@@ -23,8 +24,8 @@ export const DeepSeekOCRDeployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false }
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
]
}
};
@@ -63,7 +64,20 @@ export const DeepSeekOCRDeployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value }));
setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter(id => {
const item = strategyItems.find(s => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -93,6 +107,9 @@ export const DeepSeekOCRDeployment = () => {
let cmd = 'python3 -m sglang.launch_server \\\n';
cmd += ` --model-path ${modelPath}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
cmd += ` \\\n --dtype float16`;
// TP strategy
@@ -110,7 +127,9 @@ export const DeepSeekOCRDeployment = () => {
cmd += ` \\\n --ep 1`;
}
cmd += ` \\\n --enable-symm-mem # Optional: improves performance, but may be unstable`;
if (hardware !== 'xeon') {
cmd += ` \\\n --enable-symm-mem # Optional: improves performance, but may be unstable`;
}
return cmd;
};
@@ -135,10 +154,11 @@ export const DeepSeekOCRDeployment = () => {
{option.type === 'checkbox' ? (
option.items.map(item => {
const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = item.required;
return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
const isDisabled = item.required || dynDisabled;
return (
<label key={item.id} title={item.disabledReason || (dynDisabled ? 'Not supported on the selected hardware' : '')} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => !dynDisabled && handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
{item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label>
docs_new/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx
+20 -3
View File
@@ -9,6 +9,7 @@ export const DeepSeekOCR2Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false },
]
},
quantization: {
@@ -24,8 +25,8 @@ export const DeepSeekOCR2Deployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false }
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
]
},
};
@@ -39,6 +40,9 @@ export const DeepSeekOCR2Deployment = () => {
let cmd = 'sglang serve \\\n';
cmd += ` --model-path ${modelPath}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule \\\n --trust-remote-code`;
}
cmd += ` \\\n --enable-multimodal`;
if (strategyArray.includes('tp')) {
@@ -120,7 +124,20 @@ export const DeepSeekOCR2Deployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value }));
setValues((prev) => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter((id) => {
const item = strategyItems.find((s) => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
docs_new/src/snippets/autoregressive/deepseek-r1-basic-deployment.jsx
+49 -7
View File
@@ -10,13 +10,15 @@ export const DeepSeekR1BasicDeployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false },
],
},
quantization: {
name: 'quantization',
title: 'Quantization',
getDynamicItems: (values) => {
const fp4Disabled = values.hardware === 'h100' || values.hardware === 'mi300x';
const isXeon = values.hardware === 'xeon';
const fp4Disabled = values.hardware === 'h100' || values.hardware === 'mi300x' || isXeon;
return [
{ id: 'fp8', label: 'FP8', default: true },
{
@@ -24,7 +26,16 @@ export const DeepSeekR1BasicDeployment = () => {
label: 'FP4',
default: false,
disabled: fp4Disabled,
disabledReason: 'H100 and MI300X only support FP8 quantization',
disabledReason: isXeon
? 'Intel Xeon CPUs do not support FP4 quantization'
: 'H100 and MI300X only support FP8 quantization',
},
{
id: 'int8',
label: 'INT8',
default: false,
disabled: !isXeon,
disabledReason: 'INT8 is only available when XEON hardware is selected',
},
];
},
@@ -35,9 +46,9 @@ export const DeepSeekR1BasicDeployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs do not support Multi-token Prediction' },
],
},
thinking: {
@@ -89,16 +100,19 @@ export const DeepSeekR1BasicDeployment = () => {
return '# Error: H100 and MI300X only support FP8 quantization';
}
const isXeon = hardware === 'xeon';
const modelPath =
quantization === 'fp4'
? 'nvidia/DeepSeek-R1-0528-FP4-v2'
: quantization === 'int8'
? 'Conexis/DeepSeek-R1-0528-Channel-INT8'
: 'deepseek-ai/DeepSeek-R1-0528';
let command = 'python3 -m sglang.launch_server \\\n';
command += ` --model-path ${modelPath}`;
if (strategyValues.includes('tp')) {
command += ' \\\n --tp 8';
command += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
}
if (strategyValues.includes('dp')) {
command += ' \\\n --dp 8 \\\n --enable-dp-attention';
@@ -115,13 +129,22 @@ export const DeepSeekR1BasicDeployment = () => {
' \\\n --speculative-num-draft-tokens 4';
}
command += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable';
if (!isXeon) {
command += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable';
}
if (hardware === 'b200' || (hardware === 'mi355x' && quantization === 'fp8')) {
command +=
' \\\n --kv-cache-dtype fp8_e4m3 # Optional: enables fp8 kv cache and fp8 attention kernels to improve performance';
}
if (isXeon) {
command += ' \\\n --device cpu \\\n --disable-overlap-schedule';
if (quantization === 'int8') {
command += ' \\\n --quantization w8a8_int8';
}
}
if (thinking === 'enabled') {
command += ' \\\n --reasoning-parser deepseek-r1';
}
@@ -170,6 +193,25 @@ export const DeepSeekR1BasicDeployment = () => {
}
}
}
if (optionName === 'hardware') {
if (next.hardware === 'xeon') {
next.quantization = 'int8';
} else if (next.quantization === 'int8') {
next.quantization = 'fp8';
}
}
const strategyItems = options.strategy.items || [];
const currentStrategy = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = currentStrategy.filter((id) => {
const item = strategyItems.find((s) => s.id === id);
if (!item) {
return false;
}
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) {
return false;
}
return true;
});
return next;
});
};
docs_new/src/snippets/autoregressive/deepseek-v3-deployment.jsx
+65 -20
View File
@@ -10,16 +10,26 @@ export const DeepSeekV3Deployment = () => {
{ id: 'b200', label: 'B200', default: true },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
quantization: {
name: 'quantization',
title: 'Quantization',
items: [
{ id: 'fp8', label: 'FP8', default: true },
{ id: 'fp4', label: 'FP4', default: false }
]
getDynamicItems: (values) => {
const isXeon = values.hardware === 'xeon';
return [
{ id: 'fp8', label: 'FP8', default: true },
{
id: 'fp4',
label: 'FP4',
default: false,
disabled: isXeon,
disabledReason: 'Intel Xeon CPUs do not support FP4 quantization'
}
];
}
},
strategy: {
name: 'strategy',
@@ -27,9 +37,9 @@ export const DeepSeekV3Deployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false }
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
]
},
thinking: {
@@ -57,8 +67,11 @@ export const DeepSeekV3Deployment = () => {
if (option.type === 'checkbox') {
initialState[key] = option.items.filter(item => item.default).map(item => item.id);
} else {
const defaultItem = option.items.find(item => item.default);
initialState[key] = defaultItem ? defaultItem.id : option.items[0].id;
const items = typeof option.getDynamicItems === 'function'
? option.getDynamicItems(initialState)
: option.items;
const defaultItem = items.find(item => item.default && !item.disabled) || items.find(item => !item.disabled);
initialState[key] = defaultItem ? defaultItem.id : items[0].id;
}
});
return initialState;
@@ -84,7 +97,30 @@ export const DeepSeekV3Deployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value }));
setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
const quantizationItems = typeof options.quantization.getDynamicItems === 'function'
? options.quantization.getDynamicItems(next)
: options.quantization.items || [];
const currentQuantization = quantizationItems.find(item => item.id === next.quantization);
if (!currentQuantization || currentQuantization.disabled) {
const fallback = quantizationItems.find(item => item.default && !item.disabled) || quantizationItems.find(item => !item.disabled);
if (fallback) {
next.quantization = fallback.id;
}
}
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter(id => {
const item = strategyItems.find(s => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -103,17 +139,18 @@ export const DeepSeekV3Deployment = () => {
const { hardware, quantization, strategy, thinking, toolcall } = values;
const strategyArray = Array.isArray(strategy) ? strategy : [];
// Validation - H100/H200/MI300X/MI325X only supports FP8
// Validation - H100/H200/MI300X/MI325X/XEON only supports FP8
if (['h100', 'h200', 'mi300x', 'mi325x'].includes(hardware) && quantization === 'fp4') {
return '# Error: This hardware only supports FP8 quantization\n# Please select FP8 quantization or use B200/MI355X hardware';
}
const modelPath = quantization === 'fp4' ? 'nvidia/DeepSeek-V3-0324-NVFP4' : 'deepseek-ai/DeepSeek-V3';
const isXeon = hardware === 'xeon';
let cmd = 'python3 -m sglang.launch_server \\\n';
cmd += ` --model-path ${modelPath}`;
if (strategyArray.includes('tp')) cmd += ' \\\n --tp 8';
if (strategyArray.includes('tp')) cmd += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
if (strategyArray.includes('dp')) cmd += ' \\\n --dp 8 \\\n --enable-dp-attention';
if (strategyArray.includes('ep')) cmd += ' \\\n --ep 8';
if (strategyArray.includes('mtp')) {
@@ -121,12 +158,18 @@ export const DeepSeekV3Deployment = () => {
cmd += ' \\\n --speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4';
}
cmd += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable';
if (!isXeon) {
cmd += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable';
}
if (hardware === 'b200') {
cmd += ' \\\n --kv-cache-dtype fp8_e4m3 # Optional: enables fp8 kv cache and fp8 attention kernels';
}
if (isXeon) {
cmd += ' \\\n --device cpu \\\n --disable-overlap-schedule';
}
if (thinking === 'enabled') cmd += ' \\\n --reasoning-parser deepseek-v3';
if (toolcall === 'enabled') cmd += ' \\\n --tool-call-parser deepseekv3 \\\n --chat-template examples/chat_template/tool_chat_template_deepseekv3.jinja';
@@ -153,21 +196,23 @@ export const DeepSeekV3Deployment = () => {
{option.type === 'checkbox' ? (
option.items.map(item => {
const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = item.required;
const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
const isDisabled = item.required || dynDisabled;
return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
<label key={item.id} title={dynDisabled ? 'Not supported on the selected hardware' : ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => !dynDisabled && handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
{item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label>
);
})
) : (
option.items.map(item => {
(option.getDynamicItems ? option.getDynamicItems(values) : option.items).map(item => {
const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled);
return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}) }}>
<input type="radio" name={option.name} value={item.id} checked={isChecked} onChange={() => handleRadioChange(option.name, item.id)} style={{ display: 'none' }} />
<label key={item.id} title={item.disabledReason || ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="radio" name={option.name} value={item.id} checked={isChecked} disabled={isDisabled} onChange={() => !isDisabled && handleRadioChange(option.name, item.id)} style={{ display: 'none' }} />
{item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label>
docs_new/src/snippets/autoregressive/deepseek-v31-deployment.jsx
+50 -15
View File
@@ -9,7 +9,8 @@ export const DeepSeekV31Deployment = () => {
{ id: 'b200', label: 'B200', default: false },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
modelname: {
@@ -17,7 +18,8 @@ export const DeepSeekV31Deployment = () => {
title: 'Model Name',
items: [
{ id: 'v31', label: 'DeepSeek-V3.1', default: true },
{ id: 'v31terminus', label: 'DeepSeek-V3.1-Terminus', default: false }
{ id: 'v31terminus', label: 'DeepSeek-V3.1-Terminus', default: false },
{ id: 'v31terminusint8', label: 'DeepSeek-V3.1-Terminus-Channel-int8', default: false, xeonOnly: true }
]
},
strategy: {
@@ -26,9 +28,9 @@ export const DeepSeekV31Deployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', default: true, required: true },
{ id: 'dp', label: 'DP attention', default: false },
{ id: 'ep', label: 'EP', default: false },
{ id: 'mtp', label: 'Multi-token Prediction', default: false }
{ id: 'dp', label: 'DP attention', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'ep', label: 'EP', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'mtp', label: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon' }
]
},
reasoningParser: {
@@ -83,7 +85,28 @@ export const DeepSeekV31Deployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value }));
setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
if (next.hardware === 'xeon') {
next.modelname = 'v31terminusint8';
} else {
const m = options.modelname.items.find(i => i.id === next.modelname);
if (m && m.xeonOnly) {
next.modelname = options.modelname.items.find(i => !i.xeonOnly && i.default)?.id || 'v31';
}
}
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter(id => {
const item = strategyItems.find(s => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -104,17 +127,26 @@ export const DeepSeekV31Deployment = () => {
// Model name mapping
const modelMap = {
'v31': 'DeepSeek-V3.1',
'v31terminus': 'DeepSeek-V3.1-Terminus'
'v31': 'deepseek-ai/DeepSeek-V3.1',
'v31terminus': 'deepseek-ai/DeepSeek-V3.1-Terminus',
'v31terminusint8': 'IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8'
};
const modelName = `deepseek-ai/${modelMap[modelname]}`;
const modelName = modelMap[modelname];
const isXeon = hardware === 'xeon';
let cmd = 'python3 -m sglang.launch_server \\\n';
cmd += ` --model-path ${modelName}`;
if (isXeon) {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
if (modelname === 'v31terminusint8') {
cmd += ` \\\n --quantization w8a8_int8`;
}
}
// TP is mandatory
cmd += ` \\\n --tp 8`;
cmd += isXeon ? ` \\\n --tp 6` : ` \\\n --tp 8`;
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`;
}
@@ -141,6 +173,7 @@ export const DeepSeekV31Deployment = () => {
cmd += ` \\\n --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja`;
}
return cmd;
};
@@ -164,10 +197,11 @@ export const DeepSeekV31Deployment = () => {
{option.type === 'checkbox' ? (
option.items.map(item => {
const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = item.required;
const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
const isDisabled = item.required || dynDisabled;
return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
<label key={item.id} title={dynDisabled ? 'Not supported on the selected hardware' : ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => !dynDisabled && handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
{item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label>
@@ -176,9 +210,10 @@ export const DeepSeekV31Deployment = () => {
) : (
option.items.map(item => {
const isChecked = values[option.name] === item.id;
const isDisabled = item.xeonOnly && values.hardware !== 'xeon';
return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}) }}>
<input type="radio" name={option.name} value={item.id} checked={isChecked} onChange={() => handleRadioChange(option.name, item.id)} style={{ display: 'none' }} />
<label key={item.id} title={isDisabled ? 'Only available when XEON hardware is selected' : undefined} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="radio" name={option.name} value={item.id} checked={isChecked} disabled={isDisabled} onChange={() => !isDisabled && handleRadioChange(option.name, item.id)} style={{ display: 'none' }} />
{item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label>
docs_new/src/snippets/autoregressive/hunyuan3-preview-deployment.jsx
+31 -10
View File
@@ -13,7 +13,8 @@ export const Hunyuan3PreviewDeployment = () => {
{ id: 'h200', label: 'H200', default: true },
{ id: 'b200', label: 'B200', default: false },
{ id: 'b300', label: 'B300', default: false },
{ id: 'gb300', label: 'GB300', default: false }
{ id: 'gb300', label: 'GB300', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
reasoning: {
@@ -35,10 +36,13 @@ export const Hunyuan3PreviewDeployment = () => {
speculative: {
name: 'speculative',
title: 'Speculative Decoding (MTP)',
items: [
{ id: 'disabled', label: 'Disabled', default: true },
{ id: 'enabled', label: 'Enabled', subtitle: 'Low Latency', default: false }
]
getDynamicItems: (values) => {
const isXeon = values && values.hardware === 'xeon';
return [
{ id: 'disabled', label: 'Disabled', default: true },
{ id: 'enabled', label: 'Enabled', subtitle: 'Low Latency', default: false, disabled: isXeon, disabledReason: isXeon ? 'Speculative decoding (MTP) is not supported on Intel Xeon CPUs' : '' }
];
}
}
};
@@ -46,7 +50,8 @@ export const Hunyuan3PreviewDeployment = () => {
h200: { tp: 8, mem: 0.9 },
b200: { tp: 8, mem: 0.9 },
b300: { tp: 4, mem: 0.9 },
gb300: { tp: 4, mem: 0.9 }
gb300: { tp: 4, mem: 0.9 },
xeon: { tp: 6 }
};
const resolveItems = (option, values) => {
@@ -82,19 +87,34 @@ export const Hunyuan3PreviewDeployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value }));
setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
for (const [key, option] of Object.entries(options)) {
if (key === 'hardware') continue;
const items = resolveItems(option, next);
const current = items.find(i => i.id === next[key]);
if (!current || current.disabled) {
const fallback = items.find(i => i.default && !i.disabled) || items.find(i => !i.disabled);
if (fallback) next[key] = fallback.id;
}
}
}
return next;
});
};
const generateCommand = () => {
const { hardware } = values;
const isBlackwell = hardware === 'b200' || hardware === 'b300' || hardware === 'gb300';
const isXeon = hardware === 'xeon';
const hwConfig = modelConfigs[hardware];
if (!hwConfig) return '# Configuration not available for the selected hardware.';
const modelName = 'tencent/Hy3-preview';
const tpValue = hwConfig.tp;
const memFraction = hwConfig.mem;
const enableSpec = values.speculative === 'enabled';
const enableSpec = values.speculative === 'enabled' && !isXeon;
let cmd = '';
if (enableSpec) cmd += 'SGLANG_ENABLE_SPEC_V2=1 ';
@@ -112,9 +132,10 @@ export const Hunyuan3PreviewDeployment = () => {
}
cmd += ' \\\n --trust-remote-code';
cmd += ` \\\n --mem-fraction-static ${memFraction}`;
if (memFraction !== undefined) cmd += ` \\\n --mem-fraction-static ${memFraction}`;
if (isBlackwell) cmd += ' \\\n --attention-backend trtllm_mha';
if (isBlackwell && !isXeon) cmd += ' \\\n --attention-backend trtllm_mha';
if (isXeon) cmd += ' \\\n --device cpu \\\n --disable-overlap-schedule';
return cmd;
};
docs_new/src/snippets/autoregressive/minimax-m27-deployment.jsx
+18 -4
View File
@@ -13,7 +13,8 @@ export const MiniMaxM27Deployment = () => {
{ id: 'h100', label: 'H100', default: false },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
gpuCount: {
@@ -23,6 +24,12 @@ export const MiniMaxM27Deployment = () => {
const hw = values.hardware;
const isAMD = hw === 'mi300x' || hw === 'mi325x' || hw === 'mi355x';
const isGB300 = hw === 'gb300';
const isXeon = hw === 'xeon';
if (isXeon) {
return [
{ id: 'tp6', label: 'TP=6', default: true, disabled: false }
];
}
const canUse2GPU = isAMD || isGB300;
return [
{ id: '2gpu', label: '2', default: canUse2GPU, disabled: !canUse2GPU },
@@ -112,6 +119,7 @@ export const MiniMaxM27Deployment = () => {
const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
const isGB300 = hardware === 'gb300';
const isXeon = hardware === 'xeon';
const canUse2GPU = isAMD || isGB300;
if (gpuCount === '2gpu' && !canUse2GPU) {
@@ -123,7 +131,11 @@ export const MiniMaxM27Deployment = () => {
let cmd = 'sglang serve \\\n';
cmd += ` --model-path ${modelName}`;
if (gpuCount === '8gpu') {
if (isXeon) {
cmd += ' \\\n --device cpu';
cmd += ' \\\n --disable-overlap-schedule';
cmd += ' \\\n --tp 6';
} else if (gpuCount === '8gpu') {
cmd += ' \\\n --tp 8';
cmd += ' \\\n --ep 8';
} else if (gpuCount === '4gpu') {
@@ -138,9 +150,11 @@ export const MiniMaxM27Deployment = () => {
if (thinking === 'enabled') cmd += ' \\\n --reasoning-parser minimax-append-think';
cmd += ' \\\n --trust-remote-code';
cmd += ' \\\n --mem-fraction-static 0.85';
if (!isXeon) {
cmd += ' \\\n --mem-fraction-static 0.85';
}
if (isAMD) {
if (!isXeon && isAMD) {
cmd += ' \\\n --kv-cache-dtype fp8_e4m3';
cmd += ' \\\n --attention-backend triton';
}
docs_new/src/snippets/autoregressive/qwen3-coder-deployment.jsx
+22 -11
View File
@@ -8,7 +8,8 @@ export const Qwen3CoderDeployment = () => {
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'b200', label: 'B200', default: false },
{ id: 'gb200', label: 'GB200', default: false }
{ id: 'gb200', label: 'GB200', default: false },
{ id: 'xeon', label: 'Xeon', default: false }
]
},
modelSize: {
@@ -22,11 +23,14 @@ export const Qwen3CoderDeployment = () => {
quantization: {
name: 'quantization',
title: 'Quantization',
items: [
{ id: 'bf16', label: 'BF16', default: true },
{ id: 'fp8', label: 'FP8', default: false },
{ id: 'nvfp4', label: 'NVFP4', default: false }
]
getDynamicItems: (values) => {
const isXeon = values.hardware === 'xeon';
return [
{ id: 'bf16', label: 'BF16', default: true },
{ id: 'fp8', label: 'FP8', default: false, disabled: false, disabledReason: '' },
{ id: 'nvfp4', label: 'NVFP4', default: false, disabled: isXeon, disabledReason: isXeon ? 'FP4 is not supported on Xeon' : '' }
];
}
},
toolcall: {
name: 'toolcall',
@@ -46,13 +50,15 @@ export const Qwen3CoderDeployment = () => {
mi325x: { tp: 8 },
mi355x: { tp: 8 },
b200: { tp: 8, ep: 8 },
gb200: { tp: 4, ep: 4 }
gb200: { tp: 4, ep: 4 },
xeon: { tp: 6 }
},
'30b': {
baseName: '30B-A3B',
mi300x: { tp: 1 },
mi325x: { tp: 1 },
mi355x: { tp: 1 }
mi355x: { tp: 1 },
xeon: { tp: 3 }
}
};
@@ -60,6 +66,7 @@ export const Qwen3CoderDeployment = () => {
const { hardware, modelSize, quantization } = values;
const isNvidia = hardware === 'b200' || hardware === 'gb200';
const isXeon = hardware === 'xeon';
const modelConfig = modelConfigs[modelSize];
const hwConfig = modelConfig[hardware];
@@ -88,12 +95,16 @@ export const Qwen3CoderDeployment = () => {
}
let cmd = '';
if (!isNvidia) {
if (!isNvidia && !isXeon) {
cmd += 'SGLANG_USE_AITER=0 ';
}
cmd += 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`;
if (isXeon) {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
// TP setting
cmd += ` \\\n --tp ${hwConfig.tp}`;
@@ -101,7 +112,7 @@ export const Qwen3CoderDeployment = () => {
const ep = hwConfig.ep || (quantization === 'nvfp4' ? 1 : null);
if (ep) {
cmd += ` \\\n --ep ${ep}`;
} else if (modelSize === '480b' && quantization === 'fp8') {
} else if (modelSize === '480b' && quantization === 'fp8' && !isXeon) {
// FP8 requires EP=2 for 480B model due to MoE dimension alignment
// moe_intermediate_size=2560, with tp=8 ep=1: 2560/8=320, 320%128!=0
// with tp=8 ep=2: 2560/4=640, 640%128=0
@@ -132,7 +143,7 @@ export const Qwen3CoderDeployment = () => {
});
// AMD-specific flags
if (!isNvidia) {
if (!isNvidia && !isXeon) {
// Context length verified on MI300X/MI325X/MI355X
cmd += ` \\\n --context-length 8192`;
docs_new/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx
+15 -2
View File
@@ -9,7 +9,8 @@ export const Qwen3CoderNextDeployment = () => {
{ id: 'b200', label: 'B200', default: false },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
quantization: {
@@ -32,6 +33,7 @@ export const Qwen3CoderNextDeployment = () => {
mambaCache: {
name: 'mambaCache',
title: 'Mamba Radix Cache',
condition: (values) => values.hardware !== 'xeon',
items: [
{ id: 'v1', label: 'V1', default: true },
{ id: 'v2', label: 'V2', default: false }
@@ -48,7 +50,8 @@ export const Qwen3CoderNextDeployment = () => {
b200: { bf16: { tp: 2 }, fp8: { tp: 1 } },
mi300x: { bf16: { tp: 2 }, fp8: { tp: 1 } },
mi325x: { bf16: { tp: 2 }, fp8: { tp: 1 } },
mi355x: { bf16: { tp: 2 }, fp8: { tp: 1 } }
mi355x: { bf16: { tp: 2 }, fp8: { tp: 1 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } }
}
};
@@ -61,12 +64,19 @@ export const Qwen3CoderNextDeployment = () => {
}
const quantConfig = hwConfig[quantization];
if (!quantConfig) {
return '# Configuration not available for the selected hardware/quantization.';
}
const quantSuffix = quantization === 'fp8' ? '-FP8' : '';
const modelName = `Qwen/${modelConfigs.default.baseName}${quantSuffix}`;
let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
// TP setting
if (quantConfig.tp > 1) {
cmd += ` \\\n --tp ${quantConfig.tp}`;
@@ -74,6 +84,9 @@ export const Qwen3CoderNextDeployment = () => {
// Apply commandRule from all options
Object.entries(options).forEach(([key, option]) => {
if (option.condition && !option.condition(values)) {
return;
}
if (option.commandRule && values[key]) {
const additionalCmd = option.commandRule(values[key], values);
if (additionalCmd) {
docs_new/src/snippets/autoregressive/qwen3-next-deployment.jsx
+13 -2
View File
@@ -9,7 +9,8 @@ export const Qwen3NextDeployment = () => {
{ id: 'h100', label: 'H100', default: false },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'Xeon', default: false }
]
},
modelsize: {
@@ -48,6 +49,7 @@ export const Qwen3NextDeployment = () => {
speculative: {
name: 'speculative',
title: 'Speculative Decoding',
condition: (values) => values.hardware !== 'xeon',
items: [
{ id: 'disabled', label: 'Disabled', default: true },
{ id: 'enabled', label: 'Enabled', default: false }
@@ -57,6 +59,7 @@ export const Qwen3NextDeployment = () => {
mambaCache: {
name: 'mambaCache',
title: 'Mamba Radix Cache',
condition: (values) => values.hardware !== 'xeon',
items: [
{ id: 'v1', label: 'V1', default: true },
{ id: 'v2', label: 'V2', default: false }
@@ -74,7 +77,8 @@ export const Qwen3NextDeployment = () => {
b200: { tp: 2, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 2, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 2, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 2, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 2, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
}
};
@@ -99,6 +103,10 @@ export const Qwen3NextDeployment = () => {
let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (hwConfig.tp > 1) {
cmd += ` \\\n --tp ${hwConfig.tp}`;
}
@@ -113,6 +121,9 @@ export const Qwen3NextDeployment = () => {
}
for (const [key, option] of Object.entries(options)) {
if (option.condition && !option.condition(values)) {
continue;
}
if (option.commandRule) {
const rule = option.commandRule(values[key]);
if (rule) {
docs_new/src/snippets/autoregressive/qwen3-vl-deployment.jsx
+19 -8
View File
@@ -10,7 +10,8 @@ export const Qwen3VLDeployment = () => {
{ id: 'h200', label: 'H200', default: false },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
]
},
modelsize: {
@@ -61,7 +62,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 8, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 8, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 8, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 8, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 8, ep: 0, bf16: true, fp8: true },
xeon: { tp: 6, ep: 0, bf16: true, fp8: true }
},
'30b': {
baseName: '30B-A3B',
@@ -71,7 +73,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
},
'32b': {
baseName: '32B',
@@ -81,7 +84,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 6, ep: 0, bf16: true, fp8: true }
},
'8b': {
baseName: '8B',
@@ -91,7 +95,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
},
'4b': {
baseName: '4B',
@@ -101,7 +106,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
},
'2b': {
baseName: '2B',
@@ -111,7 +117,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
}
};
@@ -152,6 +159,7 @@ export const Qwen3VLDeployment = () => {
const generateCommand = () => {
const { hardware, modelsize, quantization, thinking, toolcall } = values;
const commandKey = `${hardware}-${modelsize}-${quantization}-${thinking}`;
const isXeon = hardware === 'xeon';
// Special error handling
if (commandKey === 'h100-235b-bf16-instruct' || commandKey === 'h100-235b-bf16-thinking') {
@@ -174,6 +182,9 @@ export const Qwen3VLDeployment = () => {
let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`;
if (isXeon) {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (hwConfig.tp > 1) {
cmd += ` \\\n --tp ${hwConfig.tp}`;
@@ -188,7 +199,7 @@ export const Qwen3VLDeployment = () => {
cmd += ` \\\n --ep ${ep}`;
}
if (hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x') {
if (!isXeon && (hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x')) {
if (modelsize === '32b' && quantization === 'bf16') {
cmd += ` \\\n --context-length 65536`;
}
docs_new/src/snippets/autoregressive/qwen35-deployment.jsx
+33 -18
View File
@@ -64,7 +64,8 @@ export const Qwen35Deployment = () => {
{ id: 'b300', label: 'B300', default: isNvfp4, disabled: false },
{ id: 'mi300x', label: 'MI300X', default: false, disabled: isNvfp4 },
{ id: 'mi325x', label: 'MI325X', default: false, disabled: isNvfp4 },
{ id: 'mi355x', label: 'MI355X', default: false, disabled: isNvfp4 }
{ id: 'mi355x', label: 'MI355X', default: false, disabled: isNvfp4 },
{ id: 'xeon', label: 'XEON', default: false, disabled: isNvfp4 }
];
}
},
@@ -74,12 +75,13 @@ export const Qwen35Deployment = () => {
getDynamicItems: (values) => {
const hasFp8 = FP8_MODELS.has(values.model);
const hasFp4 = values.model === '397b';
const isXeon = values.hardware === 'xeon';
return [
{ id: 'bf16', label: 'BF16', default: !hasFp8 },
{ id: 'fp8', label: 'FP8', default: hasFp8, disabled: !hasFp8,
{ id: 'bf16', label: 'BF16', default: !hasFp8 || isXeon },
{ id: 'fp8', label: 'FP8', default: hasFp8 && !isXeon, disabled: !hasFp8,
disabledReason: 'No FP8 variant available for this model' },
{ id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4,
disabledReason: 'FP4 is only available for Qwen3.5-397B-A17B' }
{ id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4 || isXeon,
disabledReason: isXeon ? 'FP4 is not supported on Xeon' : 'FP4 is only available for Qwen3.5-397B-A17B' }
];
}
},
@@ -102,6 +104,7 @@ export const Qwen35Deployment = () => {
speculative: {
name: 'speculative',
title: 'Speculative Decoding (MTP)',
condition: (values) => values.hardware !== 'xeon',
items: [
{ id: 'disabled', label: 'Disabled', default: false },
{ id: 'enabled', label: 'Enabled', default: true }
@@ -110,7 +113,7 @@ export const Qwen35Deployment = () => {
mambaCache: {
name: 'mambaCache',
title: 'Mamba Radix Cache',
condition: (values) => MOE_MODELS.has(values.model),
condition: (values) => MOE_MODELS.has(values.model) && values.hardware !== 'xeon',
getDynamicItems: (currentValues) => {
const amdGpus = ['mi300x', 'mi325x', 'mi355x'];
const isAmdGpu = amdGpus.includes(currentValues.hardware);
@@ -149,7 +152,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 }, fp4: { tp: 2, mem: 0.8 } },
mi300x: { bf16: { tp: 8, mem: 0.8 }, fp8: { tp: 4, mem: 0.8 } },
mi325x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } },
mi355x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } }
mi355x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } }
},
'122b': {
h100: { bf16: { tp: 4, mem: 0.88 }, fp8: { tp: 2, mem: 0.8 } },
@@ -158,7 +162,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 2 }, fp8: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 2, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } }
},
'35b': {
h100: { bf16: { tp: 1, mem: 0.88 }, fp8: { tp: 1, mem: 0.8 } },
@@ -167,7 +172,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } }
},
'27b': {
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
@@ -176,7 +182,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } }
},
'9b': {
h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -185,7 +192,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
},
'4b': {
h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -194,7 +202,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
},
'2b': {
h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -203,7 +212,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
},
'0.8b': {
h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -212,7 +222,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } }
mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
}
};
@@ -322,12 +333,16 @@ export const Qwen35Deployment = () => {
// Initialize the base command
let cmd = `sglang serve --model-path ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (tpValue > 1) {
cmd += ` \\\n --tp ${tpValue}`;
}
if (epValue) {
cmd += ` \\\n --expert-parallel-size ${epValue}`;
}
// Multi-node wiring goes right after --tp / --expert-parallel-size so the
// distributed-init flags sit next to the parallelism flags they configure.
if (isMultinode) {
@@ -336,7 +351,7 @@ export const Qwen35Deployment = () => {
}
}
// Force Mamba V1 for AMD GPUs (V2 requires FLA backend).
// Force Mamba V1 for AMD GPUs and Xeon CPUs (V2 requires FLA backend).
// Force Mamba V2 when MTP is enabled.
// Dense models with MTP off: force V1 — values.mambaCache is not
// re-resolved on a speculative toggle (useEffect deps are hardware/model),
@@ -344,8 +359,8 @@ export const Qwen35Deployment = () => {
// would emit a spurious --mamba-scheduler-strategy extra_buffer. The UI
// radio is hidden for dense models, so users can't manually correct it.
// MoE keeps the old behavior — the UI radio is the recovery path there.
const amdGpus = ['mi300x', 'mi325x', 'mi355x'];
const actualMambaCache = amdGpus.includes(hardware)
const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon'];
const actualMambaCache = mamba_v1_dev.includes(hardware)
? 'v1'
: (speculative === 'enabled' ? 'v2' : (MOE_MODELS.has(model) ? mambaCache : 'v1'));
@@ -389,7 +404,7 @@ export const Qwen35Deployment = () => {
}
// Enable allreduce fusion for all Qwen3.5 configs (skip for FP4: benchmark only enables this for TP>=8).
if (quantization !== 'fp4') {
if (quantization !== 'fp4' && hardware !== 'xeon') {
cmd += ` \\\n --enable-flashinfer-allreduce-fusion`;
}
docs_new/src/snippets/autoregressive/qwen36-deployment.jsx
+21 -7
View File
@@ -8,6 +8,7 @@ export const Qwen36Deployment = () => {
{ id: 'h100', label: 'H100', default: true },
{ id: 'h200', label: 'H200', default: false },
{ id: 'b200', label: 'B200', default: false },
{ id: 'xeon', label: 'XEON', default: false },
],
},
modelSize: {
@@ -47,15 +48,20 @@ export const Qwen36Deployment = () => {
speculative: {
name: 'speculative',
title: 'Speculative Decoding (MTP)',
items: [
{ id: 'disabled', label: 'Disabled', default: false },
{ id: 'enabled', label: 'Enabled', default: true },
],
getDynamicItems: (values) => {
const isXeon = values.hardware === 'xeon';
return [
{ id: 'disabled', label: 'Disabled', default: isXeon },
{ id: 'enabled', label: 'Enabled', default: !isXeon, disabled: isXeon,
disabledReason: isXeon ? 'Speculative decoding is not supported on Xeon' : '' },
];
},
commandRule: (value) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null,
},
mambaCache: {
name: 'mambaCache',
title: 'Mamba Radix Cache',
condition: (values) => values.hardware !== 'xeon',
getDynamicItems: (values) => {
const mtpEnabled = values.speculative === 'enabled';
if (mtpEnabled) {
@@ -79,12 +85,14 @@ export const Qwen36Deployment = () => {
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
h200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
b200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } },
},
'27b': {
baseName: '27B',
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
h200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
b200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } },
},
};
@@ -136,7 +144,7 @@ export const Qwen36Deployment = () => {
}
return next;
});
}, [values.speculative]);
}, [values.speculative, values.hardware]);
const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value }));
@@ -159,6 +167,9 @@ export const Qwen36Deployment = () => {
}
cmd += `sglang serve --model-path ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (hwConfig.tp > 1) {
cmd += ` \\\n --tp ${hwConfig.tp}`;
}
@@ -170,6 +181,7 @@ export const Qwen36Deployment = () => {
for (const [key, option] of Object.entries(options)) {
if (key === 'quantization' || key === 'hardware' || key === 'modelSize') continue;
if (option.condition && !option.condition(values)) continue;
if (!option.commandRule) continue;
const rule = option.commandRule(adjustedValues[key]);
if (rule) {
@@ -180,8 +192,9 @@ export const Qwen36Deployment = () => {
if (hardware === 'b200') {
cmd += ` \\\n --attention-backend trtllm_mha`;
}
cmd += ` \\\n --mem-fraction-static ${hwConfig.mem}`;
if (hwConfig.mem !== undefined) {
cmd += ` \\\n --mem-fraction-static ${hwConfig.mem}`;
}
return cmd;
};
@@ -197,6 +210,7 @@ export const Qwen36Deployment = () => {
return (
<div style={containerStyle} className="not-prose">
{Object.entries(options).map(([key, option]) => {
if (typeof option.condition === 'function' && !option.condition(values)) return null;
const items = resolveItems(option, values);
return (
<div key={key} style={cardStyle}>