minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

Update to the cookbook for XPU-supported models (#33649)

Browse Source 
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
This commit is contained in:
Ziru
2026-09-20 20:26:09 -07:00
committed by GitHub
co-authored by Niu Ziru
parent 62ba964848
commit 1da8ac10e1
19 changed files with 253 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx
+3 -3
View File
@@ -30,15 +30,15 @@ For more details, please refer to the [official DeepSeek-OCR-2 repository](https
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs.
The DeepSeek-OCR-2 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, and Intel Arc Pro B-Series GPUs, as well as Intel Xeon CPUs.
<DeepSeekOCR2Deployment />
docs/cookbook/autoregressive/Google/Gemma4.mdx
+2
View File
@@ -97,6 +97,8 @@ For other installation methods, please refer to the [official SGLang installatio
### 3.1 Basic Configuration
The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant.
<Gemma4Deployment />
docs/cookbook/autoregressive/Meta/Llama3.1.mdx
+1 -3
View File
@@ -22,11 +22,9 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.
### 3.1 Basic Configuration
docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx
+2 -4
View File
@@ -24,15 +24,13 @@ For more details, please refer to the [official Llama models repository](https:/
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs.
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X), Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)) and Intel Xeon CPUs.
### 3.1 Interactive Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU, Intel Arc Pro B-Series GPUs or Intel Xeon CPUs setup.
import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
docs/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx
+2
View File
@@ -31,6 +31,8 @@ This section provides a progressive guide from quick deployment to performance t
### 3.1 Basic Configuration
The Nemotron3-Nano series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)).
**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
<Nemotron3NanoDeployment />
docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+1 -3
View File
@@ -111,11 +111,9 @@ docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.
### 3.1 Basic Configuration
docs/cookbook/autoregressive/Qwen/Qwen3.mdx
+1 -3
View File
@@ -26,15 +26,13 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
docs/cookbook/diffusion/FLUX/FLUX.mdx
+1 -1
View File
@@ -34,7 +34,7 @@ This section provides deployment configurations optimized for different hardware
FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2/A3 Series NPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs, Ascend A2, A3 NPUs Series NPUs and Intel Arc B-series graphics(codename: BMG (Battlemage)).
<FluxDeployment />
docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ This section provides deployment configurations optimized for different hardware
Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving Z-Image-Turbo on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs, Ascend A2, A3 NPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)).
<ZImageTurboDeployment />
docs/docs/hardware-platforms/xpu.mdx
+16 -34
View File
@@ -7,39 +7,8 @@ The document addresses how to set up the [SGLang](https://github.com/sgl-project
Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [
Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
## Optimized Model List
A list of LLMs have been optimized on Intel GPU, and more are on the way:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "50%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen2.5-1.5B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)</td>
</tr>
</tbody>
</table>
**Note:** The model identifiers listed in the table above
have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).
A number of popular LLMs are optimized and run efficiently on XPU, including the most notable open-source models like Llama series, Qwen series, and Diffusion model series like FLUX and Z-Image.
Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found.
Quantized MoE models are covered separately in
[MXFP4 MoE Quantization](#mxfp4-moe-quantization) below.
@@ -85,6 +54,20 @@ pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu
### Install Using Docker
It is recommended to use Docker for setting up the SGLang environment.
#### Pull from Docker Hub
Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository.
The [XPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu) end with `xpu` suffix.
The image pulling command is like:
```bash Command
docker pull lmsysorg/sglang:v0.5.20-xpu
```
#### Build from Dockerfile
[The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
@@ -121,7 +104,6 @@ Example command to launch SGLang serving:
sglang serve \
--model-path <MODEL_ID_OR_PATH> \
--trust-remote-code \
--disable-overlap-schedule \
--device xpu \
--host 0.0.0.0 \
--tp 2 \ # using multi GPUs
docs/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx
+12 -8
View File
@@ -10,6 +10,7 @@ export const DeepSeekOCR2Deployment = () => {
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
]
},
quantization: {
@@ -25,8 +26,8 @@ export const DeepSeekOCR2Deployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' }
]
},
};
@@ -42,6 +43,8 @@ export const DeepSeekOCR2Deployment = () => {
cmd += ` --model-path ${modelPath}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule \\\n --trust-remote-code`;
} else if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
}
cmd += ` \\\n --enable-multimodal`;
@@ -272,9 +275,8 @@ export const DeepSeekOCR2Deployment = () => {
) : option.type === 'checkbox' ? (
(option.items || []).map((item) => {
const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled =
item.required ||
(typeof item.disabledWhen === 'function' && item.disabledWhen(values));
const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
const isDisabled = item.required || dynDisabled;
return (
<label
key={item.id}
@@ -289,9 +291,11 @@ export const DeepSeekOCR2Deployment = () => {
type="checkbox"
checked={isChecked}
disabled={isDisabled}
onChange={(event) =>
handleCheckboxChange(option.name, item.id, event.target.checked)
}
onChange={(event) => {
if (!dynDisabled) {
handleCheckboxChange(option.name, item.id, event.target.checked);
}
}}
style={{ display: 'none' }}
/>
{item.label}
docs/src/snippets/autoregressive/gemma4-deployment.jsx
+45 -2
View File
@@ -30,6 +30,7 @@ export const Gemma4Deployment = () => {
{ id: 'b200', label: 'B200', default: false },
{ id: 'b300', label: 'B300', default: false },
{ id: 'mi300x', label: 'MI300X', default: false, disabled: !showMI300X },
{ id: 'arc_b', label: 'BMG', default: false },
];
}
},
@@ -88,6 +89,10 @@ export const Gemma4Deployment = () => {
'31b': { tp: 1, mem: 0.80 },
'26b-a4b': { tp: 1, mem: 0.80 },
},
arc_b: {
'31b': { tp: 4, mem: 0.80 },
'26b-a4b': { tp: 4, mem: 0.75 },
},
};
const generateCommand = (values) => {
@@ -141,6 +146,10 @@ export const Gemma4Deployment = () => {
cmd += ` \\\n --attention-backend triton`;
}
if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
}
cmd += ` \\\n --mem-fraction-static ${mem}`;
cmd += ` \\\n --host 0.0.0.0 --port 30000`;
@@ -205,7 +214,27 @@ export const Gemma4Deployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value }));
setValues((prev) => {
if (prev.hardware === 'arc_b' && optionName === 'modelSize' && !['31b', '26b-a4b'].includes(value)) {
return prev;
}
if (prev.hardware === 'arc_b' && optionName === 'checkpoint' && value !== 'standard') {
return prev;
}
if (prev.hardware === 'arc_b' && optionName === 'speculative' && value !== 'disabled') {
return prev;
}
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b') {
if (!['31b', '26b-a4b'].includes(next.modelSize)) {
next.modelSize = '31b';
}
next.checkpoint = 'standard';
next.speculative = 'disabled';
}
return next;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -379,7 +408,21 @@ export const Gemma4Deployment = () => {
) : (
items.map((item) => {
const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled);
const isArcBModelLocked =
values.hardware === 'arc_b' &&
option.name === 'modelSize' &&
!['31b', '26b-a4b'].includes(item.id);
const isArcBCheckpointLocked =
values.hardware === 'arc_b' &&
option.name === 'checkpoint' &&
item.id !== 'standard';
const isArcBSpeculativeLocked =
values.hardware === 'arc_b' &&
option.name === 'speculative' &&
item.id !== 'disabled';
const isDisabled = Boolean(
item.disabled || isArcBModelLocked || isArcBCheckpointLocked || isArcBSpeculativeLocked
);
return (
<label
key={item.id}
docs/src/snippets/autoregressive/llama31-deployment.jsx
+25 -6
View File
@@ -11,7 +11,8 @@ export const Llama31Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
{ id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
]
},
modelsize: {
@@ -65,21 +66,28 @@ export const Llama31Deployment = () => {
...options.modelsize,
items: options.modelsize.items.map(item => ({
...item,
disabled: values.hardware === 'xeon' && item.id !== '8b'
disabled: (values.hardware === 'xeon' || values.hardware === 'arc_b') && item.id !== '8b'
}))
},
category: {
...options.category,
items: options.category.items.map(item => ({
...item,
disabled: values.hardware === 'arc_b' && item.id !== 'instruct'
}))
},
quantization: {
...options.quantization,
items: options.quantization.items.map(item => ({
...item,
disabled: values.hardware === 'xeon' && item.id === 'fp8'
disabled: (values.hardware === 'xeon' && item.id === 'fp8') || (values.hardware === 'arc_b' && item.id !== 'bf16')
}))
},
optimization: {
...options.optimization,
items: options.optimization.items.map(item => ({
...item,
disabled: values.hardware === 'xeon' && item.id !== 'basic'
disabled: (values.hardware === 'xeon' || values.hardware === 'arc_b') && item.id !== 'basic'
}))
}
};
@@ -122,6 +130,12 @@ export const Llama31Deployment = () => {
next.quantization = 'bf16';
next.optimization = 'basic';
}
if (optionName === 'hardware' && value === 'arc_b') {
next.modelsize = '8b';
next.category = 'instruct';
next.quantization = 'bf16';
next.optimization = 'basic';
}
return next;
});
};
@@ -131,6 +145,7 @@ export const Llama31Deployment = () => {
const { hardware, optimization, modelsize, category, toolcall, quantization } = values;
const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
const isXeon = hardware === 'xeon';
const effectiveModelSize = isXeon ? '8b' : modelsize;
@@ -145,7 +160,7 @@ export const Llama31Deployment = () => {
// Determine model path
let modelPath;
if (quantization === 'fp8' && category === 'instruct' && !isXeon) {
if (quantization === 'fp8' && category === 'instruct' && !isXeon && hardware !== 'arc_b') {
if (effectiveModelSize === '405b') {
// Meta official FP8 for 405B
modelPath = `meta-llama/Llama-3.1-${sizeToken}${categorySuffix}-FP8`;
@@ -186,6 +201,8 @@ export const Llama31Deployment = () => {
} else if (isXeon) {
// Intel Xeon CPU TP configuration
tpSize = 3;
} else if (hardware === 'arc_b') {
tpSize = 1;
} else {
// NVIDIA GPU TP configuration
if (effectiveModelSize === '405b') {
@@ -202,6 +219,8 @@ export const Llama31Deployment = () => {
if (isXeon) {
args.push(`--device cpu`);
args.push(`--disable-overlap-schedule`);
} else if (hardware === 'arc_b') {
args.push(`--device xpu`);
}
if (tpSize) {
@@ -214,7 +233,7 @@ export const Llama31Deployment = () => {
}
// NVIDIA-specific optimizations
if (!isAMD && !isXeon) {
if (!isAMD && !isXeon && hardware !== 'arc_b') {
if (optimization === 'throughput') {
args.push(`--enable-dp-attention`);
args.push(`--mem-fraction-static 0.85`);
docs/src/snippets/autoregressive/llama33-70b-deployment.jsx
+10 -3
View File
@@ -8,7 +8,8 @@ export const Llama33Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: true },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
{ id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false }
]
},
quantization: {
@@ -35,7 +36,7 @@ export const Llama33Deployment = () => {
...options.quantization,
items: options.quantization.items.map(item => ({
...item,
disabled: values.hardware === 'xeon' && item.id === 'fp8'
disabled: (values.hardware === 'xeon' && item.id === 'fp8') || (values.hardware === 'arc_b' && item.id !== 'bf16')
}))
}
});
@@ -74,6 +75,9 @@ export const Llama33Deployment = () => {
if (optionName === 'hardware' && value === 'xeon') {
next.quantization = 'bf16';
}
if (optionName === 'hardware' && value === 'arc_b') {
next.quantization = 'bf16';
}
return next;
});
};
@@ -83,7 +87,7 @@ export const Llama33Deployment = () => {
const { hardware, quantization, toolcall } = values;
// Select model based on quantization
const modelPath = quantization === 'fp8' && hardware !== 'xeon'
const modelPath = quantization === 'fp8' && hardware !== 'xeon' && hardware !== 'arc_b'
? 'amd/Llama-3.3-70B-Instruct-FP8-KV'
: 'meta-llama/Llama-3.3-70B-Instruct';
@@ -94,6 +98,9 @@ export const Llama33Deployment = () => {
cmd += ` --device cpu \\\n`;
cmd += ` --disable-overlap-schedule \\\n`;
cmd += ` --tp 6`;
} else if (hardware === 'arc_b') {
cmd += ` --device xpu \\\n`;
cmd += ` --tp 8`;
} else {
cmd += ` --tp 1`;
}
docs/src/snippets/autoregressive/nemotron3-nano-deployment.jsx
+23 -3
View File
@@ -8,7 +8,8 @@ export const Nemotron3NanoDeployment = () => {
items: [
{ id: 'h200', label: 'H200', default: false },
{ id: 'b200', label: 'B200', default: true },
{ id: 'b300', label: 'B300', default: false }
{ id: 'b300', label: 'B300', default: false },
{ id: 'arc_b', label: 'BMG', default: false }
]
},
modelVariant: {
@@ -76,6 +77,10 @@ export const Nemotron3NanoDeployment = () => {
cmd += ` --attention-backend flashinfer \\\n`;
}
if (hardware === 'arc_b') {
cmd += ` --device xpu \\\n`;
}
// Add thinking parser and tool call parser if enabled
for (const [key, option] of Object.entries(options)) {
if (option.commandRule) {
@@ -153,7 +158,18 @@ export const Nemotron3NanoDeployment = () => {
}, []);
const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value }));
setValues((prev) => {
if (prev.hardware === 'arc_b' && optionName === 'modelVariant' && value !== 'bf16') {
return prev;
}
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b') {
next.modelVariant = 'bf16';
next.tp = '4';
}
return next;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -327,7 +343,11 @@ export const Nemotron3NanoDeployment = () => {
) : (
items.map((item) => {
const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled);
const isArcBModelLocked =
values.hardware === 'arc_b' &&
option.name === 'modelVariant' &&
item.id !== 'bf16';
const isDisabled = Boolean(item.disabled || isArcBModelLocked);
return (
<label
key={item.id}
docs/src/snippets/autoregressive/qwen3-deployment.jsx
+39 -8
View File
@@ -23,7 +23,8 @@ export const Qwen3Deployment = () => {
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
xeon: { tp: 3, ep: 0, bf16: true, fp8: true },
arc_b: { tp: 4, ep: 0, bf16: true, fp8: true },
},
'32b': {
baseName: '32B',
@@ -35,7 +36,8 @@ export const Qwen3Deployment = () => {
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 6, ep: 0, bf16: true, fp8: true }
xeon: { tp: 6, ep: 0, bf16: true, fp8: true },
arc_b: { tp: 4, ep: 0, bf16: true, fp8: true }
},
'14b': {
baseName: '14B',
@@ -112,7 +114,8 @@ export const Qwen3Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
{ id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
]
},
modelsize: {
@@ -169,8 +172,26 @@ export const Qwen3Deployment = () => {
const options = { ...baseOptions };
const currentModelConfig = modelConfigs[values.modelsize];
if (values.hardware === 'arc_b') {
options.quantization = {
...baseOptions.quantization,
items: baseOptions.quantization.items.map(item => ({
...item,
disabled: item.id !== 'bf16'
}))
};
options.modelsize = {
...baseOptions.modelsize,
items: baseOptions.modelsize.items.map(item => ({
...item,
disabled: item.id !== '30b' && item.id !== '32b'
}))
};
}
// If model doesn't have thinking variants, disable non-base category options
if (currentModelConfig && !currentModelConfig.hasThinkingVariants) {
if (values.hardware === 'arc_b' || (currentModelConfig && !currentModelConfig.hasThinkingVariants)) {
options.category = {
...baseOptions.category,
items: baseOptions.category.items.map(item => ({
@@ -220,6 +241,14 @@ export const Qwen3Deployment = () => {
setValues(prev => {
const newValues = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b') {
newValues.quantization = 'bf16';
if (newValues.modelsize !== '30b' && newValues.modelsize !== '32b') {
newValues.modelsize = '32b';
}
newValues.category = 'base';
}
// Auto-switch to 'base' category for models without thinking variants
if (optionName === 'modelsize') {
const modelConfig = modelConfigs[value];
@@ -242,10 +271,10 @@ export const Qwen3Deployment = () => {
// Generate command
const generateCommand = () => {
const { hardware, modelsize, quantization, category, reasoningParser, toolcall } = values;
const displayOptions = getDisplayOptions(values);
const effectiveQuantization = hardware === 'arc_b' ? 'bf16' : quantization;
// Special error handling
const commandKey = `${hardware}-${modelsize}-${quantization}-${category}`;
const commandKey = `${hardware}-${modelsize}-${effectiveQuantization}-${category}`;
if (commandKey === 'h100-235b-bf16-instruct' || commandKey === 'h100-235b-bf16-thinking') {
return '# Error: Model is too large, cannot fit into 8*H100\n# Please use H200 (141GB) or select FP8 quantization';
}
@@ -260,7 +289,7 @@ export const Qwen3Deployment = () => {
return `# Error: Unknown hardware platform: ${hardware}`;
}
const quantSuffix = quantization === 'fp8' ? '-FP8' : '';
const quantSuffix = effectiveQuantization === 'fp8' ? '-FP8' : '';
// Build model name based on model category
let modelName;
@@ -281,6 +310,8 @@ export const Qwen3Deployment = () => {
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
} else if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
}
if (hwConfig.tp > 1) {
@@ -288,7 +319,7 @@ export const Qwen3Deployment = () => {
}
let ep = hwConfig.ep;
if (quantization === 'fp8' && hwConfig.tp === 8) {
if (effectiveQuantization === 'fp8' && hwConfig.tp === 8) {
ep = 2;
}
docs/src/snippets/autoregressive/qwen35-deployment.jsx
+35 -13
View File
@@ -65,7 +65,8 @@ export const Qwen35Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false, disabled: isNvfp4 },
{ id: 'mi325x', label: 'MI325X', default: false, disabled: isNvfp4 },
{ id: 'mi355x', label: 'MI355X', default: false, disabled: false },
{ id: 'xeon', label: 'XEON', default: false, disabled: isNvfp4 }
{ id: 'xeon', label: 'XEON', default: false, disabled: isNvfp4 },
{ id: 'arc_b', label: 'BMG', default: false, disabled: isNvfp4 }
];
}
},
@@ -76,11 +77,12 @@ export const Qwen35Deployment = () => {
const hasFp8 = FP8_MODELS.has(values.model);
const hasFp4 = values.model === '397b';
const isXeon = values.hardware === 'xeon';
const isArcB = values.hardware === 'arc_b';
return [
{ id: 'bf16', label: 'BF16', default: !hasFp8 || isXeon },
{ id: 'fp8', label: 'FP8', default: hasFp8 && !isXeon, disabled: !hasFp8,
{ id: 'bf16', label: 'BF16', default: !hasFp8 || isXeon || isArcB },
{ id: 'fp8', label: 'FP8', default: hasFp8 && !isXeon && !isArcB, disabled: !hasFp8 || isArcB,
disabledReason: 'No FP8 variant available for this model' },
{ id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4 || isXeon,
{ id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4 || isXeon || isArcB,
disabledReason: isXeon ? 'FP4 is not supported on Xeon' : 'FP4 is only available for Qwen3.5-397B-A17B' }
];
}
@@ -104,7 +106,7 @@ export const Qwen35Deployment = () => {
speculative: {
name: 'speculative',
title: 'Speculative Decoding (MTP)',
condition: (values) => values.hardware !== 'xeon',
condition: (values) => values.hardware !== 'xeon' && values.hardware !== 'arc_b',
items: [
{ id: 'disabled', label: 'Disabled', default: false },
{ id: 'enabled', label: 'Enabled', default: true }
@@ -124,7 +126,7 @@ export const Qwen35Deployment = () => {
mambaCache: {
name: 'mambaCache',
title: 'Mamba Radix Cache',
condition: (values) => MOE_MODELS.has(values.model) && values.hardware !== 'xeon',
condition: (values) => MOE_MODELS.has(values.model) && values.hardware !== 'xeon' && values.hardware !== 'arc_b',
getDynamicItems: (currentValues) => {
const amdGpus = ['mi300x', 'mi325x', 'mi355x'];
const isAmdGpu = amdGpus.includes(currentValues.hardware);
@@ -184,7 +186,8 @@ export const Qwen35Deployment = () => {
mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } }
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } },
arc_b: { bf16: { tp: 4, mem: 0.8 } }
},
'27b': {
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
@@ -204,7 +207,8 @@ export const Qwen35Deployment = () => {
mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
xeon: { bf16: { tp: 3 } },
arc_b: { bf16: { tp: 1, mem: 0.8 } }
},
'4b': {
h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -214,7 +218,8 @@ export const Qwen35Deployment = () => {
mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
xeon: { bf16: { tp: 3 } },
arc_b: { bf16: { tp: 1, mem: 0.8 } }
},
'2b': {
h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -286,7 +291,17 @@ export const Qwen35Deployment = () => {
}, [values.hardware, values.model]);
const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value }));
setValues(prev => {
if (prev.hardware === 'arc_b' && optionName === 'model' && !['35b', '9b', '4b'].includes(value)) {
return prev;
}
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b' && !['35b', '9b', '4b'].includes(next.model)) {
next.model = '35b';
}
return next;
});
};
// Multi-node flag template — mirrors DeepSeek-V4 cookbook's multiNodeFlags.
@@ -354,6 +369,9 @@ export const Qwen35Deployment = () => {
let cmd = `sglang serve --model-path ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
} else if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
cmd += ` \\\n --linear-attn-backend intel_xpu`;
}
if (tpValue > 1) {
cmd += ` \\\n --tp ${tpValue}`;
@@ -378,7 +396,7 @@ export const Qwen35Deployment = () => {
// would emit a spurious --mamba-radix-cache-strategy extra_buffer. The UI
// radio is hidden for dense models, so users can't manually correct it.
// MoE keeps the old behavior — the UI radio is the recovery path there.
const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon'];
const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon', 'arc_b'];
const actualMambaCache = mamba_v1_dev.includes(hardware)
? 'v1'
: (speculative === 'enabled' ? 'v2' : (MOE_MODELS.has(model) ? mambaCache : 'v1'));
@@ -427,7 +445,7 @@ export const Qwen35Deployment = () => {
// benchmark only enables this for TP>=8). AMD MI GPUs use the AITER allreduce
// fusion flag instead, handled in the AMD backend block below.
const amdGpu = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
if (quantization !== 'fp4' && hardware !== 'xeon' && !amdGpu) {
if (quantization !== 'fp4' && hardware !== 'xeon' && hardware !== 'arc_b' && !amdGpu) {
cmd += ` \\\n --enable-flashinfer-allreduce-fusion`;
}
@@ -580,7 +598,11 @@ export const Qwen35Deployment = () => {
<div style={itemsStyle}>
{items.map(item => {
const isChecked = values[option.name] === item.id;
const isDisabled = !!item.disabled;
const isArcBModelLocked =
values.hardware === 'arc_b' &&
option.name === 'model' &&
!['35b', '9b', '4b'].includes(item.id);
const isDisabled = !!item.disabled || isArcBModelLocked;
return (
<label
key={item.id}
docs/src/snippets/diffusion/flux-deployment.jsx
+27 -3
View File
@@ -15,7 +15,8 @@ export const FluxDeployment = () => {
{ id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi300x', label: 'MI300X', default: false },
{ id: 'a2', label: 'A2 Series', default: false },
{ id: 'a3', label: 'A3 Series', default: false }
{ id: 'a3', label: 'A3 Series', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
]
},
version: {
@@ -58,6 +59,15 @@ sglang serve \\
--num-gpus 2`;
}
if (hardware === 'arc_b') {
return `sglang serve \\
--model-path ${config.repoId} \\
--num-gpus 4 \\
--tp-size 4 \\
--component-residency dit=resident,text_encoder=layerwise-offload \\
--dit-cpu-offload False`;
}
return `sglang serve \\
--model-path ${config.repoId} \\
--ulysses-degree=1 \\
@@ -145,7 +155,17 @@ sglang serve \\
}, [values.hardware]);
const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value }));
setValues((prev) => {
if (prev.hardware === 'arc_b' && optionName === 'version' && value === 'flux1-dev') {
return prev;
}
const nextValues = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b' && nextValues.version === 'flux1-dev') {
nextValues.version = 'flux2-dev';
}
return nextValues;
});
};
const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -322,7 +342,11 @@ sglang serve \\
) : (
items.map((item) => {
const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled);
const isArcBVersionLocked =
values.hardware === 'arc_b' &&
option.name === 'version' &&
item.id === 'flux1-dev';
const isDisabled = Boolean(item.disabled || isArcBVersionLocked);
return (
<label
docs/src/snippets/diffusion/zimage-turbo-deployment.jsx
+7 -1
View File
@@ -14,7 +14,8 @@ export const ZImageTurboDeployment = () => {
{ id: 'h200', label: 'H200', default: false },
{ id: 'h100', label: 'H100', default: false },
{ id: 'a2', label: 'A2 Series', default: false },
{ id: 'a3', label: 'A3 Series', default: false }
{ id: 'a3', label: 'A3 Series', default: false },
{ id: 'arc_b', label: 'BMG', default: false }
]
}
},
@@ -37,6 +38,11 @@ sglang serve \\
--num-gpus 2`;
}
if (hardware === 'arc_b') {
return `sglang serve \\
--model-path Tongyi-MAI/Z-Image-Turbo`;
}
return `sglang serve \\
--model-path Tongyi-MAI/Z-Image-Turbo \\
--ulysses-degree=1 \\