minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

Update to the cookbook for XPU-supported models (#33649)

Browse Source 
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
This commit is contained in:
Ziru
2026-09-20 20:26:09 -07:00
committed by GitHub
co-authored by Niu Ziru
parent 62ba964848
commit 1da8ac10e1
19 changed files with 253 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx
+3 -3
View File
@@ -30,15 +30,15 @@ For more details, please refer to the [official DeepSeek-OCR-2 repository](https
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration ### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs. The DeepSeek-OCR-2 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, and Intel Arc Pro B-Series GPUs, as well as Intel Xeon CPUs.
<DeepSeekOCR2Deployment /> <DeepSeekOCR2Deployment />
docs/cookbook/autoregressive/Google/Gemma4.mdx
+2
View File
@@ -97,6 +97,8 @@ For other installation methods, please refer to the [official SGLang installatio
### 3.1 Basic Configuration ### 3.1 Basic Configuration
The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant.
<Gemma4Deployment /> <Gemma4Deployment />
docs/cookbook/autoregressive/Meta/Llama3.1.mdx
+1 -3
View File
@@ -22,11 +22,9 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.
### 3.1 Basic Configuration ### 3.1 Basic Configuration
docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx
+2 -4
View File
@@ -24,15 +24,13 @@ For more details, please refer to the [official Llama models repository](https:/
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs. This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X), Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)) and Intel Xeon CPUs.
### 3.1 Interactive Configuration ### 3.1 Interactive Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU, Intel Arc Pro B-Series GPUs or Intel Xeon CPUs setup.
import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx"; import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
docs/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx
+2
View File
@@ -31,6 +31,8 @@ This section provides a progressive guide from quick deployment to performance t
### 3.1 Basic Configuration ### 3.1 Basic Configuration
The Nemotron3-Nano series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)).
**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command. **Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
<Nemotron3NanoDeployment /> <Nemotron3NanoDeployment />
docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+1 -3
View File
@@ -111,11 +111,9 @@ docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.
### 3.1 Basic Configuration ### 3.1 Basic Configuration
docs/cookbook/autoregressive/Qwen/Qwen3.mdx
+1 -3
View File
@@ -26,15 +26,13 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration ### 3.1 Basic Configuration
The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
docs/cookbook/diffusion/FLUX/FLUX.mdx
+1 -1
View File
@@ -34,7 +34,7 @@ This section provides deployment configurations optimized for different hardware
FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version. FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2/A3 Series NPUs. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs, Ascend A2, A3 NPUs Series NPUs and Intel Arc B-series graphics(codename: BMG (Battlemage)).
<FluxDeployment /> <FluxDeployment />
docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ This section provides deployment configurations optimized for different hardware
Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware. Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving Z-Image-Turbo on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs, Ascend A2, A3 NPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)).
<ZImageTurboDeployment /> <ZImageTurboDeployment />
docs/docs/hardware-platforms/xpu.mdx
+16 -34
View File
@@ -7,39 +7,8 @@ The document addresses how to set up the [SGLang](https://github.com/sgl-project
Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [ Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [
Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html). Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
## Optimized Model List A number of popular LLMs are optimized and run efficiently on XPU, including the most notable open-source models like Llama series, Qwen series, and Diffusion model series like FLUX and Z-Image.
Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found.
A list of LLMs have been optimized on Intel GPU, and more are on the way:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "50%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen2.5-1.5B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)</td>
</tr>
</tbody>
</table>
**Note:** The model identifiers listed in the table above
have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).
Quantized MoE models are covered separately in Quantized MoE models are covered separately in
[MXFP4 MoE Quantization](#mxfp4-moe-quantization) below. [MXFP4 MoE Quantization](#mxfp4-moe-quantization) below.
@@ -85,6 +54,20 @@ pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu
### Install Using Docker ### Install Using Docker
It is recommended to use Docker for setting up the SGLang environment.
#### Pull from Docker Hub
Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository.
The [XPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu) end with `xpu` suffix.
The image pulling command is like:
```bash Command
docker pull lmsysorg/sglang:v0.5.20-xpu
```
#### Build from Dockerfile
[The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation. [The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
@@ -121,7 +104,6 @@ Example command to launch SGLang serving:
sglang serve \ sglang serve \
--model-path <MODEL_ID_OR_PATH> \ --model-path <MODEL_ID_OR_PATH> \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \
--device xpu \ --device xpu \
--host 0.0.0.0 \ --host 0.0.0.0 \
--tp 2 \ # using multi GPUs --tp 2 \ # using multi GPUs
docs/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx
+12 -8
View File
@@ -10,6 +10,7 @@ export const DeepSeekOCR2Deployment = () => {
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }, { id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
] ]
}, },
quantization: { quantization: {
@@ -25,8 +26,8 @@ export const DeepSeekOCR2Deployment = () => {
type: 'checkbox', type: 'checkbox',
items: [ items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true }, { id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }, { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' } { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' }
] ]
}, },
}; };
@@ -42,6 +43,8 @@ export const DeepSeekOCR2Deployment = () => {
cmd += ` --model-path ${modelPath}`; cmd += ` --model-path ${modelPath}`;
if (hardware === 'xeon') { if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule \\\n --trust-remote-code`; cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule \\\n --trust-remote-code`;
} else if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
} }
cmd += ` \\\n --enable-multimodal`; cmd += ` \\\n --enable-multimodal`;
@@ -272,9 +275,8 @@ export const DeepSeekOCR2Deployment = () => {
) : option.type === 'checkbox' ? ( ) : option.type === 'checkbox' ? (
(option.items || []).map((item) => { (option.items || []).map((item) => {
const isChecked = (values[option.name] || []).includes(item.id); const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
item.required || const isDisabled = item.required || dynDisabled;
(typeof item.disabledWhen === 'function' && item.disabledWhen(values));
return ( return (
<label <label
key={item.id} key={item.id}
@@ -289,9 +291,11 @@ export const DeepSeekOCR2Deployment = () => {
type="checkbox" type="checkbox"
checked={isChecked} checked={isChecked}
disabled={isDisabled} disabled={isDisabled}
onChange={(event) => onChange={(event) => {
handleCheckboxChange(option.name, item.id, event.target.checked) if (!dynDisabled) {
} handleCheckboxChange(option.name, item.id, event.target.checked);
}
}}
style={{ display: 'none' }} style={{ display: 'none' }}
/> />
{item.label} {item.label}
docs/src/snippets/autoregressive/gemma4-deployment.jsx
+45 -2
View File
@@ -30,6 +30,7 @@ export const Gemma4Deployment = () => {
{ id: 'b200', label: 'B200', default: false }, { id: 'b200', label: 'B200', default: false },
{ id: 'b300', label: 'B300', default: false }, { id: 'b300', label: 'B300', default: false },
{ id: 'mi300x', label: 'MI300X', default: false, disabled: !showMI300X }, { id: 'mi300x', label: 'MI300X', default: false, disabled: !showMI300X },
{ id: 'arc_b', label: 'BMG', default: false },
]; ];
} }
}, },
@@ -88,6 +89,10 @@ export const Gemma4Deployment = () => {
'31b': { tp: 1, mem: 0.80 }, '31b': { tp: 1, mem: 0.80 },
'26b-a4b': { tp: 1, mem: 0.80 }, '26b-a4b': { tp: 1, mem: 0.80 },
}, },
arc_b: {
'31b': { tp: 4, mem: 0.80 },
'26b-a4b': { tp: 4, mem: 0.75 },
},
}; };
const generateCommand = (values) => { const generateCommand = (values) => {
@@ -141,6 +146,10 @@ export const Gemma4Deployment = () => {
cmd += ` \\\n --attention-backend triton`; cmd += ` \\\n --attention-backend triton`;
} }
if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
}
cmd += ` \\\n --mem-fraction-static ${mem}`; cmd += ` \\\n --mem-fraction-static ${mem}`;
cmd += ` \\\n --host 0.0.0.0 --port 30000`; cmd += ` \\\n --host 0.0.0.0 --port 30000`;
@@ -205,7 +214,27 @@ export const Gemma4Deployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value })); setValues((prev) => {
if (prev.hardware === 'arc_b' && optionName === 'modelSize' && !['31b', '26b-a4b'].includes(value)) {
return prev;
}
if (prev.hardware === 'arc_b' && optionName === 'checkpoint' && value !== 'standard') {
return prev;
}
if (prev.hardware === 'arc_b' && optionName === 'speculative' && value !== 'disabled') {
return prev;
}
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b') {
if (!['31b', '26b-a4b'].includes(next.modelSize)) {
next.modelSize = '31b';
}
next.checkpoint = 'standard';
next.speculative = 'disabled';
}
return next;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -379,7 +408,21 @@ export const Gemma4Deployment = () => {
) : ( ) : (
items.map((item) => { items.map((item) => {
const isChecked = values[option.name] === item.id; const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled); const isArcBModelLocked =
values.hardware === 'arc_b' &&
option.name === 'modelSize' &&
!['31b', '26b-a4b'].includes(item.id);
const isArcBCheckpointLocked =
values.hardware === 'arc_b' &&
option.name === 'checkpoint' &&
item.id !== 'standard';
const isArcBSpeculativeLocked =
values.hardware === 'arc_b' &&
option.name === 'speculative' &&
item.id !== 'disabled';
const isDisabled = Boolean(
item.disabled || isArcBModelLocked || isArcBCheckpointLocked || isArcBSpeculativeLocked
);
return ( return (
<label <label
key={item.id} key={item.id}
docs/src/snippets/autoregressive/llama31-deployment.jsx
+25 -6
View File
@@ -11,7 +11,8 @@ export const Llama31Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false } { id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
] ]
}, },
modelsize: { modelsize: {
@@ -65,21 +66,28 @@ export const Llama31Deployment = () => {
...options.modelsize, ...options.modelsize,
items: options.modelsize.items.map(item => ({ items: options.modelsize.items.map(item => ({
...item, ...item,
disabled: values.hardware === 'xeon' && item.id !== '8b' disabled: (values.hardware === 'xeon' || values.hardware === 'arc_b') && item.id !== '8b'
}))
},
category: {
...options.category,
items: options.category.items.map(item => ({
...item,
disabled: values.hardware === 'arc_b' && item.id !== 'instruct'
})) }))
}, },
quantization: { quantization: {
...options.quantization, ...options.quantization,
items: options.quantization.items.map(item => ({ items: options.quantization.items.map(item => ({
...item, ...item,
disabled: values.hardware === 'xeon' && item.id === 'fp8' disabled: (values.hardware === 'xeon' && item.id === 'fp8') || (values.hardware === 'arc_b' && item.id !== 'bf16')
})) }))
}, },
optimization: { optimization: {
...options.optimization, ...options.optimization,
items: options.optimization.items.map(item => ({ items: options.optimization.items.map(item => ({
...item, ...item,
disabled: values.hardware === 'xeon' && item.id !== 'basic' disabled: (values.hardware === 'xeon' || values.hardware === 'arc_b') && item.id !== 'basic'
})) }))
} }
}; };
@@ -122,6 +130,12 @@ export const Llama31Deployment = () => {
next.quantization = 'bf16'; next.quantization = 'bf16';
next.optimization = 'basic'; next.optimization = 'basic';
} }
if (optionName === 'hardware' && value === 'arc_b') {
next.modelsize = '8b';
next.category = 'instruct';
next.quantization = 'bf16';
next.optimization = 'basic';
}
return next; return next;
}); });
}; };
@@ -131,6 +145,7 @@ export const Llama31Deployment = () => {
const { hardware, optimization, modelsize, category, toolcall, quantization } = values; const { hardware, optimization, modelsize, category, toolcall, quantization } = values;
const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x'; const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
const isXeon = hardware === 'xeon'; const isXeon = hardware === 'xeon';
const effectiveModelSize = isXeon ? '8b' : modelsize; const effectiveModelSize = isXeon ? '8b' : modelsize;
@@ -145,7 +160,7 @@ export const Llama31Deployment = () => {
// Determine model path // Determine model path
let modelPath; let modelPath;
if (quantization === 'fp8' && category === 'instruct' && !isXeon) { if (quantization === 'fp8' && category === 'instruct' && !isXeon && hardware !== 'arc_b') {
if (effectiveModelSize === '405b') { if (effectiveModelSize === '405b') {
// Meta official FP8 for 405B // Meta official FP8 for 405B
modelPath = `meta-llama/Llama-3.1-${sizeToken}${categorySuffix}-FP8`; modelPath = `meta-llama/Llama-3.1-${sizeToken}${categorySuffix}-FP8`;
@@ -186,6 +201,8 @@ export const Llama31Deployment = () => {
} else if (isXeon) { } else if (isXeon) {
// Intel Xeon CPU TP configuration // Intel Xeon CPU TP configuration
tpSize = 3; tpSize = 3;
} else if (hardware === 'arc_b') {
tpSize = 1;
} else { } else {
// NVIDIA GPU TP configuration // NVIDIA GPU TP configuration
if (effectiveModelSize === '405b') { if (effectiveModelSize === '405b') {
@@ -202,6 +219,8 @@ export const Llama31Deployment = () => {
if (isXeon) { if (isXeon) {
args.push(`--device cpu`); args.push(`--device cpu`);
args.push(`--disable-overlap-schedule`); args.push(`--disable-overlap-schedule`);
} else if (hardware === 'arc_b') {
args.push(`--device xpu`);
} }
if (tpSize) { if (tpSize) {
@@ -214,7 +233,7 @@ export const Llama31Deployment = () => {
} }
// NVIDIA-specific optimizations // NVIDIA-specific optimizations
if (!isAMD && !isXeon) { if (!isAMD && !isXeon && hardware !== 'arc_b') {
if (optimization === 'throughput') { if (optimization === 'throughput') {
args.push(`--enable-dp-attention`); args.push(`--enable-dp-attention`);
args.push(`--mem-fraction-static 0.85`); args.push(`--mem-fraction-static 0.85`);
docs/src/snippets/autoregressive/llama33-70b-deployment.jsx
+10 -3
View File
@@ -8,7 +8,8 @@ export const Llama33Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: true }, { id: 'mi300x', label: 'MI300X', default: true },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false } { id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false }
] ]
}, },
quantization: { quantization: {
@@ -35,7 +36,7 @@ export const Llama33Deployment = () => {
...options.quantization, ...options.quantization,
items: options.quantization.items.map(item => ({ items: options.quantization.items.map(item => ({
...item, ...item,
disabled: values.hardware === 'xeon' && item.id === 'fp8' disabled: (values.hardware === 'xeon' && item.id === 'fp8') || (values.hardware === 'arc_b' && item.id !== 'bf16')
})) }))
} }
}); });
@@ -74,6 +75,9 @@ export const Llama33Deployment = () => {
if (optionName === 'hardware' && value === 'xeon') { if (optionName === 'hardware' && value === 'xeon') {
next.quantization = 'bf16'; next.quantization = 'bf16';
} }
if (optionName === 'hardware' && value === 'arc_b') {
next.quantization = 'bf16';
}
return next; return next;
}); });
}; };
@@ -83,7 +87,7 @@ export const Llama33Deployment = () => {
const { hardware, quantization, toolcall } = values; const { hardware, quantization, toolcall } = values;
// Select model based on quantization // Select model based on quantization
const modelPath = quantization === 'fp8' && hardware !== 'xeon' const modelPath = quantization === 'fp8' && hardware !== 'xeon' && hardware !== 'arc_b'
? 'amd/Llama-3.3-70B-Instruct-FP8-KV' ? 'amd/Llama-3.3-70B-Instruct-FP8-KV'
: 'meta-llama/Llama-3.3-70B-Instruct'; : 'meta-llama/Llama-3.3-70B-Instruct';
@@ -94,6 +98,9 @@ export const Llama33Deployment = () => {
cmd += ` --device cpu \\\n`; cmd += ` --device cpu \\\n`;
cmd += ` --disable-overlap-schedule \\\n`; cmd += ` --disable-overlap-schedule \\\n`;
cmd += ` --tp 6`; cmd += ` --tp 6`;
} else if (hardware === 'arc_b') {
cmd += ` --device xpu \\\n`;
cmd += ` --tp 8`;
} else { } else {
cmd += ` --tp 1`; cmd += ` --tp 1`;
} }
docs/src/snippets/autoregressive/nemotron3-nano-deployment.jsx
+23 -3
View File
@@ -8,7 +8,8 @@ export const Nemotron3NanoDeployment = () => {
items: [ items: [
{ id: 'h200', label: 'H200', default: false }, { id: 'h200', label: 'H200', default: false },
{ id: 'b200', label: 'B200', default: true }, { id: 'b200', label: 'B200', default: true },
{ id: 'b300', label: 'B300', default: false } { id: 'b300', label: 'B300', default: false },
{ id: 'arc_b', label: 'BMG', default: false }
] ]
}, },
modelVariant: { modelVariant: {
@@ -76,6 +77,10 @@ export const Nemotron3NanoDeployment = () => {
cmd += ` --attention-backend flashinfer \\\n`; cmd += ` --attention-backend flashinfer \\\n`;
} }
if (hardware === 'arc_b') {
cmd += ` --device xpu \\\n`;
}
// Add thinking parser and tool call parser if enabled // Add thinking parser and tool call parser if enabled
for (const [key, option] of Object.entries(options)) { for (const [key, option] of Object.entries(options)) {
if (option.commandRule) { if (option.commandRule) {
@@ -153,7 +158,18 @@ export const Nemotron3NanoDeployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value })); setValues((prev) => {
if (prev.hardware === 'arc_b' && optionName === 'modelVariant' && value !== 'bf16') {
return prev;
}
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b') {
next.modelVariant = 'bf16';
next.tp = '4';
}
return next;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -327,7 +343,11 @@ export const Nemotron3NanoDeployment = () => {
) : ( ) : (
items.map((item) => { items.map((item) => {
const isChecked = values[option.name] === item.id; const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled); const isArcBModelLocked =
values.hardware === 'arc_b' &&
option.name === 'modelVariant' &&
item.id !== 'bf16';
const isDisabled = Boolean(item.disabled || isArcBModelLocked);
return ( return (
<label <label
key={item.id} key={item.id}
docs/src/snippets/autoregressive/qwen3-deployment.jsx
+39 -8
View File
@@ -23,7 +23,8 @@ export const Qwen3Deployment = () => {
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true } xeon: { tp: 3, ep: 0, bf16: true, fp8: true },
arc_b: { tp: 4, ep: 0, bf16: true, fp8: true },
}, },
'32b': { '32b': {
baseName: '32B', baseName: '32B',
@@ -35,7 +36,8 @@ export const Qwen3Deployment = () => {
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 6, ep: 0, bf16: true, fp8: true } xeon: { tp: 6, ep: 0, bf16: true, fp8: true },
arc_b: { tp: 4, ep: 0, bf16: true, fp8: true }
}, },
'14b': { '14b': {
baseName: '14B', baseName: '14B',
@@ -112,7 +114,8 @@ export const Qwen3Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false } { id: 'xeon', label: 'XEON', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
] ]
}, },
modelsize: { modelsize: {
@@ -169,8 +172,26 @@ export const Qwen3Deployment = () => {
const options = { ...baseOptions }; const options = { ...baseOptions };
const currentModelConfig = modelConfigs[values.modelsize]; const currentModelConfig = modelConfigs[values.modelsize];
if (values.hardware === 'arc_b') {
options.quantization = {
...baseOptions.quantization,
items: baseOptions.quantization.items.map(item => ({
...item,
disabled: item.id !== 'bf16'
}))
};
options.modelsize = {
...baseOptions.modelsize,
items: baseOptions.modelsize.items.map(item => ({
...item,
disabled: item.id !== '30b' && item.id !== '32b'
}))
};
}
// If model doesn't have thinking variants, disable non-base category options // If model doesn't have thinking variants, disable non-base category options
if (currentModelConfig && !currentModelConfig.hasThinkingVariants) { if (values.hardware === 'arc_b' || (currentModelConfig && !currentModelConfig.hasThinkingVariants)) {
options.category = { options.category = {
...baseOptions.category, ...baseOptions.category,
items: baseOptions.category.items.map(item => ({ items: baseOptions.category.items.map(item => ({
@@ -220,6 +241,14 @@ export const Qwen3Deployment = () => {
setValues(prev => { setValues(prev => {
const newValues = { ...prev, [optionName]: value }; const newValues = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b') {
newValues.quantization = 'bf16';
if (newValues.modelsize !== '30b' && newValues.modelsize !== '32b') {
newValues.modelsize = '32b';
}
newValues.category = 'base';
}
// Auto-switch to 'base' category for models without thinking variants // Auto-switch to 'base' category for models without thinking variants
if (optionName === 'modelsize') { if (optionName === 'modelsize') {
const modelConfig = modelConfigs[value]; const modelConfig = modelConfigs[value];
@@ -242,10 +271,10 @@ export const Qwen3Deployment = () => {
// Generate command // Generate command
const generateCommand = () => { const generateCommand = () => {
const { hardware, modelsize, quantization, category, reasoningParser, toolcall } = values; const { hardware, modelsize, quantization, category, reasoningParser, toolcall } = values;
const displayOptions = getDisplayOptions(values); const effectiveQuantization = hardware === 'arc_b' ? 'bf16' : quantization;
// Special error handling // Special error handling
const commandKey = `${hardware}-${modelsize}-${quantization}-${category}`; const commandKey = `${hardware}-${modelsize}-${effectiveQuantization}-${category}`;
if (commandKey === 'h100-235b-bf16-instruct' || commandKey === 'h100-235b-bf16-thinking') { if (commandKey === 'h100-235b-bf16-instruct' || commandKey === 'h100-235b-bf16-thinking') {
return '# Error: Model is too large, cannot fit into 8*H100\n# Please use H200 (141GB) or select FP8 quantization'; return '# Error: Model is too large, cannot fit into 8*H100\n# Please use H200 (141GB) or select FP8 quantization';
} }
@@ -260,7 +289,7 @@ export const Qwen3Deployment = () => {
return `# Error: Unknown hardware platform: ${hardware}`; return `# Error: Unknown hardware platform: ${hardware}`;
} }
const quantSuffix = quantization === 'fp8' ? '-FP8' : ''; const quantSuffix = effectiveQuantization === 'fp8' ? '-FP8' : '';
// Build model name based on model category // Build model name based on model category
let modelName; let modelName;
@@ -281,6 +310,8 @@ export const Qwen3Deployment = () => {
if (hardware === 'xeon') { if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`; cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
} else if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
} }
if (hwConfig.tp > 1) { if (hwConfig.tp > 1) {
@@ -288,7 +319,7 @@ export const Qwen3Deployment = () => {
} }
let ep = hwConfig.ep; let ep = hwConfig.ep;
if (quantization === 'fp8' && hwConfig.tp === 8) { if (effectiveQuantization === 'fp8' && hwConfig.tp === 8) {
ep = 2; ep = 2;
} }
docs/src/snippets/autoregressive/qwen35-deployment.jsx
+35 -13
View File
@@ -65,7 +65,8 @@ export const Qwen35Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false, disabled: isNvfp4 }, { id: 'mi300x', label: 'MI300X', default: false, disabled: isNvfp4 },
{ id: 'mi325x', label: 'MI325X', default: false, disabled: isNvfp4 }, { id: 'mi325x', label: 'MI325X', default: false, disabled: isNvfp4 },
{ id: 'mi355x', label: 'MI355X', default: false, disabled: false }, { id: 'mi355x', label: 'MI355X', default: false, disabled: false },
{ id: 'xeon', label: 'XEON', default: false, disabled: isNvfp4 } { id: 'xeon', label: 'XEON', default: false, disabled: isNvfp4 },
{ id: 'arc_b', label: 'BMG', default: false, disabled: isNvfp4 }
]; ];
} }
}, },
@@ -76,11 +77,12 @@ export const Qwen35Deployment = () => {
const hasFp8 = FP8_MODELS.has(values.model); const hasFp8 = FP8_MODELS.has(values.model);
const hasFp4 = values.model === '397b'; const hasFp4 = values.model === '397b';
const isXeon = values.hardware === 'xeon'; const isXeon = values.hardware === 'xeon';
const isArcB = values.hardware === 'arc_b';
return [ return [
{ id: 'bf16', label: 'BF16', default: !hasFp8 || isXeon }, { id: 'bf16', label: 'BF16', default: !hasFp8 || isXeon || isArcB },
{ id: 'fp8', label: 'FP8', default: hasFp8 && !isXeon, disabled: !hasFp8, { id: 'fp8', label: 'FP8', default: hasFp8 && !isXeon && !isArcB, disabled: !hasFp8 || isArcB,
disabledReason: 'No FP8 variant available for this model' }, disabledReason: 'No FP8 variant available for this model' },
{ id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4 || isXeon, { id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4 || isXeon || isArcB,
disabledReason: isXeon ? 'FP4 is not supported on Xeon' : 'FP4 is only available for Qwen3.5-397B-A17B' } disabledReason: isXeon ? 'FP4 is not supported on Xeon' : 'FP4 is only available for Qwen3.5-397B-A17B' }
]; ];
} }
@@ -104,7 +106,7 @@ export const Qwen35Deployment = () => {
speculative: { speculative: {
name: 'speculative', name: 'speculative',
title: 'Speculative Decoding (MTP)', title: 'Speculative Decoding (MTP)',
condition: (values) => values.hardware !== 'xeon', condition: (values) => values.hardware !== 'xeon' && values.hardware !== 'arc_b',
items: [ items: [
{ id: 'disabled', label: 'Disabled', default: false }, { id: 'disabled', label: 'Disabled', default: false },
{ id: 'enabled', label: 'Enabled', default: true } { id: 'enabled', label: 'Enabled', default: true }
@@ -124,7 +126,7 @@ export const Qwen35Deployment = () => {
mambaCache: { mambaCache: {
name: 'mambaCache', name: 'mambaCache',
title: 'Mamba Radix Cache', title: 'Mamba Radix Cache',
condition: (values) => MOE_MODELS.has(values.model) && values.hardware !== 'xeon', condition: (values) => MOE_MODELS.has(values.model) && values.hardware !== 'xeon' && values.hardware !== 'arc_b',
getDynamicItems: (currentValues) => { getDynamicItems: (currentValues) => {
const amdGpus = ['mi300x', 'mi325x', 'mi355x']; const amdGpus = ['mi300x', 'mi325x', 'mi355x'];
const isAmdGpu = amdGpus.includes(currentValues.hardware); const isAmdGpu = amdGpus.includes(currentValues.hardware);
@@ -184,7 +186,8 @@ export const Qwen35Deployment = () => {
mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } } xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } },
arc_b: { bf16: { tp: 4, mem: 0.8 } }
}, },
'27b': { '27b': {
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
@@ -204,7 +207,8 @@ export const Qwen35Deployment = () => {
mi300x: { bf16: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } }, mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } } xeon: { bf16: { tp: 3 } },
arc_b: { bf16: { tp: 1, mem: 0.8 } }
}, },
'4b': { '4b': {
h100: { bf16: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -214,7 +218,8 @@ export const Qwen35Deployment = () => {
mi300x: { bf16: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } }, mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } } xeon: { bf16: { tp: 3 } },
arc_b: { bf16: { tp: 1, mem: 0.8 } }
}, },
'2b': { '2b': {
h100: { bf16: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -286,7 +291,17 @@ export const Qwen35Deployment = () => {
}, [values.hardware, values.model]); }, [values.hardware, values.model]);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value })); setValues(prev => {
if (prev.hardware === 'arc_b' && optionName === 'model' && !['35b', '9b', '4b'].includes(value)) {
return prev;
}
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b' && !['35b', '9b', '4b'].includes(next.model)) {
next.model = '35b';
}
return next;
});
}; };
// Multi-node flag template — mirrors DeepSeek-V4 cookbook's multiNodeFlags. // Multi-node flag template — mirrors DeepSeek-V4 cookbook's multiNodeFlags.
@@ -354,6 +369,9 @@ export const Qwen35Deployment = () => {
let cmd = `sglang serve --model-path ${modelName}`; let cmd = `sglang serve --model-path ${modelName}`;
if (hardware === 'xeon') { if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`; cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
} else if (hardware === 'arc_b') {
cmd += ` \\\n --device xpu`;
cmd += ` \\\n --linear-attn-backend intel_xpu`;
} }
if (tpValue > 1) { if (tpValue > 1) {
cmd += ` \\\n --tp ${tpValue}`; cmd += ` \\\n --tp ${tpValue}`;
@@ -378,7 +396,7 @@ export const Qwen35Deployment = () => {
// would emit a spurious --mamba-radix-cache-strategy extra_buffer. The UI // would emit a spurious --mamba-radix-cache-strategy extra_buffer. The UI
// radio is hidden for dense models, so users can't manually correct it. // radio is hidden for dense models, so users can't manually correct it.
// MoE keeps the old behavior — the UI radio is the recovery path there. // MoE keeps the old behavior — the UI radio is the recovery path there.
const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon']; const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon', 'arc_b'];
const actualMambaCache = mamba_v1_dev.includes(hardware) const actualMambaCache = mamba_v1_dev.includes(hardware)
? 'v1' ? 'v1'
: (speculative === 'enabled' ? 'v2' : (MOE_MODELS.has(model) ? mambaCache : 'v1')); : (speculative === 'enabled' ? 'v2' : (MOE_MODELS.has(model) ? mambaCache : 'v1'));
@@ -427,7 +445,7 @@ export const Qwen35Deployment = () => {
// benchmark only enables this for TP>=8). AMD MI GPUs use the AITER allreduce // benchmark only enables this for TP>=8). AMD MI GPUs use the AITER allreduce
// fusion flag instead, handled in the AMD backend block below. // fusion flag instead, handled in the AMD backend block below.
const amdGpu = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x'; const amdGpu = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
if (quantization !== 'fp4' && hardware !== 'xeon' && !amdGpu) { if (quantization !== 'fp4' && hardware !== 'xeon' && hardware !== 'arc_b' && !amdGpu) {
cmd += ` \\\n --enable-flashinfer-allreduce-fusion`; cmd += ` \\\n --enable-flashinfer-allreduce-fusion`;
} }
@@ -580,7 +598,11 @@ export const Qwen35Deployment = () => {
<div style={itemsStyle}> <div style={itemsStyle}>
{items.map(item => { {items.map(item => {
const isChecked = values[option.name] === item.id; const isChecked = values[option.name] === item.id;
const isDisabled = !!item.disabled; const isArcBModelLocked =
values.hardware === 'arc_b' &&
option.name === 'model' &&
!['35b', '9b', '4b'].includes(item.id);
const isDisabled = !!item.disabled || isArcBModelLocked;
return ( return (
<label <label
key={item.id} key={item.id}
docs/src/snippets/diffusion/flux-deployment.jsx
+27 -3
View File
@@ -15,7 +15,8 @@ export const FluxDeployment = () => {
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'a2', label: 'A2 Series', default: false }, { id: 'a2', label: 'A2 Series', default: false },
{ id: 'a3', label: 'A3 Series', default: false } { id: 'a3', label: 'A3 Series', default: false },
{ id: 'arc_b', label: 'BMG', default: false },
] ]
}, },
version: { version: {
@@ -58,6 +59,15 @@ sglang serve \\
--num-gpus 2`; --num-gpus 2`;
} }
if (hardware === 'arc_b') {
return `sglang serve \\
--model-path ${config.repoId} \\
--num-gpus 4 \\
--tp-size 4 \\
--component-residency dit=resident,text_encoder=layerwise-offload \\
--dit-cpu-offload False`;
}
return `sglang serve \\ return `sglang serve \\
--model-path ${config.repoId} \\ --model-path ${config.repoId} \\
--ulysses-degree=1 \\ --ulysses-degree=1 \\
@@ -145,7 +155,17 @@ sglang serve \\
}, [values.hardware]); }, [values.hardware]);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value })); setValues((prev) => {
if (prev.hardware === 'arc_b' && optionName === 'version' && value === 'flux1-dev') {
return prev;
}
const nextValues = { ...prev, [optionName]: value };
if (optionName === 'hardware' && value === 'arc_b' && nextValues.version === 'flux1-dev') {
nextValues.version = 'flux2-dev';
}
return nextValues;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -322,7 +342,11 @@ sglang serve \\
) : ( ) : (
items.map((item) => { items.map((item) => {
const isChecked = values[option.name] === item.id; const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled); const isArcBVersionLocked =
values.hardware === 'arc_b' &&
option.name === 'version' &&
item.id === 'flux1-dev';
const isDisabled = Boolean(item.disabled || isArcBVersionLocked);
return ( return (
<label <label
docs/src/snippets/diffusion/zimage-turbo-deployment.jsx
+7 -1
View File
@@ -14,7 +14,8 @@ export const ZImageTurboDeployment = () => {
{ id: 'h200', label: 'H200', default: false }, { id: 'h200', label: 'H200', default: false },
{ id: 'h100', label: 'H100', default: false }, { id: 'h100', label: 'H100', default: false },
{ id: 'a2', label: 'A2 Series', default: false }, { id: 'a2', label: 'A2 Series', default: false },
{ id: 'a3', label: 'A3 Series', default: false } { id: 'a3', label: 'A3 Series', default: false },
{ id: 'arc_b', label: 'BMG', default: false }
] ]
} }
}, },
@@ -37,6 +38,11 @@ sglang serve \\
--num-gpus 2`; --num-gpus 2`;
} }
if (hardware === 'arc_b') {
return `sglang serve \\
--model-path Tongyi-MAI/Z-Image-Turbo`;
}
return `sglang serve \\ return `sglang serve \\
--model-path Tongyi-MAI/Z-Image-Turbo \\ --model-path Tongyi-MAI/Z-Image-Turbo \\
--ulysses-degree=1 \\ --ulysses-degree=1 \\