[Doc][CPU]Update Cookbook with Xeon support info (#27248)

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Zaili Wang
2026-06-05 22:39:40 -07:00
committed by GitHub
co-authored by gemini-code-assist[bot]
parent 38ae22e08c
commit caeb449cd6
26 changed files with 443 additions and 121 deletions
@@ -30,13 +30,15 @@ For more details, please refer to the [official DeepSeek-OCR-2 repository](https
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration ### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, and AMD MI300X, MI355X, and MI325X GPUs. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs.
<DeepSeekOCR2Deployment /> <DeepSeekOCR2Deployment />
@@ -47,6 +49,7 @@ This section provides deployment configurations optimized for different hardware
- **Single GPU Deployment:** DeepSeek-OCR-2 (~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving. - **Single GPU Deployment:** DeepSeek-OCR-2 (~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving.
- **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway). - **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway).
- **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`. - **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`.
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
@@ -26,6 +26,8 @@ For more details, please refer to the [official DeepSeek-OCR repository](https:/
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -15,7 +15,7 @@ import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deeps
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving - **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs - **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, and AMD MI300X, MI325X and MI355X GPUs - **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs
- **High Performance**: Optimized for both throughput and latency scenarios - **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:** **Available Models:**
@@ -37,6 +37,8 @@ For more details, please refer to the [official DeepSeek-R1 repository](https://
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -97,6 +99,8 @@ python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --tru
**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`. **NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -22,6 +22,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
@@ -77,6 +79,8 @@ DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGL
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`. **NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -18,6 +18,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
@@ -75,6 +77,8 @@ DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGL
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`. **NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -1,7 +1,7 @@
--- ---
title: MiniMax-M2.7 title: MiniMax-M2.7
metatags: metatags:
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA and AMD GPUs — model self-evolution, professional software engineering, and native agent teams." description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs — model self-evolution, professional software engineering, and native agent teams."
tag: NEW tag: NEW
--- ---
@@ -26,6 +26,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
**Docker Images by Hardware Platform:** **Docker Images by Hardware Platform:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}> <table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
@@ -139,6 +141,12 @@ import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-d
- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4 - **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8 - **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
**Hardware Requirements: Intel Xeon CPU**
- It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server.
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -26,6 +26,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later. **Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.
## 3. Model Deployment ## 3. Model Deployment
@@ -46,6 +48,7 @@ This section provides a progressive guide from quick deployment to performance o
- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: - **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
@@ -28,9 +28,11 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X and NVIDIA B200, GB200 hardware platforms. This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms.
### 3.1 Configuration ### 3.1 Configuration
@@ -54,6 +56,9 @@ This section provides deployment configurations verified on AMD MI300X, MI325X,
**General:** **General:**
* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command. * **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.
**Intel Xeon CPU:**
* For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -32,6 +32,8 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -56,6 +58,8 @@ The Qwen3-Next series comes in only one size but offers different thinking modes
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -26,13 +26,15 @@ SGLang offers multiple installation methods. You can choose the most suitable in
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration ### 3.1 Basic Configuration
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs. The recommended launch configurations vary by hardware and model size. The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
@@ -76,6 +78,8 @@ python -m sglang.launch_server \
--enable-metrics --enable-metrics
``` ```
* For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -111,6 +111,8 @@ docker pull lmsysorg/sglang:v0.5.9-rocm720-mi35x
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -225,6 +227,8 @@ This section provides deployment configurations optimized for different hardware
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off. **FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
</Warning> </Warning>
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
**NVIDIA:** **NVIDIA:**
@@ -79,6 +79,8 @@ docker pull lmsysorg/sglang:latest
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -141,6 +143,8 @@ All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU a
</table> </table>
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over: Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over:
@@ -78,6 +78,8 @@ Please refer to the [official SGLang installation guide](../../../docs/get-start
The `hy3-preview` tag bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime. The `hy3-preview` tag bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment ## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases. This section provides deployment configurations optimized for different hardware platforms and use cases.
@@ -167,6 +169,8 @@ SGLANG_ENABLE_SPEC_V2=1 sglang serve \
Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload. Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload.
For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation ## 4. Model Invocation
### 4.1 Basic Usage ### 4.1 Basic Usage
@@ -7,7 +7,8 @@ export const DeepSeekOCRDeployment = () => {
items: [ items: [
{ id: 'mi300x', label: 'MI300X', default: true }, { id: 'mi300x', label: 'MI300X', default: true },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
quantization: { quantization: {
@@ -23,8 +24,8 @@ export const DeepSeekOCRDeployment = () => {
type: 'checkbox', type: 'checkbox',
items: [ items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true }, { id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false }, { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false } { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
] ]
} }
}; };
@@ -63,7 +64,20 @@ export const DeepSeekOCRDeployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value })); setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter(id => {
const item = strategyItems.find(s => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -93,6 +107,9 @@ export const DeepSeekOCRDeployment = () => {
let cmd = 'python3 -m sglang.launch_server \\\n'; let cmd = 'python3 -m sglang.launch_server \\\n';
cmd += ` --model-path ${modelPath}`; cmd += ` --model-path ${modelPath}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
cmd += ` \\\n --dtype float16`; cmd += ` \\\n --dtype float16`;
// TP strategy // TP strategy
@@ -110,7 +127,9 @@ export const DeepSeekOCRDeployment = () => {
cmd += ` \\\n --ep 1`; cmd += ` \\\n --ep 1`;
} }
cmd += ` \\\n --enable-symm-mem # Optional: improves performance, but may be unstable`; if (hardware !== 'xeon') {
cmd += ` \\\n --enable-symm-mem # Optional: improves performance, but may be unstable`;
}
return cmd; return cmd;
}; };
@@ -135,10 +154,11 @@ export const DeepSeekOCRDeployment = () => {
{option.type === 'checkbox' ? ( {option.type === 'checkbox' ? (
option.items.map(item => { option.items.map(item => {
const isChecked = (values[option.name] || []).includes(item.id); const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = item.required; const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
return ( const isDisabled = item.required || dynDisabled;
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}> return (
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} /> <label key={item.id} title={item.disabledReason || (dynDisabled ? 'Not supported on the selected hardware' : '')} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => !dynDisabled && handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
{item.label} {item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>} {item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label> </label>
@@ -9,6 +9,7 @@ export const DeepSeekOCR2Deployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false },
] ]
}, },
quantization: { quantization: {
@@ -24,8 +25,8 @@ export const DeepSeekOCR2Deployment = () => {
type: 'checkbox', type: 'checkbox',
items: [ items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true }, { id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false }, { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false } { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
] ]
}, },
}; };
@@ -39,6 +40,9 @@ export const DeepSeekOCR2Deployment = () => {
let cmd = 'sglang serve \\\n'; let cmd = 'sglang serve \\\n';
cmd += ` --model-path ${modelPath}`; cmd += ` --model-path ${modelPath}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule \\\n --trust-remote-code`;
}
cmd += ` \\\n --enable-multimodal`; cmd += ` \\\n --enable-multimodal`;
if (strategyArray.includes('tp')) { if (strategyArray.includes('tp')) {
@@ -120,7 +124,20 @@ export const DeepSeekOCR2Deployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value })); setValues((prev) => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter((id) => {
const item = strategyItems.find((s) => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -10,13 +10,15 @@ export const DeepSeekR1BasicDeployment = () => {
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false },
], ],
}, },
quantization: { quantization: {
name: 'quantization', name: 'quantization',
title: 'Quantization', title: 'Quantization',
getDynamicItems: (values) => { getDynamicItems: (values) => {
const fp4Disabled = values.hardware === 'h100' || values.hardware === 'mi300x'; const isXeon = values.hardware === 'xeon';
const fp4Disabled = values.hardware === 'h100' || values.hardware === 'mi300x' || isXeon;
return [ return [
{ id: 'fp8', label: 'FP8', default: true }, { id: 'fp8', label: 'FP8', default: true },
{ {
@@ -24,7 +26,16 @@ export const DeepSeekR1BasicDeployment = () => {
label: 'FP4', label: 'FP4',
default: false, default: false,
disabled: fp4Disabled, disabled: fp4Disabled,
disabledReason: 'H100 and MI300X only support FP8 quantization', disabledReason: isXeon
? 'Intel Xeon CPUs do not support FP4 quantization'
: 'H100 and MI300X only support FP8 quantization',
},
{
id: 'int8',
label: 'INT8',
default: false,
disabled: !isXeon,
disabledReason: 'INT8 is only available when XEON hardware is selected',
}, },
]; ];
}, },
@@ -35,9 +46,9 @@ export const DeepSeekR1BasicDeployment = () => {
type: 'checkbox', type: 'checkbox',
items: [ items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true }, { id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false }, { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false }, { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false }, { id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs do not support Multi-token Prediction' },
], ],
}, },
thinking: { thinking: {
@@ -89,16 +100,19 @@ export const DeepSeekR1BasicDeployment = () => {
return '# Error: H100 and MI300X only support FP8 quantization'; return '# Error: H100 and MI300X only support FP8 quantization';
} }
const isXeon = hardware === 'xeon';
const modelPath = const modelPath =
quantization === 'fp4' quantization === 'fp4'
? 'nvidia/DeepSeek-R1-0528-FP4-v2' ? 'nvidia/DeepSeek-R1-0528-FP4-v2'
: quantization === 'int8'
? 'Conexis/DeepSeek-R1-0528-Channel-INT8'
: 'deepseek-ai/DeepSeek-R1-0528'; : 'deepseek-ai/DeepSeek-R1-0528';
let command = 'python3 -m sglang.launch_server \\\n'; let command = 'python3 -m sglang.launch_server \\\n';
command += ` --model-path ${modelPath}`; command += ` --model-path ${modelPath}`;
if (strategyValues.includes('tp')) { if (strategyValues.includes('tp')) {
command += ' \\\n --tp 8'; command += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
} }
if (strategyValues.includes('dp')) { if (strategyValues.includes('dp')) {
command += ' \\\n --dp 8 \\\n --enable-dp-attention'; command += ' \\\n --dp 8 \\\n --enable-dp-attention';
@@ -115,13 +129,22 @@ export const DeepSeekR1BasicDeployment = () => {
' \\\n --speculative-num-draft-tokens 4'; ' \\\n --speculative-num-draft-tokens 4';
} }
command += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable'; if (!isXeon) {
command += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable';
}
if (hardware === 'b200' || (hardware === 'mi355x' && quantization === 'fp8')) { if (hardware === 'b200' || (hardware === 'mi355x' && quantization === 'fp8')) {
command += command +=
' \\\n --kv-cache-dtype fp8_e4m3 # Optional: enables fp8 kv cache and fp8 attention kernels to improve performance'; ' \\\n --kv-cache-dtype fp8_e4m3 # Optional: enables fp8 kv cache and fp8 attention kernels to improve performance';
} }
if (isXeon) {
command += ' \\\n --device cpu \\\n --disable-overlap-schedule';
if (quantization === 'int8') {
command += ' \\\n --quantization w8a8_int8';
}
}
if (thinking === 'enabled') { if (thinking === 'enabled') {
command += ' \\\n --reasoning-parser deepseek-r1'; command += ' \\\n --reasoning-parser deepseek-r1';
} }
@@ -170,6 +193,25 @@ export const DeepSeekR1BasicDeployment = () => {
} }
} }
} }
if (optionName === 'hardware') {
if (next.hardware === 'xeon') {
next.quantization = 'int8';
} else if (next.quantization === 'int8') {
next.quantization = 'fp8';
}
}
const strategyItems = options.strategy.items || [];
const currentStrategy = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = currentStrategy.filter((id) => {
const item = strategyItems.find((s) => s.id === id);
if (!item) {
return false;
}
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) {
return false;
}
return true;
});
return next; return next;
}); });
}; };
@@ -10,16 +10,26 @@ export const DeepSeekV3Deployment = () => {
{ id: 'b200', label: 'B200', default: true }, { id: 'b200', label: 'B200', default: true },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
quantization: { quantization: {
name: 'quantization', name: 'quantization',
title: 'Quantization', title: 'Quantization',
items: [ getDynamicItems: (values) => {
{ id: 'fp8', label: 'FP8', default: true }, const isXeon = values.hardware === 'xeon';
{ id: 'fp4', label: 'FP4', default: false } return [
] { id: 'fp8', label: 'FP8', default: true },
{
id: 'fp4',
label: 'FP4',
default: false,
disabled: isXeon,
disabledReason: 'Intel Xeon CPUs do not support FP4 quantization'
}
];
}
}, },
strategy: { strategy: {
name: 'strategy', name: 'strategy',
@@ -27,9 +37,9 @@ export const DeepSeekV3Deployment = () => {
type: 'checkbox', type: 'checkbox',
items: [ items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true }, { id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false }, { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false }, { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false } { id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
] ]
}, },
thinking: { thinking: {
@@ -57,8 +67,11 @@ export const DeepSeekV3Deployment = () => {
if (option.type === 'checkbox') { if (option.type === 'checkbox') {
initialState[key] = option.items.filter(item => item.default).map(item => item.id); initialState[key] = option.items.filter(item => item.default).map(item => item.id);
} else { } else {
const defaultItem = option.items.find(item => item.default); const items = typeof option.getDynamicItems === 'function'
initialState[key] = defaultItem ? defaultItem.id : option.items[0].id; ? option.getDynamicItems(initialState)
: option.items;
const defaultItem = items.find(item => item.default && !item.disabled) || items.find(item => !item.disabled);
initialState[key] = defaultItem ? defaultItem.id : items[0].id;
} }
}); });
return initialState; return initialState;
@@ -84,7 +97,30 @@ export const DeepSeekV3Deployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value })); setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
const quantizationItems = typeof options.quantization.getDynamicItems === 'function'
? options.quantization.getDynamicItems(next)
: options.quantization.items || [];
const currentQuantization = quantizationItems.find(item => item.id === next.quantization);
if (!currentQuantization || currentQuantization.disabled) {
const fallback = quantizationItems.find(item => item.default && !item.disabled) || quantizationItems.find(item => !item.disabled);
if (fallback) {
next.quantization = fallback.id;
}
}
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter(id => {
const item = strategyItems.find(s => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -103,17 +139,18 @@ export const DeepSeekV3Deployment = () => {
const { hardware, quantization, strategy, thinking, toolcall } = values; const { hardware, quantization, strategy, thinking, toolcall } = values;
const strategyArray = Array.isArray(strategy) ? strategy : []; const strategyArray = Array.isArray(strategy) ? strategy : [];
// Validation - H100/H200/MI300X/MI325X only supports FP8 // Validation - H100/H200/MI300X/MI325X/XEON only supports FP8
if (['h100', 'h200', 'mi300x', 'mi325x'].includes(hardware) && quantization === 'fp4') { if (['h100', 'h200', 'mi300x', 'mi325x'].includes(hardware) && quantization === 'fp4') {
return '# Error: This hardware only supports FP8 quantization\n# Please select FP8 quantization or use B200/MI355X hardware'; return '# Error: This hardware only supports FP8 quantization\n# Please select FP8 quantization or use B200/MI355X hardware';
} }
const modelPath = quantization === 'fp4' ? 'nvidia/DeepSeek-V3-0324-NVFP4' : 'deepseek-ai/DeepSeek-V3'; const modelPath = quantization === 'fp4' ? 'nvidia/DeepSeek-V3-0324-NVFP4' : 'deepseek-ai/DeepSeek-V3';
const isXeon = hardware === 'xeon';
let cmd = 'python3 -m sglang.launch_server \\\n'; let cmd = 'python3 -m sglang.launch_server \\\n';
cmd += ` --model-path ${modelPath}`; cmd += ` --model-path ${modelPath}`;
if (strategyArray.includes('tp')) cmd += ' \\\n --tp 8'; if (strategyArray.includes('tp')) cmd += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
if (strategyArray.includes('dp')) cmd += ' \\\n --dp 8 \\\n --enable-dp-attention'; if (strategyArray.includes('dp')) cmd += ' \\\n --dp 8 \\\n --enable-dp-attention';
if (strategyArray.includes('ep')) cmd += ' \\\n --ep 8'; if (strategyArray.includes('ep')) cmd += ' \\\n --ep 8';
if (strategyArray.includes('mtp')) { if (strategyArray.includes('mtp')) {
@@ -121,12 +158,18 @@ export const DeepSeekV3Deployment = () => {
cmd += ' \\\n --speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4'; cmd += ' \\\n --speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4';
} }
cmd += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable'; if (!isXeon) {
cmd += ' \\\n --enable-symm-mem # Optional: improves performance, but may be unstable';
}
if (hardware === 'b200') { if (hardware === 'b200') {
cmd += ' \\\n --kv-cache-dtype fp8_e4m3 # Optional: enables fp8 kv cache and fp8 attention kernels'; cmd += ' \\\n --kv-cache-dtype fp8_e4m3 # Optional: enables fp8 kv cache and fp8 attention kernels';
} }
if (isXeon) {
cmd += ' \\\n --device cpu \\\n --disable-overlap-schedule';
}
if (thinking === 'enabled') cmd += ' \\\n --reasoning-parser deepseek-v3'; if (thinking === 'enabled') cmd += ' \\\n --reasoning-parser deepseek-v3';
if (toolcall === 'enabled') cmd += ' \\\n --tool-call-parser deepseekv3 \\\n --chat-template examples/chat_template/tool_chat_template_deepseekv3.jinja'; if (toolcall === 'enabled') cmd += ' \\\n --tool-call-parser deepseekv3 \\\n --chat-template examples/chat_template/tool_chat_template_deepseekv3.jinja';
@@ -153,21 +196,23 @@ export const DeepSeekV3Deployment = () => {
{option.type === 'checkbox' ? ( {option.type === 'checkbox' ? (
option.items.map(item => { option.items.map(item => {
const isChecked = (values[option.name] || []).includes(item.id); const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = item.required; const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
const isDisabled = item.required || dynDisabled;
return ( return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}> <label key={item.id} title={dynDisabled ? 'Not supported on the selected hardware' : ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} /> <input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => !dynDisabled && handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
{item.label} {item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>} {item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label> </label>
); );
}) })
) : ( ) : (
option.items.map(item => { (option.getDynamicItems ? option.getDynamicItems(values) : option.items).map(item => {
const isChecked = values[option.name] === item.id; const isChecked = values[option.name] === item.id;
const isDisabled = Boolean(item.disabled);
return ( return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}) }}> <label key={item.id} title={item.disabledReason || ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="radio" name={option.name} value={item.id} checked={isChecked} onChange={() => handleRadioChange(option.name, item.id)} style={{ display: 'none' }} /> <input type="radio" name={option.name} value={item.id} checked={isChecked} disabled={isDisabled} onChange={() => !isDisabled && handleRadioChange(option.name, item.id)} style={{ display: 'none' }} />
{item.label} {item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>} {item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label> </label>
@@ -9,7 +9,8 @@ export const DeepSeekV31Deployment = () => {
{ id: 'b200', label: 'B200', default: false }, { id: 'b200', label: 'B200', default: false },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
modelname: { modelname: {
@@ -17,7 +18,8 @@ export const DeepSeekV31Deployment = () => {
title: 'Model Name', title: 'Model Name',
items: [ items: [
{ id: 'v31', label: 'DeepSeek-V3.1', default: true }, { id: 'v31', label: 'DeepSeek-V3.1', default: true },
{ id: 'v31terminus', label: 'DeepSeek-V3.1-Terminus', default: false } { id: 'v31terminus', label: 'DeepSeek-V3.1-Terminus', default: false },
{ id: 'v31terminusint8', label: 'DeepSeek-V3.1-Terminus-Channel-int8', default: false, xeonOnly: true }
] ]
}, },
strategy: { strategy: {
@@ -26,9 +28,9 @@ export const DeepSeekV31Deployment = () => {
type: 'checkbox', type: 'checkbox',
items: [ items: [
{ id: 'tp', label: 'TP', default: true, required: true }, { id: 'tp', label: 'TP', default: true, required: true },
{ id: 'dp', label: 'DP attention', default: false }, { id: 'dp', label: 'DP attention', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'ep', label: 'EP', default: false }, { id: 'ep', label: 'EP', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
{ id: 'mtp', label: 'Multi-token Prediction', default: false } { id: 'mtp', label: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon' }
] ]
}, },
reasoningParser: { reasoningParser: {
@@ -83,7 +85,28 @@ export const DeepSeekV31Deployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value })); setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
if (next.hardware === 'xeon') {
next.modelname = 'v31terminusint8';
} else {
const m = options.modelname.items.find(i => i.id === next.modelname);
if (m && m.xeonOnly) {
next.modelname = options.modelname.items.find(i => !i.xeonOnly && i.default)?.id || 'v31';
}
}
const strategyItems = options.strategy.items || [];
const current = Array.isArray(next.strategy) ? next.strategy : [];
next.strategy = current.filter(id => {
const item = strategyItems.find(s => s.id === id);
if (!item) return false;
if (typeof item.disabledWhen === 'function' && item.disabledWhen(next)) return false;
return true;
});
}
return next;
});
}; };
const handleCheckboxChange = (optionName, itemId, isChecked) => { const handleCheckboxChange = (optionName, itemId, isChecked) => {
@@ -104,17 +127,26 @@ export const DeepSeekV31Deployment = () => {
// Model name mapping // Model name mapping
const modelMap = { const modelMap = {
'v31': 'DeepSeek-V3.1', 'v31': 'deepseek-ai/DeepSeek-V3.1',
'v31terminus': 'DeepSeek-V3.1-Terminus' 'v31terminus': 'deepseek-ai/DeepSeek-V3.1-Terminus',
'v31terminusint8': 'IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8'
}; };
const modelName = `deepseek-ai/${modelMap[modelname]}`; const modelName = modelMap[modelname];
const isXeon = hardware === 'xeon';
let cmd = 'python3 -m sglang.launch_server \\\n'; let cmd = 'python3 -m sglang.launch_server \\\n';
cmd += ` --model-path ${modelName}`; cmd += ` --model-path ${modelName}`;
if (isXeon) {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
if (modelname === 'v31terminusint8') {
cmd += ` \\\n --quantization w8a8_int8`;
}
}
// TP is mandatory // TP is mandatory
cmd += ` \\\n --tp 8`; cmd += isXeon ? ` \\\n --tp 6` : ` \\\n --tp 8`;
if (strategyArray.includes('dp')) { if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`; cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`;
} }
@@ -141,6 +173,7 @@ export const DeepSeekV31Deployment = () => {
cmd += ` \\\n --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja`; cmd += ` \\\n --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja`;
} }
return cmd; return cmd;
}; };
@@ -164,10 +197,11 @@ export const DeepSeekV31Deployment = () => {
{option.type === 'checkbox' ? ( {option.type === 'checkbox' ? (
option.items.map(item => { option.items.map(item => {
const isChecked = (values[option.name] || []).includes(item.id); const isChecked = (values[option.name] || []).includes(item.id);
const isDisabled = item.required; const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values);
const isDisabled = item.required || dynDisabled;
return ( return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}> <label key={item.id} title={dynDisabled ? 'Not supported on the selected hardware' : ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} /> <input type="checkbox" checked={isChecked} disabled={isDisabled} onChange={(e) => !dynDisabled && handleCheckboxChange(option.name, item.id, e.target.checked)} style={{ display: 'none' }} />
{item.label} {item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>} {item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label> </label>
@@ -176,9 +210,10 @@ export const DeepSeekV31Deployment = () => {
) : ( ) : (
option.items.map(item => { option.items.map(item => {
const isChecked = values[option.name] === item.id; const isChecked = values[option.name] === item.id;
const isDisabled = item.xeonOnly && values.hardware !== 'xeon';
return ( return (
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}) }}> <label key={item.id} title={isDisabled ? 'Only available when XEON hardware is selected' : undefined} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(isDisabled ? disabledStyle : {}) }}>
<input type="radio" name={option.name} value={item.id} checked={isChecked} onChange={() => handleRadioChange(option.name, item.id)} style={{ display: 'none' }} /> <input type="radio" name={option.name} value={item.id} checked={isChecked} disabled={isDisabled} onChange={() => !isDisabled && handleRadioChange(option.name, item.id)} style={{ display: 'none' }} />
{item.label} {item.label}
{item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>} {item.subtitle && <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>{item.subtitle}</small>}
</label> </label>
@@ -13,7 +13,8 @@ export const Hunyuan3PreviewDeployment = () => {
{ id: 'h200', label: 'H200', default: true }, { id: 'h200', label: 'H200', default: true },
{ id: 'b200', label: 'B200', default: false }, { id: 'b200', label: 'B200', default: false },
{ id: 'b300', label: 'B300', default: false }, { id: 'b300', label: 'B300', default: false },
{ id: 'gb300', label: 'GB300', default: false } { id: 'gb300', label: 'GB300', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
reasoning: { reasoning: {
@@ -35,10 +36,13 @@ export const Hunyuan3PreviewDeployment = () => {
speculative: { speculative: {
name: 'speculative', name: 'speculative',
title: 'Speculative Decoding (MTP)', title: 'Speculative Decoding (MTP)',
items: [ getDynamicItems: (values) => {
{ id: 'disabled', label: 'Disabled', default: true }, const isXeon = values && values.hardware === 'xeon';
{ id: 'enabled', label: 'Enabled', subtitle: 'Low Latency', default: false } return [
] { id: 'disabled', label: 'Disabled', default: true },
{ id: 'enabled', label: 'Enabled', subtitle: 'Low Latency', default: false, disabled: isXeon, disabledReason: isXeon ? 'Speculative decoding (MTP) is not supported on Intel Xeon CPUs' : '' }
];
}
} }
}; };
@@ -46,7 +50,8 @@ export const Hunyuan3PreviewDeployment = () => {
h200: { tp: 8, mem: 0.9 }, h200: { tp: 8, mem: 0.9 },
b200: { tp: 8, mem: 0.9 }, b200: { tp: 8, mem: 0.9 },
b300: { tp: 4, mem: 0.9 }, b300: { tp: 4, mem: 0.9 },
gb300: { tp: 4, mem: 0.9 } gb300: { tp: 4, mem: 0.9 },
xeon: { tp: 6 }
}; };
const resolveItems = (option, values) => { const resolveItems = (option, values) => {
@@ -82,19 +87,34 @@ export const Hunyuan3PreviewDeployment = () => {
}, []); }, []);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues(prev => ({ ...prev, [optionName]: value })); setValues(prev => {
const next = { ...prev, [optionName]: value };
if (optionName === 'hardware') {
for (const [key, option] of Object.entries(options)) {
if (key === 'hardware') continue;
const items = resolveItems(option, next);
const current = items.find(i => i.id === next[key]);
if (!current || current.disabled) {
const fallback = items.find(i => i.default && !i.disabled) || items.find(i => !i.disabled);
if (fallback) next[key] = fallback.id;
}
}
}
return next;
});
}; };
const generateCommand = () => { const generateCommand = () => {
const { hardware } = values; const { hardware } = values;
const isBlackwell = hardware === 'b200' || hardware === 'b300' || hardware === 'gb300'; const isBlackwell = hardware === 'b200' || hardware === 'b300' || hardware === 'gb300';
const isXeon = hardware === 'xeon';
const hwConfig = modelConfigs[hardware]; const hwConfig = modelConfigs[hardware];
if (!hwConfig) return '# Configuration not available for the selected hardware.'; if (!hwConfig) return '# Configuration not available for the selected hardware.';
const modelName = 'tencent/Hy3-preview'; const modelName = 'tencent/Hy3-preview';
const tpValue = hwConfig.tp; const tpValue = hwConfig.tp;
const memFraction = hwConfig.mem; const memFraction = hwConfig.mem;
const enableSpec = values.speculative === 'enabled'; const enableSpec = values.speculative === 'enabled' && !isXeon;
let cmd = ''; let cmd = '';
if (enableSpec) cmd += 'SGLANG_ENABLE_SPEC_V2=1 '; if (enableSpec) cmd += 'SGLANG_ENABLE_SPEC_V2=1 ';
@@ -112,9 +132,10 @@ export const Hunyuan3PreviewDeployment = () => {
} }
cmd += ' \\\n --trust-remote-code'; cmd += ' \\\n --trust-remote-code';
cmd += ` \\\n --mem-fraction-static ${memFraction}`; if (memFraction !== undefined) cmd += ` \\\n --mem-fraction-static ${memFraction}`;
if (isBlackwell) cmd += ' \\\n --attention-backend trtllm_mha'; if (isBlackwell && !isXeon) cmd += ' \\\n --attention-backend trtllm_mha';
if (isXeon) cmd += ' \\\n --device cpu \\\n --disable-overlap-schedule';
return cmd; return cmd;
}; };
@@ -13,7 +13,8 @@ export const MiniMaxM27Deployment = () => {
{ id: 'h100', label: 'H100', default: false }, { id: 'h100', label: 'H100', default: false },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
gpuCount: { gpuCount: {
@@ -23,6 +24,12 @@ export const MiniMaxM27Deployment = () => {
const hw = values.hardware; const hw = values.hardware;
const isAMD = hw === 'mi300x' || hw === 'mi325x' || hw === 'mi355x'; const isAMD = hw === 'mi300x' || hw === 'mi325x' || hw === 'mi355x';
const isGB300 = hw === 'gb300'; const isGB300 = hw === 'gb300';
const isXeon = hw === 'xeon';
if (isXeon) {
return [
{ id: 'tp6', label: 'TP=6', default: true, disabled: false }
];
}
const canUse2GPU = isAMD || isGB300; const canUse2GPU = isAMD || isGB300;
return [ return [
{ id: '2gpu', label: '2', default: canUse2GPU, disabled: !canUse2GPU }, { id: '2gpu', label: '2', default: canUse2GPU, disabled: !canUse2GPU },
@@ -112,6 +119,7 @@ export const MiniMaxM27Deployment = () => {
const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x'; const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
const isGB300 = hardware === 'gb300'; const isGB300 = hardware === 'gb300';
const isXeon = hardware === 'xeon';
const canUse2GPU = isAMD || isGB300; const canUse2GPU = isAMD || isGB300;
if (gpuCount === '2gpu' && !canUse2GPU) { if (gpuCount === '2gpu' && !canUse2GPU) {
@@ -123,7 +131,11 @@ export const MiniMaxM27Deployment = () => {
let cmd = 'sglang serve \\\n'; let cmd = 'sglang serve \\\n';
cmd += ` --model-path ${modelName}`; cmd += ` --model-path ${modelName}`;
if (gpuCount === '8gpu') { if (isXeon) {
cmd += ' \\\n --device cpu';
cmd += ' \\\n --disable-overlap-schedule';
cmd += ' \\\n --tp 6';
} else if (gpuCount === '8gpu') {
cmd += ' \\\n --tp 8'; cmd += ' \\\n --tp 8';
cmd += ' \\\n --ep 8'; cmd += ' \\\n --ep 8';
} else if (gpuCount === '4gpu') { } else if (gpuCount === '4gpu') {
@@ -138,9 +150,11 @@ export const MiniMaxM27Deployment = () => {
if (thinking === 'enabled') cmd += ' \\\n --reasoning-parser minimax-append-think'; if (thinking === 'enabled') cmd += ' \\\n --reasoning-parser minimax-append-think';
cmd += ' \\\n --trust-remote-code'; cmd += ' \\\n --trust-remote-code';
cmd += ' \\\n --mem-fraction-static 0.85'; if (!isXeon) {
cmd += ' \\\n --mem-fraction-static 0.85';
}
if (isAMD) { if (!isXeon && isAMD) {
cmd += ' \\\n --kv-cache-dtype fp8_e4m3'; cmd += ' \\\n --kv-cache-dtype fp8_e4m3';
cmd += ' \\\n --attention-backend triton'; cmd += ' \\\n --attention-backend triton';
} }
@@ -8,7 +8,8 @@ export const Qwen3CoderDeployment = () => {
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false }, { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'b200', label: 'B200', default: false }, { id: 'b200', label: 'B200', default: false },
{ id: 'gb200', label: 'GB200', default: false } { id: 'gb200', label: 'GB200', default: false },
{ id: 'xeon', label: 'Xeon', default: false }
] ]
}, },
modelSize: { modelSize: {
@@ -22,11 +23,14 @@ export const Qwen3CoderDeployment = () => {
quantization: { quantization: {
name: 'quantization', name: 'quantization',
title: 'Quantization', title: 'Quantization',
items: [ getDynamicItems: (values) => {
{ id: 'bf16', label: 'BF16', default: true }, const isXeon = values.hardware === 'xeon';
{ id: 'fp8', label: 'FP8', default: false }, return [
{ id: 'nvfp4', label: 'NVFP4', default: false } { id: 'bf16', label: 'BF16', default: true },
] { id: 'fp8', label: 'FP8', default: false, disabled: false, disabledReason: '' },
{ id: 'nvfp4', label: 'NVFP4', default: false, disabled: isXeon, disabledReason: isXeon ? 'FP4 is not supported on Xeon' : '' }
];
}
}, },
toolcall: { toolcall: {
name: 'toolcall', name: 'toolcall',
@@ -46,13 +50,15 @@ export const Qwen3CoderDeployment = () => {
mi325x: { tp: 8 }, mi325x: { tp: 8 },
mi355x: { tp: 8 }, mi355x: { tp: 8 },
b200: { tp: 8, ep: 8 }, b200: { tp: 8, ep: 8 },
gb200: { tp: 4, ep: 4 } gb200: { tp: 4, ep: 4 },
xeon: { tp: 6 }
}, },
'30b': { '30b': {
baseName: '30B-A3B', baseName: '30B-A3B',
mi300x: { tp: 1 }, mi300x: { tp: 1 },
mi325x: { tp: 1 }, mi325x: { tp: 1 },
mi355x: { tp: 1 } mi355x: { tp: 1 },
xeon: { tp: 3 }
} }
}; };
@@ -60,6 +66,7 @@ export const Qwen3CoderDeployment = () => {
const { hardware, modelSize, quantization } = values; const { hardware, modelSize, quantization } = values;
const isNvidia = hardware === 'b200' || hardware === 'gb200'; const isNvidia = hardware === 'b200' || hardware === 'gb200';
const isXeon = hardware === 'xeon';
const modelConfig = modelConfigs[modelSize]; const modelConfig = modelConfigs[modelSize];
const hwConfig = modelConfig[hardware]; const hwConfig = modelConfig[hardware];
@@ -88,12 +95,16 @@ export const Qwen3CoderDeployment = () => {
} }
let cmd = ''; let cmd = '';
if (!isNvidia) { if (!isNvidia && !isXeon) {
cmd += 'SGLANG_USE_AITER=0 '; cmd += 'SGLANG_USE_AITER=0 ';
} }
cmd += 'python -m sglang.launch_server \\\n'; cmd += 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`; cmd += ` --model ${modelName}`;
if (isXeon) {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
// TP setting // TP setting
cmd += ` \\\n --tp ${hwConfig.tp}`; cmd += ` \\\n --tp ${hwConfig.tp}`;
@@ -101,7 +112,7 @@ export const Qwen3CoderDeployment = () => {
const ep = hwConfig.ep || (quantization === 'nvfp4' ? 1 : null); const ep = hwConfig.ep || (quantization === 'nvfp4' ? 1 : null);
if (ep) { if (ep) {
cmd += ` \\\n --ep ${ep}`; cmd += ` \\\n --ep ${ep}`;
} else if (modelSize === '480b' && quantization === 'fp8') { } else if (modelSize === '480b' && quantization === 'fp8' && !isXeon) {
// FP8 requires EP=2 for 480B model due to MoE dimension alignment // FP8 requires EP=2 for 480B model due to MoE dimension alignment
// moe_intermediate_size=2560, with tp=8 ep=1: 2560/8=320, 320%128!=0 // moe_intermediate_size=2560, with tp=8 ep=1: 2560/8=320, 320%128!=0
// with tp=8 ep=2: 2560/4=640, 640%128=0 // with tp=8 ep=2: 2560/4=640, 640%128=0
@@ -132,7 +143,7 @@ export const Qwen3CoderDeployment = () => {
}); });
// AMD-specific flags // AMD-specific flags
if (!isNvidia) { if (!isNvidia && !isXeon) {
// Context length verified on MI300X/MI325X/MI355X // Context length verified on MI300X/MI325X/MI355X
cmd += ` \\\n --context-length 8192`; cmd += ` \\\n --context-length 8192`;
@@ -9,7 +9,8 @@ export const Qwen3CoderNextDeployment = () => {
{ id: 'b200', label: 'B200', default: false }, { id: 'b200', label: 'B200', default: false },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
quantization: { quantization: {
@@ -32,6 +33,7 @@ export const Qwen3CoderNextDeployment = () => {
mambaCache: { mambaCache: {
name: 'mambaCache', name: 'mambaCache',
title: 'Mamba Radix Cache', title: 'Mamba Radix Cache',
condition: (values) => values.hardware !== 'xeon',
items: [ items: [
{ id: 'v1', label: 'V1', default: true }, { id: 'v1', label: 'V1', default: true },
{ id: 'v2', label: 'V2', default: false } { id: 'v2', label: 'V2', default: false }
@@ -48,7 +50,8 @@ export const Qwen3CoderNextDeployment = () => {
b200: { bf16: { tp: 2 }, fp8: { tp: 1 } }, b200: { bf16: { tp: 2 }, fp8: { tp: 1 } },
mi300x: { bf16: { tp: 2 }, fp8: { tp: 1 } }, mi300x: { bf16: { tp: 2 }, fp8: { tp: 1 } },
mi325x: { bf16: { tp: 2 }, fp8: { tp: 1 } }, mi325x: { bf16: { tp: 2 }, fp8: { tp: 1 } },
mi355x: { bf16: { tp: 2 }, fp8: { tp: 1 } } mi355x: { bf16: { tp: 2 }, fp8: { tp: 1 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } }
} }
}; };
@@ -61,12 +64,19 @@ export const Qwen3CoderNextDeployment = () => {
} }
const quantConfig = hwConfig[quantization]; const quantConfig = hwConfig[quantization];
if (!quantConfig) {
return '# Configuration not available for the selected hardware/quantization.';
}
const quantSuffix = quantization === 'fp8' ? '-FP8' : ''; const quantSuffix = quantization === 'fp8' ? '-FP8' : '';
const modelName = `Qwen/${modelConfigs.default.baseName}${quantSuffix}`; const modelName = `Qwen/${modelConfigs.default.baseName}${quantSuffix}`;
let cmd = 'python -m sglang.launch_server \\\n'; let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`; cmd += ` --model ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
// TP setting // TP setting
if (quantConfig.tp > 1) { if (quantConfig.tp > 1) {
cmd += ` \\\n --tp ${quantConfig.tp}`; cmd += ` \\\n --tp ${quantConfig.tp}`;
@@ -74,6 +84,9 @@ export const Qwen3CoderNextDeployment = () => {
// Apply commandRule from all options // Apply commandRule from all options
Object.entries(options).forEach(([key, option]) => { Object.entries(options).forEach(([key, option]) => {
if (option.condition && !option.condition(values)) {
return;
}
if (option.commandRule && values[key]) { if (option.commandRule && values[key]) {
const additionalCmd = option.commandRule(values[key], values); const additionalCmd = option.commandRule(values[key], values);
if (additionalCmd) { if (additionalCmd) {
@@ -9,7 +9,8 @@ export const Qwen3NextDeployment = () => {
{ id: 'h100', label: 'H100', default: false }, { id: 'h100', label: 'H100', default: false },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'Xeon', default: false }
] ]
}, },
modelsize: { modelsize: {
@@ -48,6 +49,7 @@ export const Qwen3NextDeployment = () => {
speculative: { speculative: {
name: 'speculative', name: 'speculative',
title: 'Speculative Decoding', title: 'Speculative Decoding',
condition: (values) => values.hardware !== 'xeon',
items: [ items: [
{ id: 'disabled', label: 'Disabled', default: true }, { id: 'disabled', label: 'Disabled', default: true },
{ id: 'enabled', label: 'Enabled', default: false } { id: 'enabled', label: 'Enabled', default: false }
@@ -57,6 +59,7 @@ export const Qwen3NextDeployment = () => {
mambaCache: { mambaCache: {
name: 'mambaCache', name: 'mambaCache',
title: 'Mamba Radix Cache', title: 'Mamba Radix Cache',
condition: (values) => values.hardware !== 'xeon',
items: [ items: [
{ id: 'v1', label: 'V1', default: true }, { id: 'v1', label: 'V1', default: true },
{ id: 'v2', label: 'V2', default: false } { id: 'v2', label: 'V2', default: false }
@@ -74,7 +77,8 @@ export const Qwen3NextDeployment = () => {
b200: { tp: 2, ep: 0, bf16: true, fp8: true }, b200: { tp: 2, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 2, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 2, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 2, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 2, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 2, ep: 0, bf16: true, fp8: true } mi355x: { tp: 2, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
} }
}; };
@@ -99,6 +103,10 @@ export const Qwen3NextDeployment = () => {
let cmd = 'python -m sglang.launch_server \\\n'; let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`; cmd += ` --model ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (hwConfig.tp > 1) { if (hwConfig.tp > 1) {
cmd += ` \\\n --tp ${hwConfig.tp}`; cmd += ` \\\n --tp ${hwConfig.tp}`;
} }
@@ -113,6 +121,9 @@ export const Qwen3NextDeployment = () => {
} }
for (const [key, option] of Object.entries(options)) { for (const [key, option] of Object.entries(options)) {
if (option.condition && !option.condition(values)) {
continue;
}
if (option.commandRule) { if (option.commandRule) {
const rule = option.commandRule(values[key]); const rule = option.commandRule(values[key]);
if (rule) { if (rule) {
@@ -10,7 +10,8 @@ export const Qwen3VLDeployment = () => {
{ id: 'h200', label: 'H200', default: false }, { id: 'h200', label: 'H200', default: false },
{ id: 'mi300x', label: 'MI300X', default: false }, { id: 'mi300x', label: 'MI300X', default: false },
{ id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi325x', label: 'MI325X', default: false },
{ id: 'mi355x', label: 'MI355X', default: false } { id: 'mi355x', label: 'MI355X', default: false },
{ id: 'xeon', label: 'XEON', default: false }
] ]
}, },
modelsize: { modelsize: {
@@ -61,7 +62,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 8, ep: 0, bf16: true, fp8: true }, b200: { tp: 8, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 8, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 8, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 8, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 8, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 8, ep: 0, bf16: true, fp8: true } mi355x: { tp: 8, ep: 0, bf16: true, fp8: true },
xeon: { tp: 6, ep: 0, bf16: true, fp8: true }
}, },
'30b': { '30b': {
baseName: '30B-A3B', baseName: '30B-A3B',
@@ -71,7 +73,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true }, b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true } mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
}, },
'32b': { '32b': {
baseName: '32B', baseName: '32B',
@@ -81,7 +84,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true }, b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true } mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 6, ep: 0, bf16: true, fp8: true }
}, },
'8b': { '8b': {
baseName: '8B', baseName: '8B',
@@ -91,7 +95,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true }, b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true } mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
}, },
'4b': { '4b': {
baseName: '4B', baseName: '4B',
@@ -101,7 +106,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true }, b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true } mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
}, },
'2b': { '2b': {
baseName: '2B', baseName: '2B',
@@ -111,7 +117,8 @@ export const Qwen3VLDeployment = () => {
b200: { tp: 1, ep: 0, bf16: true, fp8: true }, b200: { tp: 1, ep: 0, bf16: true, fp8: true },
mi300x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi300x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi325x: { tp: 1, ep: 0, bf16: true, fp8: true }, mi325x: { tp: 1, ep: 0, bf16: true, fp8: true },
mi355x: { tp: 1, ep: 0, bf16: true, fp8: true } mi355x: { tp: 1, ep: 0, bf16: true, fp8: true },
xeon: { tp: 3, ep: 0, bf16: true, fp8: true }
} }
}; };
@@ -152,6 +159,7 @@ export const Qwen3VLDeployment = () => {
const generateCommand = () => { const generateCommand = () => {
const { hardware, modelsize, quantization, thinking, toolcall } = values; const { hardware, modelsize, quantization, thinking, toolcall } = values;
const commandKey = `${hardware}-${modelsize}-${quantization}-${thinking}`; const commandKey = `${hardware}-${modelsize}-${quantization}-${thinking}`;
const isXeon = hardware === 'xeon';
// Special error handling // Special error handling
if (commandKey === 'h100-235b-bf16-instruct' || commandKey === 'h100-235b-bf16-thinking') { if (commandKey === 'h100-235b-bf16-instruct' || commandKey === 'h100-235b-bf16-thinking') {
@@ -174,6 +182,9 @@ export const Qwen3VLDeployment = () => {
let cmd = 'python -m sglang.launch_server \\\n'; let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`; cmd += ` --model ${modelName}`;
if (isXeon) {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (hwConfig.tp > 1) { if (hwConfig.tp > 1) {
cmd += ` \\\n --tp ${hwConfig.tp}`; cmd += ` \\\n --tp ${hwConfig.tp}`;
@@ -188,7 +199,7 @@ export const Qwen3VLDeployment = () => {
cmd += ` \\\n --ep ${ep}`; cmd += ` \\\n --ep ${ep}`;
} }
if (hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x') { if (!isXeon && (hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x')) {
if (modelsize === '32b' && quantization === 'bf16') { if (modelsize === '32b' && quantization === 'bf16') {
cmd += ` \\\n --context-length 65536`; cmd += ` \\\n --context-length 65536`;
} }
@@ -64,7 +64,8 @@ export const Qwen35Deployment = () => {
{ id: 'b300', label: 'B300', default: isNvfp4, disabled: false }, { id: 'b300', label: 'B300', default: isNvfp4, disabled: false },
{ id: 'mi300x', label: 'MI300X', default: false, disabled: isNvfp4 }, { id: 'mi300x', label: 'MI300X', default: false, disabled: isNvfp4 },
{ id: 'mi325x', label: 'MI325X', default: false, disabled: isNvfp4 }, { id: 'mi325x', label: 'MI325X', default: false, disabled: isNvfp4 },
{ id: 'mi355x', label: 'MI355X', default: false, disabled: isNvfp4 } { id: 'mi355x', label: 'MI355X', default: false, disabled: isNvfp4 },
{ id: 'xeon', label: 'XEON', default: false, disabled: isNvfp4 }
]; ];
} }
}, },
@@ -74,12 +75,13 @@ export const Qwen35Deployment = () => {
getDynamicItems: (values) => { getDynamicItems: (values) => {
const hasFp8 = FP8_MODELS.has(values.model); const hasFp8 = FP8_MODELS.has(values.model);
const hasFp4 = values.model === '397b'; const hasFp4 = values.model === '397b';
const isXeon = values.hardware === 'xeon';
return [ return [
{ id: 'bf16', label: 'BF16', default: !hasFp8 }, { id: 'bf16', label: 'BF16', default: !hasFp8 || isXeon },
{ id: 'fp8', label: 'FP8', default: hasFp8, disabled: !hasFp8, { id: 'fp8', label: 'FP8', default: hasFp8 && !isXeon, disabled: !hasFp8,
disabledReason: 'No FP8 variant available for this model' }, disabledReason: 'No FP8 variant available for this model' },
{ id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4, { id: 'fp4', label: 'FP4', default: false, disabled: !hasFp4 || isXeon,
disabledReason: 'FP4 is only available for Qwen3.5-397B-A17B' } disabledReason: isXeon ? 'FP4 is not supported on Xeon' : 'FP4 is only available for Qwen3.5-397B-A17B' }
]; ];
} }
}, },
@@ -102,6 +104,7 @@ export const Qwen35Deployment = () => {
speculative: { speculative: {
name: 'speculative', name: 'speculative',
title: 'Speculative Decoding (MTP)', title: 'Speculative Decoding (MTP)',
condition: (values) => values.hardware !== 'xeon',
items: [ items: [
{ id: 'disabled', label: 'Disabled', default: false }, { id: 'disabled', label: 'Disabled', default: false },
{ id: 'enabled', label: 'Enabled', default: true } { id: 'enabled', label: 'Enabled', default: true }
@@ -110,7 +113,7 @@ export const Qwen35Deployment = () => {
mambaCache: { mambaCache: {
name: 'mambaCache', name: 'mambaCache',
title: 'Mamba Radix Cache', title: 'Mamba Radix Cache',
condition: (values) => MOE_MODELS.has(values.model), condition: (values) => MOE_MODELS.has(values.model) && values.hardware !== 'xeon',
getDynamicItems: (currentValues) => { getDynamicItems: (currentValues) => {
const amdGpus = ['mi300x', 'mi325x', 'mi355x']; const amdGpus = ['mi300x', 'mi325x', 'mi355x'];
const isAmdGpu = amdGpus.includes(currentValues.hardware); const isAmdGpu = amdGpus.includes(currentValues.hardware);
@@ -149,7 +152,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 }, fp4: { tp: 2, mem: 0.8 } }, b300: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 }, fp4: { tp: 2, mem: 0.8 } },
mi300x: { bf16: { tp: 8, mem: 0.8 }, fp8: { tp: 4, mem: 0.8 } }, mi300x: { bf16: { tp: 8, mem: 0.8 }, fp8: { tp: 4, mem: 0.8 } },
mi325x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } }, mi325x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } },
mi355x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } } mi355x: { bf16: { tp: 4, mem: 0.8 }, fp8: { tp: 2, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } }
}, },
'122b': { '122b': {
h100: { bf16: { tp: 4, mem: 0.88 }, fp8: { tp: 2, mem: 0.8 } }, h100: { bf16: { tp: 4, mem: 0.88 }, fp8: { tp: 2, mem: 0.8 } },
@@ -158,7 +162,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 2 }, fp8: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 2 }, fp8: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 2, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 2, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } }
}, },
'35b': { '35b': {
h100: { bf16: { tp: 1, mem: 0.88 }, fp8: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.88 }, fp8: { tp: 1, mem: 0.8 } },
@@ -167,7 +172,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } }
}, },
'27b': { '27b': {
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
@@ -176,7 +182,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } }
}, },
'9b': { '9b': {
h100: { bf16: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -185,7 +192,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
}, },
'4b': { '4b': {
h100: { bf16: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -194,7 +202,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
}, },
'2b': { '2b': {
h100: { bf16: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -203,7 +212,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
}, },
'0.8b': { '0.8b': {
h100: { bf16: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 } },
@@ -212,7 +222,8 @@ export const Qwen35Deployment = () => {
b300: { bf16: { tp: 1, mem: 0.8 } }, b300: { bf16: { tp: 1, mem: 0.8 } },
mi300x: { bf16: { tp: 1, mem: 0.8 } }, mi300x: { bf16: { tp: 1, mem: 0.8 } },
mi325x: { bf16: { tp: 1, mem: 0.8 } }, mi325x: { bf16: { tp: 1, mem: 0.8 } },
mi355x: { bf16: { tp: 1, mem: 0.8 } } mi355x: { bf16: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 } }
} }
}; };
@@ -322,12 +333,16 @@ export const Qwen35Deployment = () => {
// Initialize the base command // Initialize the base command
let cmd = `sglang serve --model-path ${modelName}`; let cmd = `sglang serve --model-path ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (tpValue > 1) { if (tpValue > 1) {
cmd += ` \\\n --tp ${tpValue}`; cmd += ` \\\n --tp ${tpValue}`;
} }
if (epValue) { if (epValue) {
cmd += ` \\\n --expert-parallel-size ${epValue}`; cmd += ` \\\n --expert-parallel-size ${epValue}`;
} }
// Multi-node wiring goes right after --tp / --expert-parallel-size so the // Multi-node wiring goes right after --tp / --expert-parallel-size so the
// distributed-init flags sit next to the parallelism flags they configure. // distributed-init flags sit next to the parallelism flags they configure.
if (isMultinode) { if (isMultinode) {
@@ -336,7 +351,7 @@ export const Qwen35Deployment = () => {
} }
} }
// Force Mamba V1 for AMD GPUs (V2 requires FLA backend). // Force Mamba V1 for AMD GPUs and Xeon CPUs (V2 requires FLA backend).
// Force Mamba V2 when MTP is enabled. // Force Mamba V2 when MTP is enabled.
// Dense models with MTP off: force V1 — values.mambaCache is not // Dense models with MTP off: force V1 — values.mambaCache is not
// re-resolved on a speculative toggle (useEffect deps are hardware/model), // re-resolved on a speculative toggle (useEffect deps are hardware/model),
@@ -344,8 +359,8 @@ export const Qwen35Deployment = () => {
// would emit a spurious --mamba-scheduler-strategy extra_buffer. The UI // would emit a spurious --mamba-scheduler-strategy extra_buffer. The UI
// radio is hidden for dense models, so users can't manually correct it. // radio is hidden for dense models, so users can't manually correct it.
// MoE keeps the old behavior — the UI radio is the recovery path there. // MoE keeps the old behavior — the UI radio is the recovery path there.
const amdGpus = ['mi300x', 'mi325x', 'mi355x']; const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon'];
const actualMambaCache = amdGpus.includes(hardware) const actualMambaCache = mamba_v1_dev.includes(hardware)
? 'v1' ? 'v1'
: (speculative === 'enabled' ? 'v2' : (MOE_MODELS.has(model) ? mambaCache : 'v1')); : (speculative === 'enabled' ? 'v2' : (MOE_MODELS.has(model) ? mambaCache : 'v1'));
@@ -389,7 +404,7 @@ export const Qwen35Deployment = () => {
} }
// Enable allreduce fusion for all Qwen3.5 configs (skip for FP4: benchmark only enables this for TP>=8). // Enable allreduce fusion for all Qwen3.5 configs (skip for FP4: benchmark only enables this for TP>=8).
if (quantization !== 'fp4') { if (quantization !== 'fp4' && hardware !== 'xeon') {
cmd += ` \\\n --enable-flashinfer-allreduce-fusion`; cmd += ` \\\n --enable-flashinfer-allreduce-fusion`;
} }
@@ -8,6 +8,7 @@ export const Qwen36Deployment = () => {
{ id: 'h100', label: 'H100', default: true }, { id: 'h100', label: 'H100', default: true },
{ id: 'h200', label: 'H200', default: false }, { id: 'h200', label: 'H200', default: false },
{ id: 'b200', label: 'B200', default: false }, { id: 'b200', label: 'B200', default: false },
{ id: 'xeon', label: 'XEON', default: false },
], ],
}, },
modelSize: { modelSize: {
@@ -47,15 +48,20 @@ export const Qwen36Deployment = () => {
speculative: { speculative: {
name: 'speculative', name: 'speculative',
title: 'Speculative Decoding (MTP)', title: 'Speculative Decoding (MTP)',
items: [ getDynamicItems: (values) => {
{ id: 'disabled', label: 'Disabled', default: false }, const isXeon = values.hardware === 'xeon';
{ id: 'enabled', label: 'Enabled', default: true }, return [
], { id: 'disabled', label: 'Disabled', default: isXeon },
{ id: 'enabled', label: 'Enabled', default: !isXeon, disabled: isXeon,
disabledReason: isXeon ? 'Speculative decoding is not supported on Xeon' : '' },
];
},
commandRule: (value) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null, commandRule: (value) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null,
}, },
mambaCache: { mambaCache: {
name: 'mambaCache', name: 'mambaCache',
title: 'Mamba Radix Cache', title: 'Mamba Radix Cache',
condition: (values) => values.hardware !== 'xeon',
getDynamicItems: (values) => { getDynamicItems: (values) => {
const mtpEnabled = values.speculative === 'enabled'; const mtpEnabled = values.speculative === 'enabled';
if (mtpEnabled) { if (mtpEnabled) {
@@ -79,12 +85,14 @@ export const Qwen36Deployment = () => {
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
h200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, h200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
b200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, b200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 3 }, fp8: { tp: 3 } },
}, },
'27b': { '27b': {
baseName: '27B', baseName: '27B',
h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, h100: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
h200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, h200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
b200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } }, b200: { bf16: { tp: 1, mem: 0.8 }, fp8: { tp: 1, mem: 0.8 } },
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } },
}, },
}; };
@@ -136,7 +144,7 @@ export const Qwen36Deployment = () => {
} }
return next; return next;
}); });
}, [values.speculative]); }, [values.speculative, values.hardware]);
const handleRadioChange = (optionName, value) => { const handleRadioChange = (optionName, value) => {
setValues((prev) => ({ ...prev, [optionName]: value })); setValues((prev) => ({ ...prev, [optionName]: value }));
@@ -159,6 +167,9 @@ export const Qwen36Deployment = () => {
} }
cmd += `sglang serve --model-path ${modelName}`; cmd += `sglang serve --model-path ${modelName}`;
if (hardware === 'xeon') {
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
}
if (hwConfig.tp > 1) { if (hwConfig.tp > 1) {
cmd += ` \\\n --tp ${hwConfig.tp}`; cmd += ` \\\n --tp ${hwConfig.tp}`;
} }
@@ -170,6 +181,7 @@ export const Qwen36Deployment = () => {
for (const [key, option] of Object.entries(options)) { for (const [key, option] of Object.entries(options)) {
if (key === 'quantization' || key === 'hardware' || key === 'modelSize') continue; if (key === 'quantization' || key === 'hardware' || key === 'modelSize') continue;
if (option.condition && !option.condition(values)) continue;
if (!option.commandRule) continue; if (!option.commandRule) continue;
const rule = option.commandRule(adjustedValues[key]); const rule = option.commandRule(adjustedValues[key]);
if (rule) { if (rule) {
@@ -180,8 +192,9 @@ export const Qwen36Deployment = () => {
if (hardware === 'b200') { if (hardware === 'b200') {
cmd += ` \\\n --attention-backend trtllm_mha`; cmd += ` \\\n --attention-backend trtllm_mha`;
} }
if (hwConfig.mem !== undefined) {
cmd += ` \\\n --mem-fraction-static ${hwConfig.mem}`; cmd += ` \\\n --mem-fraction-static ${hwConfig.mem}`;
}
return cmd; return cmd;
}; };
@@ -197,6 +210,7 @@ export const Qwen36Deployment = () => {
return ( return (
<div style={containerStyle} className="not-prose"> <div style={containerStyle} className="not-prose">
{Object.entries(options).map(([key, option]) => { {Object.entries(options).map(([key, option]) => {
if (typeof option.condition === 'function' && !option.condition(values)) return null;
const items = resolveItems(option, values); const items = resolveItems(option, values);
return ( return (
<div key={key} style={cardStyle}> <div key={key} style={cardStyle}>