[Doc] [CPU] Cookbook: Update to the cookbook for models supported by Intel Xeon CPUs (#40092)
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com> Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com>
This commit is contained in:
co-authored by
Niu Ziru
Niu Ziru
parent
4b273537de
commit
2b8e4cde95
@@ -26,8 +26,6 @@ For more details, please refer to the [official DeepSeek-OCR repository](https:/
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
@@ -37,8 +37,6 @@ For more details, please refer to the [official DeepSeek-R1 repository](https://
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
@@ -22,8 +22,6 @@ SGLang offers multiple installation methods. You can choose the most suitable in
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
@@ -18,8 +18,6 @@ SGLang offers multiple installation methods. You can choose the most suitable in
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
@@ -15,7 +15,7 @@ ERNIE-4.5 delivers advanced features as below:
|
||||
- **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
|
||||
- **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
|
||||
- **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, MI355X GPUs and Intel Xeon CPUs.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
|
||||
@@ -97,7 +97,7 @@ For other installation methods, please refer to the [official SGLang installatio
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). The recommended launch configurations vary by hardware and model size.
|
||||
The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), Xeon CPUs. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant.
|
||||
|
||||
|
||||
@@ -25,8 +25,6 @@ SGLang offers multiple installation methods. You can choose the most suitable in
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3.Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
@@ -32,8 +32,6 @@ SGLang offers multiple installation methods. You can choose the most suitable in
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
@@ -69,6 +69,12 @@ The default `flow_shift` is `3.0` for T2I, `10.0` for non-Edge video and all act
|
||||
|
||||
## 3. Serve Cosmos3
|
||||
|
||||
For Intel Xeon CPU deployment, use `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` with the selected Cosmos3 checkpoint.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path nvidia/Cosmos3-Nano --port 30010
|
||||
```
|
||||
|
||||
Serve `Cosmos3-Nano` directly from the Hugging Face model ID:
|
||||
|
||||
```bash Command
|
||||
|
||||
@@ -57,6 +57,20 @@ sglang serve \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
Intel Xeon CPU deployment is supported by prefixing the server command with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` and omitting GPU-only flags.
|
||||
|
||||
Serve ERNIE-Image:
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path baidu/ERNIE-Image --port 30010
|
||||
```
|
||||
|
||||
Serve ERNIE-Image-Turbo:
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path baidu/ERNIE-Image-Turbo --port 30010
|
||||
```
|
||||
|
||||
## 4. Generate an image
|
||||
|
||||
Use the OpenAI-compatible image generation API after the server starts:
|
||||
|
||||
@@ -61,6 +61,12 @@ HF_TOKEN=$HF_TOKEN sglang serve \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
Intel Xeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path ideogram-ai/ideogram-4-nf4 --port 30010
|
||||
```
|
||||
|
||||
FP8:
|
||||
|
||||
```bash Command
|
||||
@@ -71,6 +77,12 @@ HF_TOKEN=$HF_TOKEN sglang serve \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
Intel Xeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path ideogram-ai/ideogram-4-fp8 --port 30010
|
||||
```
|
||||
|
||||
Comfy-Org NVFP4:
|
||||
|
||||
```bash Command
|
||||
@@ -93,6 +105,12 @@ HF_TOKEN=$HF_TOKEN sglang serve \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
IntelXeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path fal/ideogram-v4-fast --port 30010
|
||||
```
|
||||
|
||||
fal Instant:
|
||||
|
||||
```bash Command
|
||||
@@ -103,6 +121,12 @@ HF_TOKEN=$HF_TOKEN sglang serve \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
Intel Xeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path fal/ideogram-v4-instant --port 30010
|
||||
```
|
||||
|
||||
### Tensor and sequence parallelism
|
||||
|
||||
Both fal variants support native DiT tensor parallelism and Ulysses sequence parallelism. The following two-GPU layouts were validated at 1024×1024 on B200 GPUs for both `fal/ideogram-v4-fast` and `fal/ideogram-v4-instant`.
|
||||
|
||||
@@ -39,6 +39,12 @@ This section covers deploying Krea-2-Turbo for fast, high-quality image generati
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` and the selected Krea-2 checkpoint.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path krea/Krea-2-Turbo
|
||||
```
|
||||
|
||||
Krea-2-Turbo generates high-quality images in only 8 inference steps. Launch the server with:
|
||||
|
||||
```bash Command
|
||||
|
||||
@@ -48,6 +48,12 @@ sglang serve \
|
||||
--enable-torch-compile false
|
||||
```
|
||||
|
||||
Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only parallel and offload flags.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path robbyant/lingbot-world-v2-14b-causal-fast-diffusers --pipeline-class-name LingBotWorldCausalDMDPipeline
|
||||
```
|
||||
|
||||
### Optional KV-Cache Compression
|
||||
|
||||
Long-running sessions can enable lossy int4 PRQ compression for completed
|
||||
|
||||
@@ -43,6 +43,14 @@ sglang serve \
|
||||
--text-encoder-cpu-offload false
|
||||
```
|
||||
|
||||
|
||||
Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only parallel and offload flags.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path robbyant/lingbot-world-fast-diffusers --pipeline-class-name LingBotWorldCausalDMDPipeline
|
||||
```
|
||||
|
||||
|
||||
### Optional KV-Cache Compression
|
||||
|
||||
Long-running sessions can enable lossy int4 PRQ compression for completed
|
||||
|
||||
@@ -51,6 +51,12 @@ sglang serve \
|
||||
|
||||
`Rabinovich/LongLive-2.0-5B-Diffusers` is the Diffusers-format conversion of the official `Efficient-Large-Model/LongLive-2.0-5B` weights.
|
||||
|
||||
Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` and the selected LongLive checkpoint.
|
||||
|
||||
```bash Command
|
||||
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path Rabinovich/LongLive-2.0-5B-Diffusers
|
||||
```
|
||||
|
||||
## 4. Generation
|
||||
|
||||
### 4.1 Single prompt
|
||||
|
||||
@@ -34,7 +34,7 @@ This section provides deployment configurations optimized for different hardware
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs.
|
||||
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs, Intel Xeon CPUs, and Ascend A2/A3 Series NPUs.
|
||||
|
||||
Use the [command builder](#1-quick-start) above.
|
||||
|
||||
|
||||
@@ -34,7 +34,7 @@ This section provides deployment configurations optimized for different hardware
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom.
|
||||
The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs, Intel Xeon CPUs, and Ascend A2/A3 Series NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom.
|
||||
|
||||
Use the [command builder](#1-quick-start) above.
|
||||
|
||||
|
||||
@@ -123,7 +123,7 @@ cd python
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
# Install SGLang dependent libs, and build SGLang main package
|
||||
uv pip install --upgrade pip setuptools
|
||||
uv pip install .
|
||||
uv pip install . # If installing sgl-diffusion, please use: uv pip install ".[diffusion]"
|
||||
|
||||
# Build the CPU backend kernels
|
||||
cd sglang/kernels/aot
|
||||
@@ -148,7 +148,7 @@ Notes:
|
||||
|
||||
- If you encounter code compilation issues during the `sgl-kernel` building process,
|
||||
please check your `gcc` and `g++` versions and upgrade them if they are outdated.
|
||||
It is recommended to use `gcc-13` and `g++-13` as they have been verified
|
||||
It is recommended to use `gcc-15` and `g++-15` as they have been verified
|
||||
in the official Docker container.
|
||||
|
||||
- The system library path is typically located in one of the following directories:
|
||||
|
||||
@@ -130,10 +130,10 @@ test("legacy image hardware, precision and checkpoint choices remain available",
|
||||
const qwen = get("/Qwen-Image/Qwen-Image.mdx");
|
||||
const edit = get("/Qwen-Image/Qwen-Image-Edit.mdx");
|
||||
const zimage = get("/Z-Image/Z-Image-Turbo.mdx");
|
||||
assert.equal(flux.supportedHardware.length, 10);
|
||||
assert.equal(qwen.supportedHardware.length, 9);
|
||||
assert.equal(edit.supportedHardware.length, 7);
|
||||
assert.equal(zimage.supportedHardware.length, 9);
|
||||
assert.equal(flux.supportedHardware.length, 11);
|
||||
assert.equal(qwen.supportedHardware.length, 10);
|
||||
assert.equal(edit.supportedHardware.length, 8);
|
||||
assert.equal(zimage.supportedHardware.length, 10);
|
||||
assert.ok(resolve(flux, { hw: "arc_b", version: "flux2-dev", gpus_per_node: 4 }).flags.includes("--tp-size 4"));
|
||||
assert.ok(resolve(flux, { hw: "a2", version: "flux2-dev", gpus_per_node: 2 }).flags.includes("--tp-size 2"));
|
||||
assert.ok(
|
||||
@@ -151,7 +151,7 @@ test("Wan preserves standard, multi-GPU, Ascend and LoRA recipes", () => {
|
||||
for (const version of ["1", "2"]) {
|
||||
const config = get(`/Wan/Wan2.${version}.mdx`);
|
||||
assert.equal(config.overlayDims.find((dim) => dim.id === "variant").options.length, 3);
|
||||
for (const hw of config.supportedHardware.filter((id) => !["a2", "a3"].includes(id))) {
|
||||
for (const hw of config.supportedHardware.filter((id) => !["a2", "a3", "xeon"].includes(id))) {
|
||||
const gpus = version === "2" && hw === "b300" ? 8 : 4;
|
||||
const result = resolve(config, { hw, gpus_per_node: gpus });
|
||||
assert.deepEqual(result.builder.errors, []);
|
||||
|
||||
@@ -820,7 +820,8 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
const modelName = resolveModelName(sel);
|
||||
if (cell.pd && cell.commands) return interpolate(cell.commands[mode] || cell.commands.python, envValues, modelName);
|
||||
const nnodes = cellNnodes(cell, sel);
|
||||
const multinode = nnodes > 1;
|
||||
// Xeon's "nodes" selector sizes --tp-size, not a real multi-machine launch.
|
||||
const multinode = nnodes > 1 && sel.hw !== "xeon";
|
||||
const cellEnv = [...(cell.env || []), ...overlayEnv(sel)];
|
||||
const flags = overlayCompose(cell.flags, sel);
|
||||
if (multinode) {
|
||||
@@ -1330,9 +1331,21 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
}
|
||||
}
|
||||
for (const [key, bounds] of Object.entries(commandBuilder.resource?.limits || {})) {
|
||||
if (typeof bounds.disabledWhen === "function" && bounds.disabledWhen(out)) {
|
||||
out[key] = bounds.disabledValue ?? 0;
|
||||
continue;
|
||||
}
|
||||
const fallback = Number(commandBuilder.defaultSelection?.[key] ?? bounds.min ?? 1);
|
||||
const value = Number.parseInt(out[key], 10);
|
||||
out[key] = Math.min(bounds.max, Math.max(bounds.min, Number.isFinite(value) ? value : fallback));
|
||||
const values = typeof bounds.allowedValues === "function"
|
||||
? bounds.allowedValues(out)
|
||||
: (bounds.allowedValues || []);
|
||||
const bounded = Math.min(bounds.max, Math.max(bounds.min, Number.isFinite(value) ? value : fallback));
|
||||
out[key] = values.length
|
||||
? values.reduce((closest, candidate) =>
|
||||
Math.abs(candidate - bounded) < Math.abs(closest - bounded) ? candidate : closest,
|
||||
values[0])
|
||||
: bounded;
|
||||
}
|
||||
for (const key of ["tp_size", "ulysses_degree", "ring_degree"]) {
|
||||
const value = Number.parseInt(out[key], 10);
|
||||
@@ -1662,9 +1675,15 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
nodes: resourcesFollowPlatformDefault
|
||||
? (nextRecipe?.nodes ?? next.nodes)
|
||||
: next.nodes,
|
||||
gpus_per_node: resourcesFollowPlatformDefault
|
||||
? (nextRecipe?.gpus_per_node ?? next.gpus_per_node)
|
||||
: next.gpus_per_node,
|
||||
gpus_per_node: (() => {
|
||||
const bounds = commandBuilder.resource?.limits?.gpus_per_node;
|
||||
if (typeof bounds?.disabledWhen === "function" && bounds.disabledWhen(next)) {
|
||||
return bounds.disabledValue ?? 0;
|
||||
}
|
||||
return resourcesFollowPlatformDefault
|
||||
? (nextRecipe?.gpus_per_node ?? next.gpus_per_node)
|
||||
: next.gpus_per_node;
|
||||
})(),
|
||||
topology_mode: "auto",
|
||||
tp_size: resourcesFollowPlatformDefault
|
||||
? (nextRecipe?.tp_size ?? 1)
|
||||
@@ -1711,11 +1730,12 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
commit(value);
|
||||
};
|
||||
|
||||
const renderBuilderNumberInput = ({ identity, value, min, max, label, onCommit }) => (
|
||||
const renderBuilderNumberInput = ({ identity, value, min, max, label, disabled, onCommit }) => (
|
||||
<input
|
||||
key={identity}
|
||||
type="number"
|
||||
inputMode="numeric"
|
||||
disabled={disabled}
|
||||
min={min}
|
||||
max={max}
|
||||
step="1"
|
||||
@@ -1738,7 +1758,15 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
if (!commandBuilder) return;
|
||||
const bounds = commandBuilder.resource?.limits?.[key] || { min: 1, max: 8 };
|
||||
setSel((prev) => {
|
||||
const value = Math.min(bounds.max, Math.max(bounds.min, Number(prev[key]) + delta));
|
||||
if (typeof bounds.disabledWhen === "function" && bounds.disabledWhen(prev)) return prev;
|
||||
const values = typeof bounds.allowedValues === "function"
|
||||
? bounds.allowedValues(prev)
|
||||
: (bounds.allowedValues || []);
|
||||
const current = Number(prev[key]);
|
||||
const value = values.length
|
||||
? values[Math.max(0, Math.min(values.length - 1,
|
||||
values.findIndex((candidate) => candidate === current) + delta))]
|
||||
: Math.min(bounds.max, Math.max(bounds.min, current + delta));
|
||||
return normalizeBuilderSelection({ ...prev, [key]: value, topology_mode: "auto" });
|
||||
});
|
||||
};
|
||||
@@ -1748,11 +1776,22 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
const value = Number.parseInt(rawValue, 10);
|
||||
if (!Number.isFinite(value)) return;
|
||||
const bounds = commandBuilder.resource?.limits?.[key] || { min: 1, max: 8 };
|
||||
setSel((prev) => normalizeBuilderSelection({
|
||||
setSel((prev) => {
|
||||
if (typeof bounds.disabledWhen === "function" && bounds.disabledWhen(prev)) return prev;
|
||||
const values = typeof bounds.allowedValues === "function"
|
||||
? bounds.allowedValues(prev)
|
||||
: (bounds.allowedValues || []);
|
||||
const bounded = Math.min(bounds.max, Math.max(bounds.min, value));
|
||||
return normalizeBuilderSelection({
|
||||
...prev,
|
||||
[key]: Math.min(bounds.max, Math.max(bounds.min, value)),
|
||||
[key]: values.length
|
||||
? values.reduce((closest, candidate) =>
|
||||
Math.abs(candidate - bounded) < Math.abs(closest - bounded) ? candidate : closest,
|
||||
values[0])
|
||||
: bounded,
|
||||
topology_mode: "auto",
|
||||
}));
|
||||
});
|
||||
});
|
||||
};
|
||||
|
||||
const editBuilderTopology = (key, value) => {
|
||||
@@ -1977,6 +2016,7 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
|
||||
const renderStepper = (key, label, detail) => {
|
||||
const bounds = commandBuilder.resource?.limits?.[key] || { min: 1, max: 8 };
|
||||
const disabled = typeof bounds.disabledWhen === "function" && bounds.disabledWhen(sel);
|
||||
return (
|
||||
<div className="sgd-builder-stepper-field">
|
||||
<div>
|
||||
@@ -1987,7 +2027,7 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
<button
|
||||
type="button"
|
||||
aria-label={`Decrease ${label}`}
|
||||
disabled={Number(sel[key]) <= bounds.min}
|
||||
disabled={disabled || Number(sel[key]) <= bounds.min}
|
||||
onClick={() => updateBuilderResource(key, -1)}
|
||||
>−</button>
|
||||
{renderBuilderNumberInput({
|
||||
@@ -1996,12 +2036,13 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
min: bounds.min,
|
||||
max: bounds.max,
|
||||
label,
|
||||
disabled,
|
||||
onCommit: (value) => setBuilderResource(key, value),
|
||||
})}
|
||||
<button
|
||||
type="button"
|
||||
aria-label={`Increase ${label}`}
|
||||
disabled={Number(sel[key]) >= bounds.max}
|
||||
disabled={disabled || Number(sel[key]) >= bounds.max}
|
||||
onClick={() => updateBuilderResource(key, 1)}
|
||||
>+</button>
|
||||
</div>
|
||||
@@ -2297,7 +2338,11 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
<div className="sgd-builder-node-fields">
|
||||
<label>
|
||||
<span>Head address</span>
|
||||
<input value={builderHeadAddress} onChange={(event) => setBuilderHeadAddress(event.target.value)} />
|
||||
<input
|
||||
value={builderHeadAddress}
|
||||
disabled={sel.hw === "xeon"}
|
||||
onChange={(event) => setBuilderHeadAddress(event.target.value)}
|
||||
/>
|
||||
</label>
|
||||
<label>
|
||||
<span>Node rank</span>
|
||||
@@ -2307,6 +2352,7 @@ export const Deployment = ({ config, benchmarks, agenticLink }) => {
|
||||
min: 0,
|
||||
max: Number(sel.nodes) - 1,
|
||||
label: "Node rank",
|
||||
disabled: sel.hw === "xeon",
|
||||
onCommit: setBuilderNodeRank,
|
||||
})}
|
||||
</label>
|
||||
|
||||
@@ -24,7 +24,8 @@ export const DeepSeekOCRDeployment = () => {
|
||||
type: 'checkbox',
|
||||
items: [
|
||||
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
|
||||
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware !== 'xeon', disabledReason: 'This DP recipe is only available on Intel Xeon CPUs' },
|
||||
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Expert Parallel (EP) is not supported on Intel Xeon CPUs' }
|
||||
]
|
||||
}
|
||||
};
|
||||
@@ -94,6 +95,7 @@ export const DeepSeekOCRDeployment = () => {
|
||||
const generateCommand = () => {
|
||||
const { hardware, quantization, strategy } = values;
|
||||
const strategyArray = Array.isArray(strategy) ? strategy : [];
|
||||
const tpSize = 1;
|
||||
|
||||
// Validation checks
|
||||
// Check MI300X compatibility - MI300X + DeepSeek-OCR only supports FP16 quantization
|
||||
@@ -113,7 +115,11 @@ export const DeepSeekOCRDeployment = () => {
|
||||
|
||||
// TP strategy
|
||||
if (strategyArray.includes('tp')) {
|
||||
cmd += ` \\\n --tp 1`;
|
||||
cmd += ` \\\n --tp ${tpSize}`;
|
||||
}
|
||||
|
||||
if (hardware === 'xeon' && strategyArray.includes('dp')) {
|
||||
cmd += ` \\\n --dp ${tpSize}`;
|
||||
}
|
||||
|
||||
// EP strategy
|
||||
|
||||
@@ -47,7 +47,7 @@ export const DeepSeekR1BasicDeployment = () => {
|
||||
type: 'checkbox',
|
||||
items: [
|
||||
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
|
||||
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
|
||||
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs do not support Multi-token Prediction' },
|
||||
],
|
||||
@@ -116,7 +116,13 @@ export const DeepSeekR1BasicDeployment = () => {
|
||||
command += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
|
||||
}
|
||||
if (strategyValues.includes('dp')) {
|
||||
command += ' \\\n --attn-dp-size 8';
|
||||
if(!isXeon) {
|
||||
command += ' \\\n --attn-dp-size 8';
|
||||
}
|
||||
else {
|
||||
command += ' \\\n --dp 6';
|
||||
command += ' \\\n --enable-dp-attention';
|
||||
}
|
||||
}
|
||||
if (strategyValues.includes('ep')) {
|
||||
command += ' \\\n --ep 8';
|
||||
|
||||
@@ -37,7 +37,7 @@ export const DeepSeekV3Deployment = () => {
|
||||
type: 'checkbox',
|
||||
items: [
|
||||
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
|
||||
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
|
||||
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
|
||||
]
|
||||
@@ -151,7 +151,14 @@ export const DeepSeekV3Deployment = () => {
|
||||
cmd += ` --model-path ${modelPath}`;
|
||||
|
||||
if (strategyArray.includes('tp')) cmd += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
|
||||
if (strategyArray.includes('dp')) cmd += ' \\\n --attn-dp-size 8';
|
||||
if (strategyArray.includes('dp')) {
|
||||
if(!isXeon) {
|
||||
cmd += ' \\\n --attn-dp-size 8';
|
||||
}
|
||||
else {
|
||||
cmd += ' \\\n --dp 6 \\\n --enable-dp-attention';
|
||||
}
|
||||
}
|
||||
if (strategyArray.includes('ep')) cmd += ' \\\n --ep 8';
|
||||
if (strategyArray.includes('mtp')) {
|
||||
cmd += ' \\\n --speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4';
|
||||
|
||||
@@ -28,7 +28,7 @@ export const DeepSeekV31Deployment = () => {
|
||||
type: 'checkbox',
|
||||
items: [
|
||||
{ id: 'tp', label: 'TP', default: true, required: true },
|
||||
{ id: 'dp', label: 'DP attention', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
|
||||
{ id: 'dp', label: 'DP attention', default: false },
|
||||
{ id: 'ep', label: 'EP', default: false, disabledWhen: (v) => v.hardware === 'xeon' },
|
||||
{ id: 'mtp', label: 'Multi-token Prediction', default: false, disabledWhen: (v) => v.hardware === 'xeon' }
|
||||
]
|
||||
@@ -148,7 +148,14 @@ export const DeepSeekV31Deployment = () => {
|
||||
// TP is mandatory
|
||||
cmd += isXeon ? ` \\\n --tp 6` : ` \\\n --tp 8`;
|
||||
if (strategyArray.includes('dp')) {
|
||||
cmd += ` \\\n --attn-dp-size 8`;
|
||||
if(!isXeon){
|
||||
cmd += ` \\\n --attn-dp-size 8`;
|
||||
}
|
||||
else{
|
||||
cmd += ` \\\n --dp 6`;
|
||||
cmd += ` \\\n --enable-dp-attention`;
|
||||
}
|
||||
|
||||
}
|
||||
if (strategyArray.includes('ep')) {
|
||||
cmd += ` \\\n --ep 8`;
|
||||
|
||||
@@ -14,7 +14,8 @@ export const Ernie45Deployment = () => {
|
||||
items: [
|
||||
{ id: 'mi300x', label: 'MI300X', default: true },
|
||||
{ id: 'mi325x', label: 'MI325X', default: false },
|
||||
{ id: 'mi355x', label: 'MI355X', default: false }
|
||||
{ id: 'mi355x', label: 'MI355X', default: false },
|
||||
{ id: 'xeon', label: 'XEON', default: false }
|
||||
]
|
||||
},
|
||||
strategy: {
|
||||
@@ -23,8 +24,8 @@ export const Ernie45Deployment = () => {
|
||||
type: 'checkbox',
|
||||
items: [
|
||||
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (values) => values.modelsize === '21b' },
|
||||
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (values) => values.modelsize === '21b' }
|
||||
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (values) => values.modelsize === '21b' || values.hardware === 'xeon' },
|
||||
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (values) => values.modelsize === '21b' || values.hardware === 'xeon' }
|
||||
]
|
||||
}
|
||||
};
|
||||
@@ -46,7 +47,12 @@ export const Ernie45Deployment = () => {
|
||||
let cmd = 'python3 -m sglang.launch_server \\\n';
|
||||
cmd += ` --model-path ${modelPath}`;
|
||||
|
||||
const tpValue = modelsize === '300b' ? 8 : 1;
|
||||
const isXeon = hardware === 'xeon';
|
||||
if (isXeon) {
|
||||
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
|
||||
}
|
||||
|
||||
const tpValue = isXeon ? 6 : (modelsize === '300b' ? 8 : 1);
|
||||
const dpValue = modelsize === '300b' ? 8 : null;
|
||||
const epValue = modelsize === '300b' ? 8 : null;
|
||||
|
||||
@@ -54,11 +60,11 @@ export const Ernie45Deployment = () => {
|
||||
cmd += ` \\\n --tp ${tpValue}`;
|
||||
}
|
||||
|
||||
if (strategyArray.includes('dp') && modelsize === '300b') {
|
||||
if (strategyArray.includes('dp') && modelsize === '300b' && !isXeon) {
|
||||
cmd += ` \\\n --attn-dp-size ${dpValue}`;
|
||||
}
|
||||
|
||||
if (strategyArray.includes('ep') && modelsize === '300b') {
|
||||
if (strategyArray.includes('ep') && modelsize === '300b' && !isXeon) {
|
||||
cmd += ` \\\n --ep ${epValue}`;
|
||||
}
|
||||
|
||||
|
||||
@@ -25,11 +25,13 @@ export const Gemma4Deployment = () => {
|
||||
getDynamicItems: (values) => {
|
||||
const size = values.modelSize;
|
||||
const showMI300X = size === '31b' || size === '26b-a4b';
|
||||
const showXeon = ['e2b', 'e4b', '31b', '26b-a4b'].includes(size);
|
||||
return [
|
||||
{ id: 'h200', label: 'H200', default: true },
|
||||
{ id: 'b200', label: 'B200', default: false },
|
||||
{ id: 'b300', label: 'B300', default: false },
|
||||
{ id: 'mi300x', label: 'MI300X', default: false, disabled: !showMI300X },
|
||||
{ id: 'xeon', label: 'XEON', default: false, disabled: !showXeon },
|
||||
{ id: 'arc_b', label: 'BMG', default: false },
|
||||
];
|
||||
}
|
||||
@@ -93,6 +95,12 @@ export const Gemma4Deployment = () => {
|
||||
'31b': { tp: 4, mem: 0.80 },
|
||||
'26b-a4b': { tp: 4, mem: 0.75 },
|
||||
},
|
||||
xeon: {
|
||||
e2b: { tp: 1, mem: 0.9 },
|
||||
e4b: { tp: 1, mem: 0.9 },
|
||||
'31b': { tp: 4, mem: 0.5 },
|
||||
'26b-a4b': { tp: 2, mem: 0.5 },
|
||||
},
|
||||
};
|
||||
|
||||
const generateCommand = (values) => {
|
||||
@@ -117,11 +125,14 @@ export const Gemma4Deployment = () => {
|
||||
const modelPath = `${modelNames[modelSize]}${qatSuffix}`;
|
||||
|
||||
const mtpEnabled = values.speculative === 'enabled';
|
||||
if (mtpEnabled && modelSize === '26b-a4b' && hardware !== 'mi300x') {
|
||||
if (mtpEnabled && modelSize === '26b-a4b' && hardware !== 'mi300x' && hardware !== 'xeon') {
|
||||
tp = 2;
|
||||
}
|
||||
|
||||
let cmd = `sglang serve --model-path ${modelPath}`;
|
||||
if (hardware === 'xeon') {
|
||||
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
|
||||
}
|
||||
if (tp > 1) {
|
||||
cmd += ` \\\n --tp ${tp}`;
|
||||
}
|
||||
@@ -135,11 +146,15 @@ export const Gemma4Deployment = () => {
|
||||
});
|
||||
|
||||
if (mtpEnabled) {
|
||||
cmd += ` \\\n --speculative-algorithm NEXTN`;
|
||||
cmd += ` \\\n --speculative-draft-model-path ${modelPath}-assistant`;
|
||||
cmd += ` \\\n --speculative-num-steps 5`;
|
||||
cmd += ` \\\n --speculative-num-draft-tokens 6`;
|
||||
cmd += ` \\\n --speculative-eagle-topk 1`;
|
||||
if (hardware === 'xeon') {
|
||||
cmd += ` \\\n --speculative-algorithm NGRAM`;
|
||||
} else {
|
||||
cmd += ` \\\n --speculative-algorithm NEXTN`;
|
||||
cmd += ` \\\n --speculative-draft-model-path ${modelPath}-assistant`;
|
||||
cmd += ` \\\n --speculative-num-steps 5`;
|
||||
cmd += ` \\\n --speculative-num-draft-tokens 6`;
|
||||
cmd += ` \\\n --speculative-eagle-topk 1`;
|
||||
}
|
||||
}
|
||||
|
||||
if (hardware === 'b300') {
|
||||
@@ -233,6 +248,9 @@ export const Gemma4Deployment = () => {
|
||||
next.checkpoint = 'standard';
|
||||
next.speculative = 'disabled';
|
||||
}
|
||||
if (optionName === 'hardware' && value === 'xeon') {
|
||||
next.checkpoint = 'standard';
|
||||
}
|
||||
return next;
|
||||
});
|
||||
};
|
||||
@@ -420,8 +438,17 @@ export const Gemma4Deployment = () => {
|
||||
values.hardware === 'arc_b' &&
|
||||
option.name === 'speculative' &&
|
||||
item.id !== 'disabled';
|
||||
const isXeonQatLocked =
|
||||
values.hardware === 'xeon' &&
|
||||
option.name === 'checkpoint' &&
|
||||
item.id === 'qat';
|
||||
const isXeon12bLocked =
|
||||
values.hardware === 'xeon' &&
|
||||
option.name === 'modelSize' &&
|
||||
item.id === '12b';
|
||||
const isDisabled = Boolean(
|
||||
item.disabled || isArcBModelLocked || isArcBCheckpointLocked || isArcBSpeculativeLocked
|
||||
item.disabled || isArcBModelLocked || isArcBCheckpointLocked ||
|
||||
isArcBSpeculativeLocked || isXeonQatLocked || isXeon12bLocked
|
||||
);
|
||||
return (
|
||||
<label
|
||||
|
||||
@@ -57,18 +57,6 @@ export const GPTOSSDeployment = () => {
|
||||
}
|
||||
};
|
||||
|
||||
const getDisplayOptions = (values) => ({
|
||||
...options,
|
||||
quantization: options.quantization,
|
||||
speculative: {
|
||||
...options.speculative,
|
||||
items: options.speculative.items.map(item => ({
|
||||
...item,
|
||||
disabled: values.hardware === 'xeon' && item.id === 'enabled'
|
||||
}))
|
||||
}
|
||||
});
|
||||
|
||||
// Initialize state
|
||||
const getInitialState = () => {
|
||||
const initialState = {};
|
||||
@@ -102,13 +90,7 @@ export const GPTOSSDeployment = () => {
|
||||
}, []);
|
||||
|
||||
const handleRadioChange = (optionName, value) => {
|
||||
setValues(prev => {
|
||||
const next = { ...prev, [optionName]: value };
|
||||
if (optionName === 'hardware' && value === 'xeon') {
|
||||
next.speculative = 'disabled';
|
||||
}
|
||||
return next;
|
||||
});
|
||||
setValues(prev => ({ ...prev, [optionName]: value }));
|
||||
};
|
||||
|
||||
// Generate command
|
||||
@@ -173,7 +155,7 @@ export const GPTOSSDeployment = () => {
|
||||
cmd += 'SGLANG_USE_AITER=0 ';
|
||||
}
|
||||
|
||||
if (speculative === 'enabled') {
|
||||
if (speculative === 'enabled' && hardware !== 'xeon') {
|
||||
cmd += 'SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 ';
|
||||
}
|
||||
|
||||
@@ -209,12 +191,16 @@ export const GPTOSSDeployment = () => {
|
||||
|
||||
// Add speculative decoding if enabled (MI30x handled above)
|
||||
if (speculative === 'enabled') {
|
||||
cmd += ` \\\n --speculative-algorithm EAGLE3 \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4`;
|
||||
if (hardware === 'xeon') {
|
||||
cmd += ` \\\n --speculative-algorithm NGRAM`;
|
||||
} else {
|
||||
cmd += ` \\\n --speculative-algorithm EAGLE3 \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4`;
|
||||
|
||||
if (modelsize === '120b') {
|
||||
cmd += ` \\\n --speculative-draft-model-path nvidia/gpt-oss-120b-Eagle3`;
|
||||
} else if (modelsize === '20b') {
|
||||
cmd += ` \\\n --speculative-draft-model-path zhuyksir/EAGLE3-gpt-oss-20b-bf16`;
|
||||
if (modelsize === '120b') {
|
||||
cmd += ` \\\n --speculative-draft-model-path nvidia/gpt-oss-120b-Eagle3`;
|
||||
} else if (modelsize === '20b') {
|
||||
cmd += ` \\\n --speculative-draft-model-path zhuyksir/EAGLE3-gpt-oss-20b-bf16`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -234,7 +220,7 @@ export const GPTOSSDeployment = () => {
|
||||
|
||||
return (
|
||||
<div style={containerStyle} className="not-prose">
|
||||
{Object.entries(getDisplayOptions(values)).map(([key, option]) => (
|
||||
{Object.entries(options).map(([key, option]) => (
|
||||
<div key={key} style={cardStyle}>
|
||||
<div style={titleStyle}>{option.title}</div>
|
||||
<div style={itemsStyle}>
|
||||
|
||||
@@ -8,6 +8,7 @@ export const InternS1Deployment = () => {
|
||||
{ id: 'b300', label: 'B300', default: false },
|
||||
{ id: 'h100', label: 'H100', default: false },
|
||||
{ id: 'h200', label: 'H200', default: false },
|
||||
{ id: 'xeon', label: 'XEON', default: false },
|
||||
],
|
||||
},
|
||||
modelsize: {
|
||||
@@ -51,6 +52,7 @@ export const InternS1Deployment = () => {
|
||||
h200: { bf16: { tp: 8 }, fp8: { tp: 8, ep: 2 } },
|
||||
b200: { bf16: { tp: 8 }, fp8: { tp: 8, ep: 2 } },
|
||||
b300: { bf16: { tp: 8 }, fp8: { tp: 8, ep: 2 } },
|
||||
xeon: { bf16: { tp: 6 }, fp8: { tp: 6 } },
|
||||
},
|
||||
'S1-mini': {
|
||||
baseName: 'S1-mini',
|
||||
@@ -58,15 +60,20 @@ export const InternS1Deployment = () => {
|
||||
h200: { bf16: { tp: 1 }, fp8: { tp: 1 } },
|
||||
b200: { bf16: { tp: 1 }, fp8: { tp: 1 } },
|
||||
b300: { bf16: { tp: 1 }, fp8: { tp: 1 } },
|
||||
xeon: { bf16: { tp: 1 }, fp8: { tp: 1 } },
|
||||
},
|
||||
};
|
||||
|
||||
const resolveItems = (option, vals) =>
|
||||
typeof option.getDynamicItems === 'function' ? option.getDynamicItems(vals) : option.items;
|
||||
|
||||
const getInitialState = () => {
|
||||
const initialState = {};
|
||||
Object.entries(options).forEach(([key, option]) => {
|
||||
const defaultItem = option.items.find((item) => item.default);
|
||||
initialState[key] = defaultItem ? defaultItem.id : option.items[0].id;
|
||||
});
|
||||
for (const [key, option] of Object.entries(options)) {
|
||||
const items = resolveItems(option, initialState);
|
||||
const defaultItem = items.find((item) => item.default && !item.disabled) || items.find((item) => !item.disabled) || items[0];
|
||||
initialState[key] = defaultItem.id;
|
||||
}
|
||||
return initialState;
|
||||
};
|
||||
|
||||
@@ -95,6 +102,22 @@ export const InternS1Deployment = () => {
|
||||
setValues((prev) => ({ ...prev, [optionName]: value }));
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
setValues((prev) => {
|
||||
const next = { ...prev };
|
||||
for (const [key, option] of Object.entries(options)) {
|
||||
if (typeof option.getDynamicItems !== 'function') continue;
|
||||
const items = option.getDynamicItems(next);
|
||||
const current = items.find((item) => item.id === next[key]);
|
||||
if (!current || current.disabled) {
|
||||
const fallback = items.find((item) => item.default && !item.disabled) || items.find((item) => !item.disabled);
|
||||
if (fallback) next[key] = fallback.id;
|
||||
}
|
||||
}
|
||||
return next;
|
||||
});
|
||||
}, [values.hardware]);
|
||||
|
||||
const generateCommand = () => {
|
||||
const { hardware, modelsize, quantization, reasoning, toolcall } = values;
|
||||
const modelConfig = modelConfigs[modelsize];
|
||||
@@ -103,11 +126,16 @@ export const InternS1Deployment = () => {
|
||||
return '# Please select a valid hardware and quantization combination';
|
||||
}
|
||||
|
||||
const isXeon = hardware === 'xeon';
|
||||
const quantSuffix = quantization === 'fp8' ? '-FP8' : '';
|
||||
const modelName = `internlm/Intern-${modelConfig.baseName}${quantSuffix}`;
|
||||
|
||||
const flags = [];
|
||||
flags.push(` --model ${modelName}`);
|
||||
if (isXeon) {
|
||||
flags.push(' --device cpu');
|
||||
flags.push(' --disable-overlap-schedule');
|
||||
}
|
||||
if (hwConfig.tp > 1) flags.push(` --tp ${hwConfig.tp}`);
|
||||
if (hwConfig.ep) flags.push(` --ep ${hwConfig.ep}`);
|
||||
if (quantization === 'fp8') flags.push(` --tokenizer-path internlm/Intern-${modelConfig.baseName}`);
|
||||
@@ -130,20 +158,23 @@ export const InternS1Deployment = () => {
|
||||
|
||||
return (
|
||||
<div style={containerStyle} className="not-prose">
|
||||
{Object.entries(options).map(([key, option]) => (
|
||||
{Object.entries(options).map(([key, option]) => {
|
||||
const items = resolveItems(option, values);
|
||||
return (
|
||||
<div key={key} style={cardStyle}>
|
||||
<div style={titleStyle}>{option.title}</div>
|
||||
<div style={itemsStyle}>
|
||||
{option.items.map((item) => {
|
||||
{items.map((item) => {
|
||||
const isChecked = values[option.name] === item.id;
|
||||
return (
|
||||
<label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}) }}>
|
||||
<label key={item.id} title={item.disabledReason || ''} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}), ...(item.disabled ? { cursor: 'not-allowed', opacity: 0.5 } : {}) }}>
|
||||
<input
|
||||
type="radio"
|
||||
name={option.name}
|
||||
value={item.id}
|
||||
checked={isChecked}
|
||||
onChange={() => handleRadioChange(option.name, item.id)}
|
||||
disabled={Boolean(item.disabled)}
|
||||
onChange={() => !item.disabled && handleRadioChange(option.name, item.id)}
|
||||
style={{ display: 'none' }}
|
||||
/>
|
||||
{item.label}
|
||||
@@ -157,7 +188,8 @@ export const InternS1Deployment = () => {
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
);
|
||||
})}
|
||||
<div style={cardStyle}>
|
||||
<div style={titleStyle}>Run this Command:</div>
|
||||
<pre style={commandDisplayStyle}>{generateCommand()}</pre>
|
||||
|
||||
@@ -8,7 +8,8 @@ export const MiniMaxM2Deployment = () => {
|
||||
items: [
|
||||
{ id: 'mi300x', label: 'MI300X', default: true },
|
||||
{ id: 'mi325x', label: 'MI325X', default: false },
|
||||
{ id: 'mi355x', label: 'MI355X', default: false }
|
||||
{ id: 'mi355x', label: 'MI355X', default: false },
|
||||
{ id: 'xeon', label: 'XEON', default: false }
|
||||
]
|
||||
},
|
||||
modelname: {
|
||||
@@ -54,11 +55,20 @@ export const MiniMaxM2Deployment = () => {
|
||||
};
|
||||
|
||||
const modelName = `${modelFamily}/${modelMap[modelname]}`;
|
||||
const isXeon = hardware === 'xeon';
|
||||
|
||||
let cmd = 'sglang serve \\\n';
|
||||
cmd += ` --model-path ${modelName}`;
|
||||
|
||||
cmd += ` \\\n --tp 4`;
|
||||
if (isXeon) {
|
||||
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
|
||||
}
|
||||
|
||||
cmd += ` \\\n --tp ${isXeon ? 6 : 4}`;
|
||||
|
||||
if (isXeon) {
|
||||
cmd += ` \\\n --dtype bfloat16`;
|
||||
}
|
||||
|
||||
cmd += ` \\\n --trust-remote-code`;
|
||||
|
||||
|
||||
@@ -19,7 +19,8 @@ export const MiniMaxM25Deployment = () => {
|
||||
{ id: 'mi325x', label: 'MI325X', default: false },
|
||||
{ id: 'mi355x', label: 'MI355X', default: false },
|
||||
{ id: 'a2', label: 'Ascend A2', description: '8 × 910B4 / 32 GiB per device. The fixed single-request configuration is smoke tested.', default: false },
|
||||
{ id: 'a3', label: 'Ascend A3', description: '8 cards / 16 dies · W8A8 + EAGLE3. Reference configuration only; A3 hardware validation is pending.', default: false }
|
||||
{ id: 'a3', label: 'Ascend A3', description: '8 cards / 16 dies · W8A8 + EAGLE3. Reference configuration only; A3 hardware validation is pending.', default: false },
|
||||
{ id: 'xeon', label: 'XEON', default: false }
|
||||
]
|
||||
},
|
||||
gpuCount: {
|
||||
@@ -34,6 +35,9 @@ export const MiniMaxM25Deployment = () => {
|
||||
{ id: '8gpu', label: '8', disabled: true }
|
||||
];
|
||||
const isAMD = values.hardware === 'mi300x' || values.hardware === 'mi325x' || values.hardware === 'mi355x';
|
||||
if (values.hardware === 'xeon') {
|
||||
return [{ id: 'tp6', label: 'TP=6', default: true, disabled: true }];
|
||||
}
|
||||
return [
|
||||
{
|
||||
id: '2gpu',
|
||||
@@ -229,6 +233,7 @@ export const MiniMaxM25Deployment = () => {
|
||||
}
|
||||
|
||||
const isAMD = hardware === 'mi300x' || hardware === 'mi325x' || hardware === 'mi355x';
|
||||
const isXeon = hardware === 'xeon';
|
||||
if (gpuCount === '2gpu' && !isAMD) {
|
||||
return '# Please select compatible hardware\n# 2-GPU requires AMD MI300X/MI325X/MI355X';
|
||||
}
|
||||
@@ -245,7 +250,10 @@ export const MiniMaxM25Deployment = () => {
|
||||
cmd += 'python -m sglang.launch_server \\\n';
|
||||
cmd += ` --model-path ${modelName}`;
|
||||
|
||||
if (gpuCount === '8gpu') {
|
||||
if (isXeon) {
|
||||
cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule`;
|
||||
cmd += ` \\\n --tp 6`;
|
||||
} else if (gpuCount === '8gpu') {
|
||||
cmd += ` \\\n --tp 8`;
|
||||
cmd += ` \\\n --ep 8`;
|
||||
} else if (gpuCount === '4gpu') {
|
||||
@@ -271,6 +279,10 @@ export const MiniMaxM25Deployment = () => {
|
||||
cmd += ` \\\n --trust-remote-code`;
|
||||
cmd += ` \\\n --mem-fraction-static 0.85`;
|
||||
|
||||
if (isXeon) {
|
||||
cmd += ` \\\n --dtype bfloat16`;
|
||||
}
|
||||
|
||||
if (isBlackwell) {
|
||||
cmd += ` \\\n --moe-runner-backend flashinfer_trtllm_routed`;
|
||||
cmd += ` \\\n --fp8-gemm-backend flashinfer_trtllm`;
|
||||
@@ -327,11 +339,12 @@ export const MiniMaxM25Deployment = () => {
|
||||
|
||||
const getUpdatedValues = (previous, optionName, value) => {
|
||||
if (isAscend(previous) && ['gpuCount', 'thinking', 'toolcall'].includes(optionName)) return previous;
|
||||
if (previous.hardware === 'xeon' && optionName === 'gpuCount') return previous;
|
||||
const next = { ...previous, [optionName]: value };
|
||||
if (optionName === 'hardware' && value !== 'a3' && value !== 'a2') {
|
||||
const items = options.gpuCount.getDynamicItems(next);
|
||||
if (!items.some((item) => item.id === next.gpuCount && !item.disabled)) {
|
||||
next.gpuCount = items.find((item) => item.default && !item.disabled).id;
|
||||
next.gpuCount = (items.find((item) => item.default && !item.disabled) || items[0]).id;
|
||||
}
|
||||
}
|
||||
return next;
|
||||
|
||||
@@ -29,7 +29,7 @@ export const MiniMaxM27Deployment = () => {
|
||||
const isXeon = hw === 'xeon';
|
||||
if (isXeon) {
|
||||
return [
|
||||
{ id: 'tp6', label: 'TP=6', default: true, disabled: false }
|
||||
{ id: 'tp6', label: 'TP=6', default: true, disabled: true }
|
||||
];
|
||||
}
|
||||
const canUse2GPU = isAMD || isGB300;
|
||||
@@ -116,7 +116,7 @@ export const MiniMaxM27Deployment = () => {
|
||||
const items = option.getDynamicItems(next);
|
||||
const current = items.find(i => i.id === next[key]);
|
||||
if (!current || current.disabled) {
|
||||
const fallback = items.find(i => i.default && !i.disabled) || items.find(i => !i.disabled);
|
||||
const fallback = items.find(i => i.default && !i.disabled) || items.find(i => !i.disabled) || items[0];
|
||||
if (fallback) next[key] = fallback.id;
|
||||
}
|
||||
}
|
||||
@@ -125,7 +125,9 @@ export const MiniMaxM27Deployment = () => {
|
||||
}, [values.hardware]);
|
||||
|
||||
const handleRadioChange = (optionName, value) => {
|
||||
setValues(prev => ({ ...prev, [optionName]: value }));
|
||||
setValues(prev => prev.hardware === 'xeon' && optionName === 'gpuCount'
|
||||
? prev
|
||||
: { ...prev, [optionName]: value });
|
||||
};
|
||||
|
||||
// Generate command mirrors sgl-cookbook src/components/autoregressive/MiniMaxM27ConfigGenerator/index.js
|
||||
|
||||
@@ -50,12 +50,13 @@ export const Qwen3NextDeployment = () => {
|
||||
speculative: {
|
||||
name: 'speculative',
|
||||
title: 'Speculative Decoding',
|
||||
condition: (values) => values.hardware !== 'xeon',
|
||||
items: [
|
||||
{ id: 'disabled', label: 'Disabled', default: true },
|
||||
{ id: 'enabled', label: 'Enabled', default: false }
|
||||
],
|
||||
commandRule: (value) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null
|
||||
commandRule: (value, values) => value !== 'enabled' ? null : values.hardware === 'xeon'
|
||||
? '--speculative-algorithm NGRAM'
|
||||
: '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4'
|
||||
},
|
||||
mambaCache: {
|
||||
name: 'mambaCache',
|
||||
@@ -127,7 +128,7 @@ export const Qwen3NextDeployment = () => {
|
||||
continue;
|
||||
}
|
||||
if (option.commandRule) {
|
||||
const rule = option.commandRule(values[key]);
|
||||
const rule = option.commandRule(values[key], values);
|
||||
if (rule) {
|
||||
cmd += ` \\\n ${rule}`;
|
||||
}
|
||||
|
||||
@@ -34,11 +34,12 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "FLUX",
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "arc_b"],
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "arc_b", "xeon"],
|
||||
hardware: [
|
||||
{ id: "a2", label: "Ascend A2", vram: "", vendor: "npu" },
|
||||
{ id: "a3", label: "Ascend A3", vram: "2 NPUs / card", vendor: "npu" },
|
||||
{ id: "arc_b", label: "Intel BMG", vram: "", vendor: "intel" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
@@ -57,8 +58,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } },
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "arc_b"].map(
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 8, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "arc_b", "xeon"].map(
|
||||
(hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
export const config = (() => {
|
||||
const modelOf = (s) => "Lightricks/" + (s.model === "ltx2" ? "LTX-2" : "LTX-2.3");
|
||||
const plan = (s) => {
|
||||
const gpus = Number(s.gpus_per_node);
|
||||
const gpus = s.hw === "xeon" ? 1 : Number(s.gpus_per_node);
|
||||
const tp = gpus === 4 ? 2 : 1;
|
||||
const flags = [
|
||||
"--model-path " + modelOf(s),
|
||||
@@ -20,6 +20,7 @@ export const config = (() => {
|
||||
flags,
|
||||
gpus,
|
||||
topology: { tp_size: tp, ulysses_degree: 1, ring_degree: 1 },
|
||||
env: s.hw === "xeon" ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
errors: [
|
||||
...(![1, 2, 4].includes(gpus) ? ["Choose 1, 2 or 4 GPUs."] : []),
|
||||
...(s.model === "ltx2" && s.pipeline === "two-stage-hq" ? ["HQ requires LTX-2.3."] : []),
|
||||
@@ -41,8 +42,11 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "LTX-2 / LTX-2.3",
|
||||
supportedHardware: ["h200", "cuda"],
|
||||
hardware: [{ id: "cuda", label: "NVIDIA CUDA", vram: "", vendor: "nvidia" }],
|
||||
supportedHardware: ["h200", "cuda", "xeon"],
|
||||
hardware: [
|
||||
{ id: "cuda", label: "NVIDIA CUDA", vram: "", vendor: "nvidia" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
overlayDims: [
|
||||
@@ -73,7 +77,12 @@ export const config = (() => {
|
||||
scope: "serve",
|
||||
default: "resident",
|
||||
options: [
|
||||
{ id: "resident", label: "Resident" },
|
||||
{
|
||||
id: "resident",
|
||||
label: "Resident",
|
||||
disabled: (s) => s.hw === "xeon",
|
||||
disableReason: "Intel Xeon CPUs use the original device mode.",
|
||||
},
|
||||
{ id: "original", label: "Original" },
|
||||
],
|
||||
},
|
||||
@@ -95,9 +104,12 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 4 } },
|
||||
verifiedRecipes: ["h200", "cuda"].map((hw) => {
|
||||
const s = { ...defaults, hw, device: hw === "cuda" ? "original" : "resident" };
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 4, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["h200", "cuda", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw, device: hw === "h200" ? "resident" : "original" };
|
||||
const p = plan(s);
|
||||
return {
|
||||
...s,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
export const config = (() => {
|
||||
const modelOf = () => "Lightricks/LTX-2.5-Diffusers";
|
||||
const plan = (s) => {
|
||||
const gpus = Number(s.gpus_per_node);
|
||||
const gpus = s.hw === "xeon" ? 1 : Number(s.gpus_per_node);
|
||||
const flags = [
|
||||
"--model-path " + modelOf(s),
|
||||
"--pipeline-class-name " + (s.pipeline === "two-stage" ? "LTX2TwoStagePipeline" : "LTX2Pipeline"),
|
||||
@@ -24,6 +24,7 @@ export const config = (() => {
|
||||
ulysses_degree: gpus === 2 && s.parallel === "sp" ? 2 : 1,
|
||||
ring_degree: 1,
|
||||
},
|
||||
env: s.hw === "xeon" ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
warnings: [
|
||||
...(s.parallel === "cfg" && gpus === 2 && s.weights !== "dev"
|
||||
? ["Distilled weights run unguided; CFG parallelism does not accelerate them."]
|
||||
@@ -55,8 +56,11 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "LTX-2.5",
|
||||
supportedHardware: ["h200", "cuda"],
|
||||
hardware: [{ id: "cuda", label: "NVIDIA CUDA", vram: "", vendor: "nvidia" }],
|
||||
supportedHardware: ["h200", "cuda", "xeon"],
|
||||
hardware: [
|
||||
{ id: "cuda", label: "NVIDIA CUDA", vram: "", vendor: "nvidia" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
overlayDims: [
|
||||
@@ -145,8 +149,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 2 } },
|
||||
verifiedRecipes: ["h200", "cuda"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 2, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["h200", "cuda", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
@@ -186,6 +186,7 @@ return {
|
||||
"gb200",
|
||||
"h200",
|
||||
"h100",
|
||||
"xeon",
|
||||
"mi300x",
|
||||
"mi355x",
|
||||
"rtxpro6000",
|
||||
@@ -203,6 +204,7 @@ return {
|
||||
"rtx3060",
|
||||
],
|
||||
hardware: [
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
{ id: "rtxpro6000", label: "RTX PRO 6000", vram: "96GB", vendor: "consumer" },
|
||||
{ id: "rtx6000ada", label: "RTX 6000 Ada", vram: "48GB", vendor: "consumer" },
|
||||
{ id: "rtx5090", label: "RTX 5090", vram: "32GB", vendor: "consumer" },
|
||||
@@ -457,7 +459,7 @@ return {
|
||||
{
|
||||
id: "auto",
|
||||
label: "Auto",
|
||||
flags: (s) => (s.nodes > 1 ? ["--encoder-parallel replicate"] : []),
|
||||
flags: (s) => s.hw === "xeon" ? ["--encoder-parallel auto"] : (s.nodes > 1 ? ["--encoder-parallel replicate"] : []),
|
||||
recommended: true,
|
||||
description: "Folds on verified single-host P2P systems and resolves to replicate across nodes.",
|
||||
},
|
||||
@@ -465,7 +467,7 @@ return {
|
||||
id: "dp",
|
||||
label: "Data parallel",
|
||||
flags: ["--encoder-parallel dp"],
|
||||
disabled: (s) => CONSUMER_SINGLE.includes(s.hw) || (s.topology_mode === "manual"
|
||||
disabled: (s) => s.hw === "xeon" || CONSUMER_SINGLE.includes(s.hw) || (s.topology_mode === "manual"
|
||||
? Number(s.tp_size)
|
||||
: config.commandBuilder.resource.autoTopology(s).tp_size) > 1,
|
||||
disableReason: "Encoder DP requires TP1 and a multi-GPU DP group; TP > 1 and the single-card consumer recipes do not qualify.",
|
||||
@@ -477,7 +479,7 @@ return {
|
||||
id: "fold",
|
||||
label: "Fold",
|
||||
flags: ["--encoder-parallel fold"],
|
||||
disabled: (s) => s.nodes > 1,
|
||||
disabled: (s) => s.hw === "xeon" || s.nodes > 1,
|
||||
disableReason: "Fold assumes fast node-local peer-to-peer access.",
|
||||
description: "Uses one folded encoder copy across a node-local group and preserves native weights.",
|
||||
},
|
||||
@@ -485,6 +487,8 @@ return {
|
||||
id: "replicate",
|
||||
label: "Replicate",
|
||||
flags: ["--encoder-parallel replicate"],
|
||||
disabled: (s) => s.hw === "xeon",
|
||||
disableReason: "Xeon always runs the auto encoder policy.",
|
||||
recommendedWhen: (s) => s.nodes > 1,
|
||||
description: "The safe cross-node default because encoder auto is not node-boundary aware.",
|
||||
},
|
||||
@@ -606,8 +610,8 @@ return {
|
||||
},
|
||||
resource: {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 8 },
|
||||
gpus_per_node: { min: 1, max: 8 },
|
||||
nodes: { min: 1, max: 8, allowedValues: (s) => s.hw === "xeon" ? [1, 2, 4, 8] : [] },
|
||||
gpus_per_node: { min: 1, max: 8, disabledWhen: (s) => s.hw === "xeon", disabledValue: 0 },
|
||||
},
|
||||
verifiedRecipes: [
|
||||
{ id: "b200-resident-8", hw: "b200", nodes: 1, gpus_per_node: 8, placement: "resident", tp_size: 1, ulysses_degree: 8, ring_degree: 1, encoder: "auto", default: true },
|
||||
@@ -681,6 +685,27 @@ return {
|
||||
},
|
||||
},
|
||||
resolveDeployment: (s) => {
|
||||
if (s.hw === "xeon") {
|
||||
return {
|
||||
match: { hw: s.hw },
|
||||
nnodes: 1,
|
||||
verified: false,
|
||||
verificationStatus: "unverified",
|
||||
env: ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"],
|
||||
flags: [
|
||||
`--model-path MiniMaxAI/MiniMax-H3 \\\n --tp-size ${s.nodes}`,
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
builder: {
|
||||
topologySummary: "CPU",
|
||||
errors: [],
|
||||
warnings: ["Xeon CPU deployment is supported."],
|
||||
verification: { serve: "unverified", request: "unverified" },
|
||||
resolvedSettings: {},
|
||||
},
|
||||
};
|
||||
}
|
||||
const resource = config.commandBuilder.resource;
|
||||
const topology = s.topology_mode === "manual"
|
||||
? {
|
||||
@@ -1034,7 +1059,7 @@ return {
|
||||
|
||||
// Publish Docker only after the platform's H3 image/command has been validated.
|
||||
runModes: (s) =>
|
||||
["mi300x", "mi355x", "gb200", "gb300"].includes(s.hw)
|
||||
["mi300x", "mi355x", "gb200", "gb300", "xeon"].includes(s.hw)
|
||||
? ["python"]
|
||||
: ["docker", "python"],
|
||||
|
||||
|
||||
@@ -1,22 +1,30 @@
|
||||
export const config = (() => {
|
||||
const modelOf = (s) => "OpenMOSS-Team/MOVA-" + s.resolution;
|
||||
const plan = (s) => ({
|
||||
flags: [
|
||||
"--model-path " + modelOf(s),
|
||||
"--host 0.0.0.0",
|
||||
"--port {{PORT}}",
|
||||
"--adjust-frames false",
|
||||
"--num-gpus 8",
|
||||
"--ring-degree 2",
|
||||
"--ulysses-degree 4",
|
||||
"--tp 1",
|
||||
"--enable-torch-compile",
|
||||
"--save-output",
|
||||
'--output-dir "$SG_OUTPUT_DIR"',
|
||||
],
|
||||
gpus: 8,
|
||||
topology: { tp_size: 1, ulysses_degree: 4, ring_degree: 2 },
|
||||
});
|
||||
const plan = (s) =>
|
||||
s.hw === "xeon"
|
||||
? {
|
||||
flags: ["--model-path " + modelOf(s)],
|
||||
gpus: 1,
|
||||
topology: { tp_size: 1, ulysses_degree: 1, ring_degree: 1 },
|
||||
env: ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"],
|
||||
}
|
||||
: {
|
||||
flags: [
|
||||
"--model-path " + modelOf(s),
|
||||
"--host 0.0.0.0",
|
||||
"--port {{PORT}}",
|
||||
"--adjust-frames false",
|
||||
"--num-gpus 8",
|
||||
"--ring-degree 2",
|
||||
"--ulysses-degree 4",
|
||||
"--tp 1",
|
||||
"--enable-torch-compile",
|
||||
"--save-output",
|
||||
'--output-dir "$SG_OUTPUT_DIR"',
|
||||
],
|
||||
gpus: 8,
|
||||
topology: { tp_size: 1, ulysses_degree: 4, ring_degree: 2 },
|
||||
};
|
||||
const defaults = {
|
||||
hw: "b200",
|
||||
nodes: 1,
|
||||
@@ -29,8 +37,11 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "MOVA",
|
||||
supportedHardware: ["b200", "h200", "h100", "a100"],
|
||||
hardware: [{ id: "a100", label: "A100", vram: "", vendor: "nvidia" }],
|
||||
supportedHardware: ["b200", "h200", "h100", "a100", "xeon"],
|
||||
hardware: [
|
||||
{ id: "a100", label: "A100", vram: "", vendor: "nvidia" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
overlayDims: [
|
||||
@@ -48,8 +59,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } },
|
||||
verifiedRecipes: ["b200", "h200", "h100", "a100"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 8, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["b200", "h200", "h100", "a100", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
export const config = (() => {
|
||||
const modelOf = () => "Qwen/Qwen-Image-Edit-2511";
|
||||
const plan = (s) => ({
|
||||
flags: ["--model-path " + modelOf(s), "--ulysses-degree=1", "--ring-degree=1"],
|
||||
flags: ["--model-path " + modelOf(s), ...(s.hw === "xeon" ? [] : ["--ulysses-degree=1", "--ring-degree=1"])],
|
||||
gpus: 1,
|
||||
topology: { tp_size: 1, ulysses_degree: 1, ring_degree: 1 },
|
||||
env: s.hw === "xeon" ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
});
|
||||
const defaults = {
|
||||
hw: "b200",
|
||||
@@ -16,16 +17,19 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "Qwen-Image-Edit",
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x"],
|
||||
hardware: [],
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "xeon"],
|
||||
hardware: [{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" }],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
overlayDims: [],
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 1 } },
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 1, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
@@ -7,11 +7,12 @@ export const config = (() => {
|
||||
const flags = ["--model-path " + modelOf(s)];
|
||||
if (s.hw === "a2") flags.push("--num-gpus 1");
|
||||
else if (s.hw === "a3") flags.push("--tp-size 1", "--sp-degree 2", "--num-gpus 2");
|
||||
else flags.push("--ulysses-degree=1", "--ring-degree=1");
|
||||
else if (s.hw !== "xeon") flags.push("--ulysses-degree=1", "--ring-degree=1");
|
||||
return {
|
||||
flags,
|
||||
gpus: s.hw === "a3" ? 2 : 1,
|
||||
topology: { tp_size: 1, ulysses_degree: s.hw === "a3" ? 2 : 1, ring_degree: 1 },
|
||||
env: s.hw === "xeon" ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
warnings: s.precision === "nvfp4" ? ["NVFP4 is approximate; evaluate image quality against BF16."] : [],
|
||||
};
|
||||
};
|
||||
@@ -27,10 +28,11 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "Qwen-Image",
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3"],
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "xeon"],
|
||||
hardware: [
|
||||
{ id: "a2", label: "Ascend A2", vram: "", vendor: "npu" },
|
||||
{ id: "a3", label: "Ascend A3", vram: "2 NPUs / card", vendor: "npu" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
@@ -49,8 +51,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } },
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 8, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
export const config = {
|
||||
modelName: "Qwen3.8-27B",
|
||||
|
||||
supportedHardware: ["h200", "rtx6000", "rtx5090", "dgx-spark", "gb300"],
|
||||
supportedHardware: ["h200", "rtx6000", "rtx5090", "dgx-spark", "gb300", "xeon"],
|
||||
|
||||
// RTX PRO 6000 and RTX 5090 (SM120 / Blackwell Desktop) are workstation and
|
||||
// consumer cards, not datacenter GPUs, so they are not in the shared catalog.
|
||||
@@ -25,6 +25,7 @@ export const config = {
|
||||
hardware: [
|
||||
{ id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "blackwell" },
|
||||
{ id: "rtx5090", label: "RTX 5090", vram: "32GB", vendor: "blackwell" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
|
||||
// Every cell pins `--kv-cache-dtype fp8_e4m3` at the maintainers' direction
|
||||
@@ -78,9 +79,11 @@ export const config = {
|
||||
id: "eagle", label: "EAGLE",
|
||||
// In-checkpoint MTP head; the only availability constraint is the
|
||||
// 32GB RTX 5090, where it needs the NVFP4 weights to leave room.
|
||||
disabled: (sel) => sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4"),
|
||||
disableReason:
|
||||
"On the 32GB RTX 5090 the MTP head only fits on top of the NVFP4 weights",
|
||||
disabled: (sel) => sel.hw === "xeon" ||
|
||||
(sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4")),
|
||||
disableReason: (sel) => sel.hw === "xeon"
|
||||
? "Xeon only supports no speculative decoding"
|
||||
: "On the 32GB RTX 5090 the MTP head only fits on top of the NVFP4 weights",
|
||||
// EAGLE and DSPARK need opposite mem-fraction corrections on the
|
||||
// 5090 (EAGLE starves the state pool at boot and wants it UP;
|
||||
// DSpark starves runtime activations and wants it DOWN), so each
|
||||
@@ -134,9 +137,11 @@ export const config = {
|
||||
// EAGLE. No --min-free-slots-delay: at --max-running-requests 1 it
|
||||
// is a strict no-op, and its real semantic (disable the delayer)
|
||||
// would silently bite anyone raising concurrency to 8+.
|
||||
disabled: (sel) => sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4"),
|
||||
disableReason:
|
||||
"On the 32GB RTX 5090 the DSpark draft model only fits on top of the NVFP4 weights",
|
||||
disabled: (sel) => sel.hw === "xeon" ||
|
||||
(sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4")),
|
||||
disableReason: (sel) => sel.hw === "xeon"
|
||||
? "Xeon only supports no speculative decoding"
|
||||
: "On the 32GB RTX 5090 the DSpark draft model only fits on top of the NVFP4 weights",
|
||||
stripPrefixes: (sel) =>
|
||||
sel.hw === "rtx5090"
|
||||
? ["--mem-fraction-static", "--mamba-full-memory-ratio",
|
||||
@@ -192,9 +197,11 @@ export const config = {
|
||||
// RTX PRO 6000 BF16/FP8 cells boot-and-serve). The platforms where
|
||||
// it has not been exercised carry verificationStatus "in-progress"
|
||||
// on their cells.
|
||||
disabled: (sel) => sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4"),
|
||||
disableReason:
|
||||
"On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights",
|
||||
disabled: (sel) => sel.hw === "xeon" ||
|
||||
(sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4")),
|
||||
disableReason: (sel) => sel.hw === "xeon"
|
||||
? "Xeon only supports no speculative decoding"
|
||||
: "On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights",
|
||||
// fp32 needs the balanced ratio overridden, so that family is
|
||||
// stripped as well and re-emitted below.
|
||||
stripPrefixes: (sel) =>
|
||||
@@ -253,8 +260,12 @@ export const config = {
|
||||
stripPrefixes: ["--mamba-radix-cache-strategy"],
|
||||
options: [
|
||||
{ id: "low-latency", label: "Low-Latency",
|
||||
disabled: (sel) => sel.hw === "xeon",
|
||||
disableReason: "Xeon does not support the GDN radix-cache serving strategy",
|
||||
flags: ["--mamba-radix-cache-strategy extra_buffer"] },
|
||||
{ id: "high-throughput", label: "High-Throughput",
|
||||
disabled: (sel) => sel.hw === "xeon",
|
||||
disableReason: "Xeon does not support the GDN radix-cache serving strategy",
|
||||
flags: ["--mamba-radix-cache-strategy extra_buffer_lazy"] },
|
||||
],
|
||||
},
|
||||
@@ -302,22 +313,25 @@ export const config = {
|
||||
// request's KV never coexist -- buying the fifth slot cuts KV to
|
||||
// 7,752 tokens against the 9,216 one 8192/1024 request needs, and
|
||||
// generation stops after a single token.
|
||||
disabled: (sel) =>
|
||||
sel.hw === "rtx5090" &&
|
||||
disabled: (sel) => sel.hw === "xeon" ||
|
||||
(sel.hw === "rtx5090" &&
|
||||
(sel.quant === "nvfp4-bf16-head"
|
||||
? sel.spec === "dflash" || sel.spec === "dspark"
|
||||
: sel.spec === "dflash" && sel.tier === "low-latency"),
|
||||
disableReason:
|
||||
"On the 32GB RTX 5090 this combination has no fp32 GDN state pool that " +
|
||||
"also leaves room for prefill graph capture — use bfloat16",
|
||||
: sel.spec === "dflash" && sel.tier === "low-latency")),
|
||||
disableReason: (sel) => sel.hw === "xeon"
|
||||
? "Xeon does not support selecting the Mamba SSM dtype"
|
||||
: "On the 32GB RTX 5090 this combination has no fp32 GDN state pool that " +
|
||||
"also leaves room for prefill graph capture — use bfloat16",
|
||||
flags: ["--mamba-ssm-dtype float32"],
|
||||
},
|
||||
{
|
||||
id: "bfloat16", label: "bfloat16",
|
||||
disabled: (sel) => sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4"),
|
||||
disableReason:
|
||||
"On the 32GB RTX 5090 the bf16 GDN state pool is only a live choice for NVFP4; " +
|
||||
"the BF16 and FP8 checkpoints have no serviceable cell on this card",
|
||||
disabled: (sel) => sel.hw === "xeon" ||
|
||||
(sel.hw === "rtx5090" && !String(sel.quant).startsWith("nvfp4")),
|
||||
disableReason: (sel) => sel.hw === "xeon"
|
||||
? "Xeon does not support selecting the Mamba SSM dtype"
|
||||
: "On the 32GB RTX 5090 the bf16 GDN state pool is only a live choice for NVFP4; " +
|
||||
"the BF16 and FP8 checkpoints have no serviceable cell on this card",
|
||||
flags: ["--mamba-ssm-dtype bfloat16"],
|
||||
},
|
||||
],
|
||||
@@ -371,6 +385,8 @@ export const config = {
|
||||
["gsm8k_pct", "GSM8K", "%"],
|
||||
],
|
||||
|
||||
runModes: (s) => s.hw === "xeon" ? ["python"] : ["python", "docker"],
|
||||
|
||||
dockerImages: {
|
||||
h200: "lmsysorg/sglang:latest",
|
||||
rtx6000: "lmsysorg/sglang:latest",
|
||||
@@ -1002,5 +1018,25 @@ export const config = {
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
// Xeon CPU: reference recipe only, not benchmarked on this page. No
|
||||
// NVFP4/FP8 cell — those checkpoints have no verified CPU recipe.
|
||||
match: { hw: "xeon", variant: "default", quant: "bf16", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "unverified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--device cpu",
|
||||
"--disable-overlap-schedule",
|
||||
"--tp 6",
|
||||
"--mem-fraction-static 0.8",
|
||||
"--reasoning-parser qwen3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
@@ -4,11 +4,12 @@ export const config = (() => {
|
||||
const flags = ["--model-path " + modelOf(s)];
|
||||
if (s.hw === "a2") flags.push("--num-gpus 1");
|
||||
else if (s.hw === "a3") flags.push("--tp-size 2", "--sp-degree 1", "--num-gpus 2");
|
||||
else if (s.hw !== "arc_b") flags.push("--ulysses-degree=1", "--ring-degree=1");
|
||||
else if (s.hw !== "arc_b" && s.hw !== "xeon") flags.push("--ulysses-degree=1", "--ring-degree=1");
|
||||
return {
|
||||
flags,
|
||||
gpus: s.hw === "a3" ? 2 : 1,
|
||||
topology: { tp_size: s.hw === "a3" ? 2 : 1, ulysses_degree: 1, ring_degree: 1 },
|
||||
env: s.hw === "xeon" ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
};
|
||||
};
|
||||
const defaults = {
|
||||
@@ -22,11 +23,12 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "Z-Image-Turbo",
|
||||
supportedHardware: ["mi300x", "mi325x", "mi355x", "b200", "h200", "h100", "a2", "a3", "arc_b"],
|
||||
supportedHardware: ["mi300x", "mi325x", "mi355x", "b200", "h200", "h100", "a2", "a3", "arc_b", "xeon"],
|
||||
hardware: [
|
||||
{ id: "a2", label: "Ascend A2", vram: "", vendor: "npu" },
|
||||
{ id: "a3", label: "Ascend A3", vram: "2 NPUs / card", vendor: "npu" },
|
||||
{ id: "arc_b", label: "Intel BMG", vram: "", vendor: "intel" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
@@ -34,8 +36,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 2 } },
|
||||
verifiedRecipes: ["mi300x", "mi325x", "mi355x", "b200", "h200", "h100", "a2", "a3", "arc_b"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 2, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["mi300x", "mi325x", "mi355x", "b200", "h200", "h100", "a2", "a3", "arc_b", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
@@ -8,6 +8,7 @@ export const config = (() => {
|
||||
const plan = (s) => {
|
||||
const small = !s.variant.endsWith("14b");
|
||||
const ascend = ["a2", "a3"].includes(s.hw);
|
||||
const xeon = s.hw === "xeon";
|
||||
const fast = ascend ? Number(s.gpus_per_node) === 8 : Number(s.gpus_per_node) > 1;
|
||||
let gpus = 1,
|
||||
tp = 1,
|
||||
@@ -22,6 +23,9 @@ export const config = (() => {
|
||||
flags.push("--tp-size " + tp, "--sp-degree " + sp, "--num-gpus " + gpus);
|
||||
}
|
||||
if (fast) flags.push("--attention-backend laser_attn");
|
||||
} else if (xeon) {
|
||||
tp = 4;
|
||||
flags.push("--tp-size 4");
|
||||
} else {
|
||||
flags.push("--dit-layerwise-offload true");
|
||||
if (fast) {
|
||||
@@ -34,11 +38,12 @@ export const config = (() => {
|
||||
"t2v-14b": "NIVEDAN/wan2.1-lora",
|
||||
"i2v-14b": "valiantcat/Wan2.1-Fight-LoRA",
|
||||
}[s.variant];
|
||||
if (s.lora === s.variant && lora) flags.push("--lora-path " + lora);
|
||||
if (!xeon && s.lora === s.variant && lora) flags.push("--lora-path " + lora);
|
||||
return {
|
||||
flags,
|
||||
gpus,
|
||||
topology: { tp_size: tp, ulysses_degree: sp, ring_degree: 1 },
|
||||
env: xeon ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
warnings: ascend ? ["Ascend recipes count NPU chips; one A3 card has two chips."] : [],
|
||||
};
|
||||
};
|
||||
@@ -55,10 +60,11 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "Wan2.1",
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3"],
|
||||
supportedHardware: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "xeon"],
|
||||
hardware: [
|
||||
{ id: "a2", label: "Ascend A2", vram: "", vendor: "npu" },
|
||||
{ id: "a3", label: "Ascend A3", vram: "2 NPUs / card", vendor: "npu" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
@@ -87,6 +93,8 @@ export const config = (() => {
|
||||
showWhen: function anonymous(s) {
|
||||
return s.variant === "t2v-14b";
|
||||
},
|
||||
disabled: (s) => s.hw === "xeon",
|
||||
disableReason: "LoRA is not supported on Intel Xeon CPUs.",
|
||||
},
|
||||
{
|
||||
id: "i2v-14b",
|
||||
@@ -94,6 +102,8 @@ export const config = (() => {
|
||||
showWhen: function anonymous(s) {
|
||||
return s.variant === "i2v-14b";
|
||||
},
|
||||
disabled: (s) => s.hw === "xeon",
|
||||
disableReason: "LoRA is not supported on Intel Xeon CPUs.",
|
||||
},
|
||||
],
|
||||
},
|
||||
@@ -101,8 +111,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } },
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 8, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["b200", "b300", "h200", "h100", "mi300x", "mi325x", "mi355x", "a2", "a3", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
@@ -8,6 +8,7 @@ export const config = (() => {
|
||||
const plan = (s) => {
|
||||
const small = !s.variant.endsWith("14b");
|
||||
const ascend = ["a2", "a3"].includes(s.hw);
|
||||
const xeon = s.hw === "xeon";
|
||||
const fast = ascend ? Number(s.gpus_per_node) === 8 : Number(s.gpus_per_node) > 1;
|
||||
let gpus = 1,
|
||||
tp = 1,
|
||||
@@ -27,6 +28,9 @@ export const config = (() => {
|
||||
flags.push("--tp-size 2", "--sp-degree " + sp, "--num-gpus " + gpus);
|
||||
}
|
||||
if (fast) flags.push("--attention-backend laser_attn");
|
||||
} else if (xeon) {
|
||||
tp = 4;
|
||||
flags.push("--tp-size 4");
|
||||
} else {
|
||||
flags.push("--dit-layerwise-offload true");
|
||||
if (fast) {
|
||||
@@ -46,11 +50,12 @@ export const config = (() => {
|
||||
"t2v-14b": "Cseti/wan2.2-14B-Arcane_Jinx-lora-v1",
|
||||
"i2v-14b": "lightx2v/Wan2.2-Distill-Loras",
|
||||
}[s.variant];
|
||||
if (s.lora === s.variant && lora) flags.push("--lora-path " + lora);
|
||||
if (!xeon && s.lora === s.variant && lora) flags.push("--lora-path " + lora);
|
||||
return {
|
||||
flags,
|
||||
gpus,
|
||||
topology: { tp_size: tp, ulysses_degree: sp, ring_degree: 1 },
|
||||
env: xeon ? ["SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu"] : [],
|
||||
warnings: ascend ? ["Ascend recipes count NPU chips; one A3 card has two chips."] : [],
|
||||
};
|
||||
};
|
||||
@@ -67,10 +72,11 @@ export const config = (() => {
|
||||
};
|
||||
const config = {
|
||||
modelName: "Wan2.2",
|
||||
supportedHardware: ["b200", "b300", "h200", "mi300x", "mi325x", "mi355x", "a2", "a3"],
|
||||
supportedHardware: ["b200", "b300", "h200", "mi300x", "mi325x", "mi355x", "a2", "a3", "xeon"],
|
||||
hardware: [
|
||||
{ id: "a2", label: "Ascend A2", vram: "", vendor: "npu" },
|
||||
{ id: "a3", label: "Ascend A3", vram: "2 NPUs / card", vendor: "npu" },
|
||||
{ id: "xeon", label: "XEON", vram: "CPU", vendor: "cpu" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
@@ -99,6 +105,8 @@ export const config = (() => {
|
||||
showWhen: function anonymous(s) {
|
||||
return s.variant === "t2v-14b";
|
||||
},
|
||||
disabled: (s) => s.hw === "xeon",
|
||||
disableReason: "LoRA is not supported on Intel Xeon CPUs.",
|
||||
},
|
||||
{
|
||||
id: "i2v-14b",
|
||||
@@ -106,6 +114,8 @@ export const config = (() => {
|
||||
showWhen: function anonymous(s) {
|
||||
return s.variant === "i2v-14b";
|
||||
},
|
||||
disabled: (s) => s.hw === "xeon",
|
||||
disableReason: "LoRA is not supported on Intel Xeon CPUs.",
|
||||
},
|
||||
],
|
||||
},
|
||||
@@ -113,8 +123,11 @@ export const config = (() => {
|
||||
commandBuilder: {
|
||||
defaultSelection: defaults,
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } },
|
||||
verifiedRecipes: ["b200", "b300", "h200", "mi300x", "mi325x", "mi355x", "a2", "a3"].map((hw) => {
|
||||
limits: {
|
||||
nodes: { min: 1, max: 1 },
|
||||
gpus_per_node: { min: 1, max: 8, disabledWhen: (s) => s.hw === "xeon", disabledValue: 1 },
|
||||
},
|
||||
verifiedRecipes: ["b200", "b300", "h200", "mi300x", "mi325x", "mi355x", "a2", "a3", "xeon"].map((hw) => {
|
||||
const s = { ...defaults, hw };
|
||||
const p = plan(s);
|
||||
return {
|
||||
|
||||
Reference in New Issue
Block a user