[Doc] cookbook: Add Nemotron 3.5 Super VL (#44510)
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net>
This commit is contained in:
co-authored by
Jimmy Shong
parent
28b98e5d13
commit
05cbe241be
@@ -1,7 +1,6 @@
|
||||
---
|
||||
title: Nemotron3.5-Lightning
|
||||
description: "Deploy NVIDIA Nemotron 3.5 Lightning with SGLang — NVFP4 serving with MTP, DFlash, and DSpark speculative decoding, reasoning, and tool calling."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: NVIDIA Nemotron 3.5 Super VL
|
||||
description: "Deploy Nemotron 3.5 Super VL 120B-A12B with SGLang on B200, H200, RTX PRO 6000, and DGX Spark, with NVFP4 or BF16 and optional MTP."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
The **Docker** commands use `lmsysorg/sglang:nemotron3-5-super`. The command panel selects either `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` or `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`. Set `HF_TOKEN` in the panel's Env control if the checkpoint requires access.
|
||||
|
||||
Choose **B200, H200, RTX PRO 6000, or DGX Spark** on one node. B200 offers NVFP4 and BF16. H200 offers BF16; NVFP4 is greyed out. RTX PRO 6000 and DGX Spark offer NVFP4; BF16 is greyed out. Choose `bfloat16` or `float32` in **Mamba SSM DType**; `bfloat16` is the default. **High Throughput** leaves MTP off. **Low Latency** adds EAGLE MTP with top-k 1, three steps, and four draft tokens. These names identify configuration choices; no throughput or latency comparison was measured. Both command tabs use `sglang serve`.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/nvidia/nemotron-3.5-super-vl.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/nvidia/nemotron-3.5-super-vl-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground inherits the selected recipe and lets you toggle the reasoning and tool-call parsers. Parser changes are separate experiments.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model introduction
|
||||
|
||||
**NVIDIA Nemotron 3.5 Super VL 120B-A12B** is a vision-language checkpoint in the Nemotron-H hybrid MoE family. The early-access NGC checkpoint configuration identifies `NemotronH_Omni_Reasoning_V3` and includes a vision encoder. The command panel uses the [NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4) and [BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) Hugging Face IDs; the source runs used the [NGC model](https://registry.ngc.nvidia.com/orgs/0767305323357365/teams/super3-5-vl-ea-0911/models/nemotron-3.5-super-vl-120b-a12b-ea20261008). Check the checkpoint pages for access terms and model details.
|
||||
|
||||
The SGLang model entry point is [`nano_nemotron_vl.py` on the `nemotron3-super-dev-20261015` branch](https://github.com/sgl-project/sglang/blob/nemotron3-super-dev-20261015/python/sglang/srt/models/nano_nemotron_vl.py), with its supporting kernels and runtime code on the same branch.
|
||||
|
||||
The validation used text-only GSM8K prompts. It does **not** establish image or video request correctness, long-context behavior, or tool-call accuracy.
|
||||
|
||||
## 2. Configuration tips and validation
|
||||
|
||||
- **B200, NVFP4 or BF16:** both strategies use one eight-GPU TP8 node with FP8 E4M3 KV cache and 16,384-token configured context. Select the Mamba state dtype in the wizard; SGLang selects the attention backend and memory limits by default.
|
||||
- **H200, BF16:** both strategies use one eight-GPU TP8 node with FA3 attention, Triton MoE, and `--disable-flashinfer-autotune`.
|
||||
- **RTX PRO 6000, NVFP4:** both strategies served at TP1 on GPU 1 of an eight-GPU G4 machine. The configured context was 262,144 tokens; only short prompts were tested.
|
||||
- **DGX Spark, NVFP4:** both strategies completed 300-question text-only GSM8K runs at TP1 with `bfloat16` Mamba state. The `float32` choice has no completed accuracy run.
|
||||
- **KV precision:** these recipes deliberately use `--kv-cache-dtype fp8_e4m3` to match the validated runs. KV precision can affect output quality; no BF16-KV comparison was measured.
|
||||
|
||||
Server information confirmed `speculative_algorithm: null` for High Throughput and `EAGLE` for Low Latency. The runs used installed early-access NGC weights, not the Hugging Face checkpoint IDs in the command panel or the public Docker image.
|
||||
|
||||
The Deployment panel’s Benchmark card shows text-only GSM8K accuracy for measured selections and a pending state for unmeasured selections. The existing green Verified status marks a recipe with a completed source-run. These results do not validate the public checkpoint IDs or Docker image.
|
||||
|
||||
## 3. Reasoning output
|
||||
|
||||
The base recipes enable `--reasoning-parser nemotron_3`. In the tested chat completions, the thinking trace appeared in `message.reasoning_content` and the final answer in `message.content`. The tool-call parser is configured as `qwen3_coder`, but structured tool calls were not part of this accuracy check.
|
||||
|
||||
<Accordion title="Reasoning example (Python client)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
|
||||
model = client.models.list().data[0].id
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Claire makes a 3 egg omelet every morning for breakfast. How many dozens of eggs will she eat in 4 weeks?\nSolve the problem and put the final numeric answer after ####.",
|
||||
}],
|
||||
)
|
||||
message = response.choices[0].message
|
||||
print("Reasoning:", message.reasoning_content)
|
||||
print("Content:", message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example output from the B200 NVFP4 smoke run">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user wants me to solve a math problem.
|
||||
|
||||
Claire makes a 3 egg omelet every morning.
|
||||
She eats 3 eggs per day.
|
||||
4 weeks = 4 × 7 = 28 days.
|
||||
Total eggs = 3 × 28 = 84 eggs.
|
||||
Dozens of eggs = 84 / 12 = 7 dozens.
|
||||
|
||||
Final answer: 7.
|
||||
|
||||
I need to put the final numeric answer after ####.
|
||||
Content: Claire eats 3 eggs per day.
|
||||
In 4 weeks, there are 4 × 7 = 28 days.
|
||||
Total eggs eaten: 3 × 28 = 84 eggs.
|
||||
Since 1 dozen = 12 eggs: 84 ÷ 12 = 7 dozens.
|
||||
|
||||
#### 7
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -106,7 +106,7 @@ metatags:
|
||||
<Card
|
||||
title="NVIDIA"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/NVIDIA/Nemotron3.5-Lightning"
|
||||
href="/cookbook/autoregressive/NVIDIA/Nemotron3.5-Super"
|
||||
img="/cards/logos/nvidia.png"
|
||||
/>
|
||||
<Card
|
||||
|
||||
@@ -1372,6 +1372,7 @@
|
||||
{
|
||||
"group": "NVIDIA",
|
||||
"pages": [
|
||||
"cookbook/autoregressive/NVIDIA/Nemotron3.5-Super",
|
||||
"cookbook/autoregressive/NVIDIA/Nemotron3.5-Lightning",
|
||||
"cookbook/autoregressive/NVIDIA/Nemotron3-Ultra",
|
||||
"cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni",
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
// Text-only GSM8K scores from completed sgl-eval==0.1.2 runs.
|
||||
// Accuracy verification applies to these NGC-weight source runs, not to the
|
||||
// public Hugging Face checkpoint IDs or the planned cookbook Docker tag.
|
||||
export const benchmarks = [
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 95.98 },
|
||||
notes: "1,266/1,319 correct; 20 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.33 },
|
||||
notes: "292/300 correct on a smaller cohort; three outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.13 },
|
||||
notes: "1,268/1,319 correct; 16 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.33 },
|
||||
notes: "292/300 correct on a smaller cohort; four outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 95.68 },
|
||||
notes: "1,262/1,319 correct; 25 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.67 },
|
||||
notes: "293/300 correct on a smaller cohort; five outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.44 },
|
||||
notes: "1,272/1,319 correct; 21 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.33 },
|
||||
notes: "292/300 correct on a smaller cohort; four outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.06 },
|
||||
notes: "1,267/1,319 correct; 21 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.67 },
|
||||
notes: "293/300 correct on a smaller cohort; four outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 95.83 },
|
||||
notes: "1,264/1,319 correct; 21 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.33 },
|
||||
notes: "292/300 correct on a smaller cohort; two outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "rtx-pro-6000", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.06 },
|
||||
notes: "1,267/1,319 correct; 21 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "rtx-pro-6000", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 97.0 },
|
||||
notes: "291/300 correct on a smaller cohort; five outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "rtx-pro-6000", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.44 },
|
||||
notes: "1,272/1,319 correct; 18 outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "rtx-pro-6000", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single", mambaDtype: "float32" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.33 },
|
||||
notes: "289/300 correct on a smaller cohort; four outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 96.67 },
|
||||
notes: "290/300 correct on a smaller cohort; five outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
{
|
||||
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single", mambaDtype: "bfloat16" },
|
||||
sglang_version: "dev @ 2c7511f6f7cf",
|
||||
accuracy: { gsm8k_pct: 98.0 },
|
||||
notes: "294/300 correct on a smaller cohort; three outputs hit the 8,192-token cap; zero request errors. Text-only NGC checkpoint run.",
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,328 @@
|
||||
// Instantiated from the cookbook-add-model config template. The non-Spark
|
||||
// source recipes were exercised on dev @ 2c7511f6f7cf1cdf9ae568ad0fb25f291edb10da.
|
||||
// Public HF IDs and the planned Docker image still need verification.
|
||||
|
||||
export const config = {
|
||||
modelName: "Nemotron 3.5 Super VL",
|
||||
|
||||
supportedHardware: ["b200", "h200", "rtx-pro-6000", "dgx-spark"],
|
||||
hardware: [
|
||||
{ id: "rtx-pro-6000", label: "RTX PRO 6000", vram: "96GB", vendor: "nvidia" },
|
||||
],
|
||||
|
||||
variants: [{ id: "default", label: "Default" }],
|
||||
quantizations: [
|
||||
{
|
||||
id: "nvfp4",
|
||||
label: "NVFP4",
|
||||
disabled: (sel) => sel.hw === "h200",
|
||||
disableReason: "NVFP4 is not available for H200 in this recipe.",
|
||||
},
|
||||
{ id: "bf16", label: "BF16" },
|
||||
],
|
||||
strategies: [
|
||||
{ id: "high-throughput", label: "High Throughput" },
|
||||
{ id: "low-latency", label: "Low Latency" },
|
||||
],
|
||||
nodesOptions: [{ id: "single", label: "Single node" }],
|
||||
overlayDims: [
|
||||
{
|
||||
id: "mambaDtype",
|
||||
title: "Mamba SSM DType",
|
||||
default: "bfloat16",
|
||||
options: [
|
||||
{
|
||||
id: "bfloat16",
|
||||
label: "bfloat16",
|
||||
stripPrefixes: ["--mamba-ssm-dtype"],
|
||||
flags: ["--mamba-ssm-dtype bfloat16"],
|
||||
},
|
||||
{
|
||||
id: "float32",
|
||||
label: "float32",
|
||||
soft: (sel) => sel.hw === "dgx-spark",
|
||||
softReason: "The 300-question GSM8K run was not completed on DGX Spark.",
|
||||
stripPrefixes: ["--mamba-ssm-dtype"],
|
||||
flags: ["--mamba-ssm-dtype float32"],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
runModes: ["python", "docker"],
|
||||
|
||||
accuracyLabels: [["gsm8k_pct", "GSM8K (text-only)", "%"]],
|
||||
|
||||
modelNames: {
|
||||
"default|nvfp4": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4",
|
||||
"default|bf16": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16",
|
||||
},
|
||||
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
|
||||
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||
},
|
||||
|
||||
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
|
||||
|
||||
dockerImages: {
|
||||
b200: "lmsysorg/sglang:nemotron3-5-super",
|
||||
h200: "lmsysorg/sglang:nemotron3-5-super",
|
||||
"rtx-pro-6000": "lmsysorg/sglang:nemotron3-5-super",
|
||||
"dgx-spark": "lmsysorg/sglang:nemotron3-5-super",
|
||||
},
|
||||
|
||||
github: {
|
||||
cookbookModel: "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4",
|
||||
},
|
||||
|
||||
playgroundFeatures: {
|
||||
parsers: {
|
||||
items: [
|
||||
{ id: "reasoning", label: "Reasoning parser", flag: "--reasoning-parser nemotron_3" },
|
||||
{ id: "toolCall", label: "Tool-call parser", flag: "--tool-call-parser qwen3_coder" },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
cells: [
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--context-length 16384",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--context-length 16384",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--context-length 16384",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--context-length 16384",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--context-length 16384",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--moe-runner-backend triton",
|
||||
"--disable-flashinfer-autotune",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--context-length 16384",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--moe-runner-backend triton",
|
||||
"--disable-flashinfer-autotune",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "rtx-pro-6000", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 1",
|
||||
"--context-length 262144",
|
||||
"--attention-backend flashinfer",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--max-running-requests 2",
|
||||
"--mem-fraction-static 0.90",
|
||||
"--max-total-tokens 1016382",
|
||||
"--cuda-graph-max-bs-decode 2",
|
||||
"--cuda-graph-bs-decode 1 2",
|
||||
"--max-mamba-cache-size 25",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 16384",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "rtx-pro-6000", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||
verificationStatus: "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 1",
|
||||
"--context-length 262144",
|
||||
"--attention-backend flashinfer",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--max-running-requests 2",
|
||||
"--mem-fraction-static 0.90",
|
||||
"--max-total-tokens 1016382",
|
||||
"--cuda-graph-max-bs-decode 2",
|
||||
"--cuda-graph-bs-decode 1 2",
|
||||
"--max-mamba-cache-size 25",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 16384",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||
verificationStatus: (sel) => sel.mambaDtype === "float32" ? "unverified" : "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 1",
|
||||
"--context-length 16384",
|
||||
"--attention-backend flashinfer",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--max-running-requests 2",
|
||||
"--mem-fraction-static 0.88",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||
verificationStatus: (sel) => sel.mambaDtype === "float32" ? "unverified" : "verified",
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 1",
|
||||
"--context-length 16384",
|
||||
"--attention-backend flashinfer",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--max-running-requests 2",
|
||||
"--mem-fraction-static 0.88",
|
||||
"--chunked-prefill-size 1024",
|
||||
"--max-prefill-tokens 8192",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--reasoning-parser nemotron_3",
|
||||
"--tool-call-parser qwen3_coder",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
Reference in New Issue
Block a user