[Docs] Rename docs_new/ to docs/ (#32123)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-03 16:51:00 -07:00
parent c949e91f18
commit b819d2fb5b
491 changed files with 122 additions and 102 deletions
@@ -0,0 +1,654 @@
+---
+title: Nemotron 3 Nano Omni
+metatags:
+    description: "Deploy NVIDIA Nemotron 3 Nano Omni multimodal MoE model with SGLang - text, image, video, and audio inputs with reasoning and tool calling."
+---
+
+import { Nemotron3NanoOmniDeployment } from '/src/snippets/autoregressive/nemotron3-nano-omni-deployment.jsx';
+
+## 1. Model Introduction
+
+`NVIDIA Nemotron 3 Nano Omni` is a 30B-parameter hybrid MoE multimodal model that activates only 3B parameters per forward pass, combining vision and audio encoders into a unified architecture. Part of the Nemotron 3 family, it is designed to power multimodal sub-agents that perceive and reason across vision, audio, and language in a single inference loop — eliminating the fragmented stacks of separate models for each modality.
+
+Architecture and key features:
+
+- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture for efficient routing and sequence modeling.
+- **30B total / 3B active parameters:** Delivers strong multimodal accuracy at a fraction of the cost of dense models.
+- **1M token context window:** Sustains coherent agent state across extended multimodal workflows — screen history, document content, and audio context remain in view without re-ingestion.
+- **Unified vision and audio encoders:** One model replaces fragmented multimodal stacks; vision and audio perception happen in the same forward pass.
+- **3D Convolution (Conv3D):** Efficient temporal-spatial processing for video inputs.
+- **Efficient Video Sampling (EVS):** Enables longer video processing at the same compute budget via temporal-aware perception and adaptive frame sampling.
+- **FP8 and NVFP4 quantization:** FP8 supports deployment from workstation (RTX 6000, DGX Spark) to cloud (H100, H200, B200, A100, L40S); NVFP4 requires Blackwell hardware.
+- **9x higher throughput** than other open omni models at the same interactivity level.
+- **~20% higher multimodal intelligence** compared to the best open alternative.
+- **Post-trained with multi-environment reinforcement learning** via NVIDIA NeMo RL and NeMo Gym across text, image, audio, and video environments, improving instruction following and convergence to correct multimodal answers.
+
+**Modalities:** Input: text, image, video, audio — Output: text
+
+**Supported GPUs:** NVIDIA B200, H100, H200, A100, L40S, DGX Spark, RTX 6000
+
+Available model variants on HuggingFace:
+- [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16)
+- [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8)
+- [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4)
+
+**Agentic workloads this model enables:**
+- **Computer Use Agent:** Perception loop for agents navigating GUIs — reads screens, understands UI state over time, validates outcomes. Collapses vision and reasoning into a single loop.
+- **Document Intelligence:** Interprets documents, charts, tables, screenshots, and mixed media inputs for enterprise analysis and compliance workflows.
+- **Audio & Video Understanding Agents:** Maintains continuous audio-video context for customer service, research, and monitoring workflows, tying what was said, shown, and documented into a single reasoning stream.
+
+## 2. SGLang Installation
+
+Install SGLang via pip or from source:
+
+```shell Command
+# Install via pip
+pip install sglang
+
+# Or install from source
+uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
+
+# Or use Docker
+docker pull lmsysorg/sglang:latest
+```
+
+For the full Docker setup and other installation methods, refer to the [official SGLang installation guide](../../../docs/get-started/install).
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance tuning.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
+
+<Nemotron3NanoOmniDeployment />
+
+### 3.2 Configuration Tips
+
+- **Attention backend:**
+
+    **H100/H200:** Use flash attention 3 backend by default.
+    **B200:** Use flashinfer backend by default.
+
+- **TP support:**
+
+    To set tensor parallelism, use `--tp <1|2|4|8>`. A 4×H100 setup is recommended for the BF16 variant.
+
+- **FP8 KV cache:**
+
+    To enable FP8 KV cache, append `--kv-cache-dtype fp8_e4m3`. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
+
+- **Reasoning parser:**
+
+    Append `--reasoning-parser deepseek-r1` to enable structured reasoning traces (`reasoning_content` field in the response).
+
+- **Tool calling:**
+
+    Append `--tool-call-parser qwen3_coder` to enable tool calling support.
+
+## 4. Model Invocation
+
+The command below launches the server for a 4×H100 setup with reasoning and tool calling enabled. See [Section 4.8](#4-8-fp8-and-nvfp4-deployment) for FP8 and NVFP4 variants.
+
+```shell Command
+sglang serve \
+  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --tp 4 \
+  --trust-remote-code \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser deepseek-r1
+```
+
+### 4.1 Basic Usage (Text)
+
+SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
+
+```python Example
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
+    ],
+    temperature=0.6,
+    max_tokens=512,
+)
+print(resp.choices[0].message.reasoning_content, resp.choices[0].message.content)
+```
+
+Output:
+```text Output
+Reasoning: SGLang is a serving framework I know from my training data. Let me recall the key features...
+
+Content:
+- **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn and few-shot workloads.
+- **OpenAI-compatible API** — Drop-in replacement for the OpenAI Python client; no application code changes required to serve a locally-hosted model.
+- **High-throughput serving** — Continuous batching, chunked prefill, and optimized CUDA kernels deliver state-of-the-art throughput on NVIDIA GPUs across A100, H100, and B200.
+```
+
+Streaming chat completion:
+
+```python Example
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+stream = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "What are the first 5 prime numbers?"},
+    ],
+    temperature=0.6,
+    max_tokens=512,
+    stream=True,
+)
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if delta and delta.content:
+        print(delta.content, end="", flush=True)
+```
+
+### 4.2 Image Understanding
+
+Pass image inputs using the OpenAI vision format. Supports both URLs and base64-encoded images:
+
+```python Example
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+# From URL
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"},
+                },
+                {"type": "text", "text": "Describe this image in detail."},
+            ],
+        }
+    ],
+    temperature=0.6,
+    max_tokens=512,
+)
+print(resp.choices[0].message.reasoning_content)
+print(resp.choices[0].message.content)
+```
+
+For local images, encode as base64:
+
+```python Example
+import base64
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+with open("screenshot.png", "rb") as f:
+    image_b64 = base64.b64encode(f.read()).decode("utf-8")
+
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/png;base64,{image_b64}"},
+                },
+                {"type": "text", "text": "What UI elements are visible on this screen? What action would you take next?"},
+            ],
+        }
+    ],
+    temperature=0.6,
+    max_tokens=512,
+)
+print(resp.choices[0].message.content)
+```
+
+### 4.3 Video Understanding
+
+Nemotron 3 Nano Omni uses Conv3D layers and Efficient Video Sampling (EVS) for temporal-spatial video reasoning, processing longer videos at the same compute budget:
+
+```python Example
+import base64
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+with open("video.mp4", "rb") as f:
+    video_b64 = base64.b64encode(f.read()).decode("utf-8")
+
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "video_url",
+                    "video_url": {"url": f"data:video/mp4;base64,{video_b64}"},
+                },
+                {"type": "text", "text": "Summarize what happens in this video step by step."},
+            ],
+        }
+    ],
+    temperature=0.6,
+    max_tokens=1024,
+)
+print(resp.choices[0].message.reasoning_content)
+print(resp.choices[0].message.content)
+```
+
+### 4.4 Audio Understanding
+
+Pass audio inputs as base64-encoded WAV or MP3 data:
+
+```python Example
+import base64
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+with open("audio.wav", "rb") as f:
+    audio_b64 = base64.b64encode(f.read()).decode("utf-8")
+
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "input_audio",
+                    "input_audio": {"data": audio_b64, "format": "wav"},
+                },
+                {"type": "text", "text": "Transcribe and summarize what was said in this audio."},
+            ],
+        }
+    ],
+    temperature=0.6,
+    max_tokens=512,
+)
+print(resp.choices[0].message.content)
+```
+
+### 4.5 Mixed Multimodal Input
+
+Combine modalities in a single request. For example, an image alongside an audio question about it:
+
+```python Example
+import base64
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+with open("chart.png", "rb") as f:
+    image_b64 = base64.b64encode(f.read()).decode("utf-8")
+
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/png;base64,{image_b64}"},
+                },
+                {"type": "text", "text": "Analyze this chart. What are the key trends and what conclusion does the data support?"},
+            ],
+        }
+    ],
+    temperature=0.6,
+    max_tokens=1024,
+)
+print(resp.choices[0].message.reasoning_content)
+print(resp.choices[0].message.content)
+```
+
+### 4.6 Reasoning
+
+The model supports two modes — Reasoning ON (default) vs OFF. Toggle per-request by setting `enable_thinking` to `False`:
+
+```python Example
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+# Reasoning ON (default)
+print("Reasoning on")
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "What is the derivative of x^3 sin(x)?"},
+    ],
+    temperature=0.6,
+    max_tokens=1024,
+)
+print(f"Reasoning:\n{resp.choices[0].message.reasoning_content[:300]}...\nContent:\n{resp.choices[0].message.content}")
+print("\n")
+
+# Reasoning OFF
+print("Reasoning off")
+resp = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "What is 15% of 200?"},
+    ],
+    temperature=0.6,
+    max_tokens=256,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+)
+print(f"Content:\n{resp.choices[0].message.content}")
+```
+
+Output:
+```text Output
+Reasoning on
+Reasoning:
+The user wants the derivative of x^3 sin(x). I'll apply the product rule: d/dx[u·v] = u'v + uv'. Here u = x^3, v = sin(x). So u' = 3x^2, v' = cos(x). The result is 3x^2·sin(x) + x^3·cos(x)...
+Content:
+Using the product rule: d/dx[x³ sin(x)] = 3x² sin(x) + x³ cos(x)
+
+
+Reasoning off
+Content:
+15% of 200 is **30**.
+```
+
+### 4.7 Tool Calling
+
+Call functions using the OpenAI Tools schema. The server must be launched with `--tool-call-parser qwen3_coder`:
+
+```python Example
+from openai import OpenAI
+
+SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "City and state, e.g. San Francisco, CA",
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                    },
+                },
+                "required": ["location"],
+            },
+        },
+    }
+]
+
+completion = client.chat.completions.create(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "What is the weather like in Santa Clara, CA?"},
+    ],
+    tools=TOOLS,
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=512,
+    stream=False,
+)
+print(completion.choices[0].message.reasoning_content)
+print(completion.choices[0].message.tool_calls)
+```
+
+Output:
+```text Output
+The user is asking about weather in Santa Clara, CA. I have a get_weather function that takes a location and optional unit. I should call it with location="Santa Clara, CA".
+
+[ChatCompletionMessageFunctionToolCall(id='call_abc123', function=Function(arguments='{"location": "Santa Clara, CA", "unit": "fahrenheit"}', name='get_weather'), type='function', index=0)]
+```
+
+### 4.8 FP8 and NVFP4 Deployment
+
+**FP8 variant** (recommended for throughput-critical serving on H100/H200/B200):
+
+```shell Command
+sglang serve \
+  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --tp 4 \
+  --trust-remote-code \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser deepseek-r1
+```
+
+**NVFP4 variant** (maximum efficiency on Blackwell B200):
+
+```shell Command
+sglang serve \
+  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --tp 4 \
+  --trust-remote-code \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser deepseek-r1
+```
+
+---
+
+## 5. Benchmark
+
+### 5.1 Efficiency Benchmark
+
+Nemotron 3 Nano Omni achieves **9x higher throughput** than other open omni models at the same interactivity level, delivering lower cost and better scalability without sacrificing responsiveness. It also achieves **~20% higher multimodal intelligence** compared to the best open alternative across image, video, and audio reasoning tasks.
+
+### 5.2 Speed Benchmark
+
+**Test Environment:**
+- Hardware: B200 (8×)
+- Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
+  --trust-remote-code \
+  --tp 4 \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --attention-backend flashinfer \
+  --port 30000
+```
+
+Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  206.52
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total generated tokens:                  2087288
+Total generated tokens (retokenized):    1945477
+Request throughput (req/s):              19.83
+Input token throughput (tok/s):          10080.25
+Output token throughput (tok/s):         10107.18
+Peak output token throughput (tok/s):    20199.00
+Peak concurrent requests:                291
+Total token throughput (tok/s):          20187.44
+Concurrency:                             250.83
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   12646.47
+Median E2E Latency (ms):                 12371.84
+P90 E2E Latency (ms):                    22889.81
+P99 E2E Latency (ms):                    26528.70
+---------------Time to First Token----------------
+Mean TTFT (ms):                          220.66
+Median TTFT (ms):                        97.67
+P99 TTFT (ms):                           2068.63
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          24.98
+Median TPOT (ms):                        24.36
+P99 TPOT (ms):                           44.97
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           24.43
+Median ITL (ms):                         10.91
+P95 ITL (ms):                            62.68
+P99 ITL (ms):                            100.60
+Max ITL (ms):                            2171.93
+==================================================
+```
+
+### 5.3 Accuracy Benchmark
+
+**Environment**
+- Hardware: B200 (8×)
+- Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+**Launch Model**
+```shell Command
+sglang serve \
+  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
+  --trust-remote-code \
+  --tp 4 \
+  --attention-backend flashinfer \
+  --reasoning-parser deepseek-r1
+```
+
+#### 5.3.1 GSM8K Benchmark
+
+**Run Benchmark**
+```shell Command
+python3 benchmark/gsm8k/bench_sglang.py --port 30000
+```
+
+**Test Results:**
+```text Output
+Accuracy: 0.830
+Invalid: 0.000
+Latency: 13.970 s
+Output throughput: 1611.623 token/s
+```
+
+#### 5.3.2 MMLU Benchmark
+
+**Run Benchmark**
+```shell Command
+python3 benchmark/mmlu/bench_sglang.py --port 30000
+```
+
+**Test Results:**
+```text Output
+subject: abstract_algebra, #q:100, acc: 0.510
+subject: anatomy, #q:135, acc: 0.711
+subject: astronomy, #q:152, acc: 0.829
+subject: business_ethics, #q:100, acc: 0.760
+subject: clinical_knowledge, #q:265, acc: 0.781
+subject: college_biology, #q:144, acc: 0.854
+subject: college_chemistry, #q:100, acc: 0.560
+subject: college_computer_science, #q:100, acc: 0.700
+subject: college_mathematics, #q:100, acc: 0.590
+subject: college_medicine, #q:173, acc: 0.775
+subject: college_physics, #q:102, acc: 0.559
+subject: computer_security, #q:100, acc: 0.750
+subject: conceptual_physics, #q:235, acc: 0.821
+subject: econometrics, #q:114, acc: 0.605
+subject: electrical_engineering, #q:145, acc: 0.759
+subject: elementary_mathematics, #q:378, acc: 0.638
+subject: formal_logic, #q:126, acc: 0.524
+subject: global_facts, #q:100, acc: 0.400
+subject: high_school_biology, #q:310, acc: 0.906
+subject: high_school_chemistry, #q:203, acc: 0.759
+subject: high_school_computer_science, #q:100, acc: 0.860
+subject: high_school_european_history, #q:165, acc: 0.812
+subject: high_school_geography, #q:198, acc: 0.889
+subject: high_school_government_and_politics, #q:193, acc: 0.933
+subject: high_school_macroeconomics, #q:390, acc: 0.785
+subject: high_school_mathematics, #q:270, acc: 0.496
+subject: high_school_microeconomics, #q:238, acc: 0.887
+subject: high_school_physics, #q:151, acc: 0.675
+subject: high_school_psychology, #q:545, acc: 0.895
+subject: high_school_statistics, #q:216, acc: 0.731
+subject: high_school_us_history, #q:204, acc: 0.858
+subject: high_school_world_history, #q:237, acc: 0.873
+subject: human_aging, #q:223, acc: 0.740
+subject: human_sexuality, #q:131, acc: 0.855
+subject: international_law, #q:121, acc: 0.851
+subject: jurisprudence, #q:108, acc: 0.815
+subject: logical_fallacies, #q:163, acc: 0.847
+subject: machine_learning, #q:112, acc: 0.598
+subject: management, #q:103, acc: 0.864
+subject: marketing, #q:234, acc: 0.910
+subject: medical_genetics, #q:100, acc: 0.880
+subject: miscellaneous, #q:783, acc: 0.881
+subject: moral_disputes, #q:346, acc: 0.780
+subject: moral_scenarios, #q:895, acc: 0.543
+subject: nutrition, #q:306, acc: 0.814
+subject: philosophy, #q:311, acc: 0.733
+subject: prehistory, #q:324, acc: 0.852
+subject: professional_accounting, #q:282, acc: 0.553
+subject: professional_law, #q:1534, acc: 0.565
+subject: professional_medicine, #q:272, acc: 0.779
+subject: professional_psychology, #q:612, acc: 0.760
+subject: public_relations, #q:110, acc: 0.709
+subject: security_studies, #q:245, acc: 0.759
+subject: sociology, #q:201, acc: 0.831
+subject: us_foreign_policy, #q:100, acc: 0.910
+subject: virology, #q:166, acc: 0.560
+subject: world_religions, #q:171, acc: 0.807
+Total latency: 67.512
+Average accuracy: 0.737
+```
@@ -0,0 +1,375 @@
+---
+title: Nemotron3-Nano
+metatags:
+    description: "Deploy NVIDIA Nemotron3-Nano 30B hybrid LLM with SGLang - MoE, Mamba2, and attention layers with BF16/FP8 precision options."
+---
+
+import { Nemotron3NanoDeployment } from '/src/snippets/autoregressive/nemotron3-nano-deployment.jsx';
+
+## 1. Model Introduction
+
+`NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks.
+
+The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported.
+
+At a high level:
+
+- **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only.
+- **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth.
+- **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs.
+
+## 2. SGLang Installation
+
+Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install nightly wheel through:
+```bash Command
+uv pip install sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/
+```
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance tuning.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
+
+<Nemotron3NanoDeployment />
+
+### 3.2 Configuration Tips
+
+- **Attention backend**:
+
+    **H200**: Use flash attention 3 backend by default.
+    **B200**: Use flashinfer backend by default.
+
+- **TP support**:
+
+    To set tp size, use `--tp <1|2|4|8>`.
+
+- **FP8 KV cache**:
+
+    To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage (OpenAI-Compatible API)
+
+SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Summarize what MoE models are in 5 bullets."},
+    ],
+    temperature=0.7,
+    max_tokens=256,
+)
+
+print(resp.choices[0].message.content)
+
+```
+
+Streaming chat completion
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "What are the first 5 prime numbers?"}
+    ],
+    temperature=0.7,
+    max_tokens=1024,
+    stream=True,
+)
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if delta and delta.content:
+        print(delta.content, end="", flush=True)
+```
+
+### 4.2 Reasoning
+To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable_thinking to False, as shown below.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+# Reasoning on (default)
+print("Reasoning on")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs."}
+    ],
+    temperature=0.7,
+    max_tokens=512,
+)
+print(resp.choices[0].message.reasoning_content)
+
+# Reasoning off
+print("Reasoning off")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs."}
+    ],
+    temperature=0.6,
+    max_tokens=256,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+print(resp.choices[0].message.reasoning_content)
+
+```
+
+### 4.3 Tool calling
+To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool_calls.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+# Tool calling via OpenAI tools schema
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "calculate_tip",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "bill_total": {
+                        "type": "integer",
+                        "description": "The total amount of the bill"
+                    },
+                    "tip_percentage": {
+                        "type": "integer",
+                        "description": "The percentage of tip to be applied"
+                    }
+                },
+                "required": ["bill_total", "tip_percentage"]
+            }
+        }
+    }
+]
+
+completion = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": ""},
+        {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
+    ],
+    tools=TOOLS,
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=512,
+    stream=False
+)
+
+print(completion.choices[0].message.reasoning_content)
+print(completion.choices[0].message.tool_calls)
+```
+
+---
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU
+
+**FP8 variant**
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
+  --trust-remote-code \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  183.18
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total input vision tokens:               0
+Total generated tokens:                  2116125
+Total generated tokens (retokenized):    1076256
+Request throughput (req/s):              22.36
+Input token throughput (tok/s):          11364.25
+Output token throughput (tok/s):         11552.04
+Peak output token throughput (tok/s):    24692.00
+Peak concurrent requests:                294
+Total token throughput (tok/s):          22916.30
+Concurrency:                             251.19
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   11233.74
+Median E2E Latency (ms):                 11142.97
+---------------Time to First Token----------------
+Mean TTFT (ms):                          172.99
+Median TTFT (ms):                        116.57
+P99 TTFT (ms):                           1193.68
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          21.74
+Median TPOT (ms):                        21.14
+P99 TPOT (ms):                           41.12
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           21.45
+Median ITL (ms):                         9.06
+P95 ITL (ms):                            62.59
+P99 ITL (ms):                            110.83
+Max ITL (ms):                            5368.19
+==================================================
+```
+
+**BF16 variant**
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
+  --trust-remote-code \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  360.22
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total input vision tokens:               0
+Total generated tokens:                  2087288
+Total generated tokens (retokenized):    1940652
+Request throughput (req/s):              11.37
+Input token throughput (tok/s):          5779.10
+Output token throughput (tok/s):         5794.55
+Peak output token throughput (tok/s):    9169.00
+Peak concurrent requests:                276
+Total token throughput (tok/s):          11573.65
+Concurrency:                             249.76
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   21965.10
+Median E2E Latency (ms):                 21706.35
+---------------Time to First Token----------------
+Mean TTFT (ms):                          211.54
+Median TTFT (ms):                        93.06
+P99 TTFT (ms):                           2637.66
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          43.27
+Median TPOT (ms):                        43.04
+P99 TPOT (ms):                           61.15
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           42.77
+Median ITL (ms):                         28.46
+P95 ITL (ms):                            71.85
+P99 ITL (ms):                            113.20
+Max ITL (ms):                            5237.28
+==================================================
+
+```
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+**Environment**
+- Hardware: NVIDIA B200 GPU
+- Model: BF16 checkpoint
+
+**Launch Model**
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
+  --trust-remote-code \
+  --reasoning-parser nemotron_3
+```
+
+**Run Benchmark with lm-eval**
+```bash Command
+pip install lm-eval[api]==0.4.9.2
+
+lm_eval --model local-completions --tasks gsm8k --model_args "model=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=4,max_retries=3,tokenized_requests=False,max_lengths=16384" --gen_kwargs '{"chat_template_kwargs":{"thinking":true}}' --batch_size 256
+```
+
+**Test Results:**
+```text Output
+|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
+|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
+|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.5603|±  |0.0137|
+|     |       |strict-match    |     5|exact_match|↑  |0.8453|±  |0.0100|
+```
@@ -0,0 +1,571 @@
+---
+title: NVIDIA Nemotron3-Super
+metatags:
+    description: "Deploy NVIDIA Nemotron3-Super with SGLang - 120B hybrid MoE model (12B active) with 1M context window optimized for multi-agent systems and tool use."
+---
+
+import { Nemotron3SuperDeployment } from '/src/snippets/autoregressive/nemotron3-super-deployment.jsx';
+
+## 1. Model Introduction
+
+`NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step.
+
+Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows.
+
+Architecture and key features:
+
+- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
+- **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5).
+- **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text.
+- **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation.
+
+## 2. SGLang Installation
+
+SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker.
+
+```bash Command
+# Install from source
+uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
+
+# Or use Docker
+docker pull lmsysorg/sglang:latest
+```
+
+For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance tuning.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command.
+
+<Nemotron3SuperDeployment />
+
+### 3.2 Configuration Tips
+
+- **Attention backend**:
+
+    **H200**: Use flash attention 3 backend by default.
+    **B200**: Use flashinfer backend by default.
+
+- **TP support**:
+
+    To set tp size, use `--tp <2|4|8>`.
+
+- **FP8 KV cache**:
+
+    To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
+
+## 4. Model Invocation
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --host 0.0.0.0 \
+  --port 5000 \
+  --trust-remote-code \
+  --tp 4 \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser nemotron_3
+```
+
+### 4.1 Basic Usage (OpenAI-Compatible API)
+
+SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
+    ],
+    temperature=0.6,
+    max_tokens=1024,
+)
+print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
+print("\n")
+```
+
+Output:
+```text Output
+Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others.
+
+First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens)
+
+Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens.
+- It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput.
+- Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers.
+```
+
+Streaming chat completion:
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "What are the first 5 prime numbers?"}
+    ],
+    temperature=0.7,
+    max_tokens=1024,
+    stream=True,
+)
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if delta and delta.content:
+        print(delta.content, end="", flush=True)
+```
+
+Output:
+```text Output
+The first 5 prime numbers are:
+**2, 3, 5, 7, 11**.
+
+### Explanation:
+- A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself.
+- **2** is the smallest and only even prime number.
+- **3** is prime (divisible only by 1 and 3).
+- **4** is not prime (divisible by 2).
+- **5** is prime.
+- **6** is not prime (divisible by 2 and 3).
+- **7** is prime.
+- **8, 9, 10** are not prime.
+- **11** is prime (the fifth in the sequence).
+
+Note: **1 is not considered a prime number** by definition, as it has only one positive divisor.
+This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊
+```
+
+### 4.2 Reasoning
+
+The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+# Reasoning on (default)
+print("Reasoning on")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."}
+    ],
+    temperature=1,
+    max_tokens=1024,
+)
+print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
+print("\n")
+# Reasoning off
+print("Reasoning off")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Give me 3 facts about SGLang."}
+    ],
+    temperature=0,
+    max_tokens=256,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...")
+```
+
+Output:
+```text Output
+Reasoning on
+Reasoning:
+We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs.
+
+Let's deci...
+Content:
+Silicon hearts beat
+Paint vivid worlds with bright light
+GPU dreams rise...
+
+Reasoning off
+Content:
+Certainly! Here are three accurate and informative facts about **SGLang**:
+
+1. **SGLang is a high-performance serving system for large language models (LLMs)**
+   Developed by researchers at UC Berk...
+```
+
+### 4.3 Tool Calling
+
+Call functions using the OpenAI Tools schema and inspect returned `tool_calls`.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+# Tool calling via OpenAI tools schema
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "calculate_tip",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "bill_total": {
+                        "type": "integer",
+                        "description": "The total amount of the bill"
+                    },
+                    "tip_percentage": {
+                        "type": "integer",
+                        "description": "The percentage of tip to be applied"
+                    }
+                },
+                "required": ["bill_total", "tip_percentage"]
+            }
+        }
+    }
+]
+
+completion = client.chat.completions.create(
+    model="nemotron",
+    messages=[
+        {"role": "system", "content": ""},
+        {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
+    ],
+    tools=TOOLS,
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=512,
+    stream=False
+)
+
+print(completion.choices[0].message.reasoning_content)
+print(completion.choices[0].message.tool_calls)
+```
+
+Output:
+```text Output
+The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that.
+
+[ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)]
+```
+
+### 4.4 Controlling Reasoning Budget
+
+The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.
+
+If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.
+
+```python Example
+from typing import Any, Dict, List
+import openai
+from transformers import AutoTokenizer
+
+class ThinkingBudgetClient:
+    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
+        self.base_url = base_url
+        self.api_key = api_key
+        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
+        self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)
+
+    def chat_completion(
+        self,
+        model: str,
+        messages: List[Dict[str, Any]],
+        reasoning_budget: int = 512,
+        max_tokens: int = 1024,
+        **kwargs,
+    ) -> Dict[str, Any]:
+        assert (
+            max_tokens > reasoning_budget
+        ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"
+
+        # 1. first call chat completion to get reasoning content
+        response = self.client.chat.completions.create(
+            model=model,
+            messages=messages,
+            max_tokens=reasoning_budget,
+            **kwargs
+        )
+
+        reasoning_content = response.choices[0].message.reasoning_content or ""
+
+        if "</think>" not in reasoning_content:
+            # reasoning content is too long, closed with a period (.)
+            reasoning_content = f"{reasoning_content}.\n</think>\n\n"
+
+        reasoning_tokens_used = len(
+            self.tokenizer.encode(reasoning_content, add_special_tokens=False)
+        )
+        remaining_tokens = max_tokens - reasoning_tokens_used
+
+        assert (
+            remaining_tokens > 0
+        ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."
+
+        # 2. append reasoning content to messages and call completion
+        messages.append({"role": "assistant", "content": reasoning_content})
+        prompt = self.tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            continue_final_message=True,
+        )
+
+        response = self.client.completions.create(
+            model=model,
+            prompt=prompt,
+            max_tokens=remaining_tokens,
+            **kwargs
+        )
+
+        response_data = {
+            "reasoning_content": reasoning_content.strip().strip("</think>").strip(),
+            "content": response.choices[0].text,
+            "finish_reason": response.choices[0].finish_reason,
+        }
+        return response_data
+```
+
+Usage example with `reasoning_budget=128`:
+
+```python Example
+SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"
+
+# Client
+client = ThinkingBudgetClient(
+    base_url="http://127.0.0.1:5000/v1",
+    api_key="null",
+    tokenizer_name_or_path=SERVED_MODEL_NAME
+)
+
+resp = client.chat_completion(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs."}
+    ],
+    temperature=1,
+    max_tokens=512,
+    reasoning_budget=128
+)
+print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
+```
+
+Output:
+```text Output
+Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact.
+
+First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once.
+
+Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor.
+Content:
+
+Silicon minds awaken,
+Thousands of cores hum in unison—
+Lightning paints the void.
+```
+
+---
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+- Hardware: H200 (4x)
+- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --trust-remote-code \
+  --tp 4 \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --port 5000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 5000 \
+  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  623.49
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total generated tokens:                  2087288
+Total generated tokens (retokenized):    2044666
+Request throughput (req/s):              6.57
+Input token throughput (tok/s):          3338.85
+Output token throughput (tok/s):         3347.77
+Peak output token throughput (tok/s):    6349.00
+Peak concurrent requests:                270
+Total token throughput (tok/s):          6686.62
+Concurrency:                             250.35
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   38108.46
+Median E2E Latency (ms):                 37186.80
+P90 E2E Latency (ms):                    69325.24
+P99 E2E Latency (ms):                    77776.90
+---------------Time to First Token----------------
+Mean TTFT (ms):                          436.49
+Median TTFT (ms):                        114.90
+P99 TTFT (ms):                           6938.11
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          75.02
+Median TPOT (ms):                        76.02
+P99 TPOT (ms):                           92.27
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           74.07
+Median ITL (ms):                         38.45
+P95 ITL (ms):                            230.42
+P99 ITL (ms):                            242.70
+Max ITL (ms):                            7181.72
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+**Environment**
+- Hardware: H200 (4x)
+- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+**Launch Model**
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --trust-remote-code \
+  --tp 4 \
+  --reasoning-parser nemotron_3
+```
+
+**Run Benchmark**
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --port 5000
+```
+
+**Test Results:**
+```text Output
+Accuracy: 0.950
+Invalid: 0.000
+Latency: 21.442 s
+Output throughput: 996.815 token/s
+```
+
+#### 5.2.2 MMLU Benchmark
+
+**Run Benchmark**
+```bash Command
+python3 benchmark/mmlu/bench_sglang.py --port 5000
+```
+
+**Test Results:**
+```text Output
+subject: abstract_algebra, #q:100, acc: 0.730
+subject: anatomy, #q:135, acc: 0.830
+subject: astronomy, #q:152, acc: 0.934
+subject: business_ethics, #q:100, acc: 0.830
+subject: clinical_knowledge, #q:265, acc: 0.879
+subject: college_biology, #q:144, acc: 0.931
+subject: college_chemistry, #q:100, acc: 0.620
+subject: college_computer_science, #q:100, acc: 0.840
+subject: college_mathematics, #q:100, acc: 0.820
+subject: college_medicine, #q:173, acc: 0.821
+subject: college_physics, #q:102, acc: 0.794
+subject: computer_security, #q:100, acc: 0.880
+subject: conceptual_physics, #q:235, acc: 0.919
+subject: econometrics, #q:114, acc: 0.746
+subject: electrical_engineering, #q:145, acc: 0.828
+subject: elementary_mathematics, #q:378, acc: 0.926
+subject: formal_logic, #q:126, acc: 0.857
+subject: global_facts, #q:100, acc: 0.570
+subject: high_school_biology, #q:310, acc: 0.952
+subject: high_school_chemistry, #q:203, acc: 0.828
+subject: high_school_computer_science, #q:100, acc: 0.940
+subject: high_school_european_history, #q:165, acc: 0.861
+subject: high_school_geography, #q:198, acc: 0.939
+subject: high_school_government_and_politics, #q:193, acc: 0.990
+subject: high_school_macroeconomics, #q:390, acc: 0.928
+subject: high_school_mathematics, #q:270, acc: 0.700
+subject: high_school_microeconomics, #q:238, acc: 0.966
+subject: high_school_physics, #q:151, acc: 0.834
+subject: high_school_psychology, #q:545, acc: 0.960
+subject: high_school_statistics, #q:216, acc: 0.852
+subject: high_school_us_history, #q:204, acc: 0.926
+subject: high_school_world_history, #q:237, acc: 0.937
+subject: human_aging, #q:223, acc: 0.879
+subject: human_sexuality, #q:131, acc: 0.939
+subject: international_law, #q:121, acc: 0.934
+subject: jurisprudence, #q:108, acc: 0.898
+subject: logical_fallacies, #q:163, acc: 0.914
+subject: machine_learning, #q:112, acc: 0.821
+subject: management, #q:103, acc: 0.903
+subject: marketing, #q:234, acc: 0.944
+subject: medical_genetics, #q:100, acc: 0.980
+subject: miscellaneous, #q:783, acc: 0.945
+subject: moral_disputes, #q:346, acc: 0.861
+subject: moral_scenarios, #q:895, acc: 0.542
+subject: nutrition, #q:306, acc: 0.902
+subject: philosophy, #q:311, acc: 0.884
+subject: prehistory, #q:324, acc: 0.920
+subject: professional_accounting, #q:282, acc: 0.805
+subject: professional_law, #q:1534, acc: 0.681
+subject: professional_medicine, #q:272, acc: 0.923
+subject: professional_psychology, #q:612, acc: 0.889
+subject: public_relations, #q:110, acc: 0.800
+subject: security_studies, #q:245, acc: 0.837
+subject: sociology, #q:201, acc: 0.960
+subject: us_foreign_policy, #q:100, acc: 0.920
+subject: virology, #q:166, acc: 0.590
+subject: world_religions, #q:171, acc: 0.906
+Total latency: 150.267
+Average accuracy: 0.841
+```
@@ -0,0 +1,554 @@
+---
+title: NVIDIA Nemotron3-Ultra
+description: "Deploy NVIDIA Nemotron3-Ultra with SGLang - 550B hybrid MoE model (55B active) with 1M context window, BF16/NVFP4 support, built for long-running autonomous agents."
+tag:
+    NEW
+---
+
+import { Nemotron3UltraDeployment } from '/src/snippets/autoregressive/nemotron3-ultra-deployment.jsx';
+
+## 1. Model Introduction
+
+`NVIDIA Nemotron3-Ultra` is an open frontier reasoning model in the Nemotron 3 family, built for long-running autonomous agents. It is optimized for complex orchestration across coding, deep research, enterprise workflows, and EDA use cases where agents must sustain reasoning across many steps and large context windows.
+
+Nemotron 3 Ultra is a 550B parameter hybrid MoE model that activates only 55B parameters per forward pass, delivering frontier reasoning accuracy with high-throughput inference. It supports a 1M token context window so agents can keep conversation history, tool outputs, and plan state in view across persistent workflows.
+
+Architecture and key features:
+
+- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
+- **Long-horizon agentic reasoning:** Tuned for agents that plan, call tools, inspect results, recover from failures, and continue working across long task horizons — coding, deep research, enterprise automation, and EDA.
+- **1M token context window:** Sustains coherent agent state across extended workflows without re-ingestion.
+- **BF16 and NVFP4 quantization:** Deployable from multi-node H100 down to a single Blackwell node with NVFP4.
+- **Multi-environment RL post-training:** Post-trained with reinforcement learning across multiple environments for robust reasoning and reliable agentic behavior.
+- **Open weights, open data, open recipes:** Customizable for domain-specific agents and deployable across your own infrastructure.
+
+**Modalities:** Input: text — Output: text
+
+**Supported GPUs:**
+- **BF16:** 16×H100, 16×H200, 8×B200/B300
+- **NVFP4:** 4/8×B200/B300, 4×GB200/GB300
+
+Available model variants on HuggingFace:
+- [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16)
+- [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4)
+
+## 2. SGLang Installation
+
+Nemotron3-Ultra support is included in the latest stable release.
+
+```bash Command
+docker pull lmsysorg/sglang:latest
+```
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance tuning.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: select model precision, hardware, tensor parallelism, and common knobs to generate a launch command.
+
+The generator only emits a runnable command for combinations that NVIDIA / SGLang have validated. Selecting an unverified tuple (e.g. NVFP4 on H100/H200, BF16 with TP=4 on H100, …) is **blocked** — the command pane shows an explicit error and the verified support matrix instead of a launch line, so unvalidated commands can't be copied by accident.
+
+<Nemotron3UltraDeployment />
+
+### 3.2 Configuration Tips
+
+- **Attention backend**:
+
+    **H100/H200**: Use flash attention 3 backend by default.
+    **B200/GB200/B300/GB300**: Set `--attention-backend trtllm_mha`. The flashinfer default breaks the overlap scheduler on Blackwell, so `trtllm_mha` is required there.
+
+- **Mamba scheduler strategy**:
+
+    Always launch with `--mamba-radix-cache-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state.
+
+- **Mamba backend**:
+
+    The Mamba layers use the Triton SSM kernels by default. For better performance, set `--mamba-backend flashinfer` to use the FlashInfer Mamba kernels instead.
+
+- **Mamba SSM precision**:
+
+    The SSM state dtype defaults to the model config value. Set `--mamba-ssm-dtype float16` to store the Mamba states in FP16, which reduces mamba cache memory without significant accuracy loss.
+
+- **Mamba SSM stochastic rounding**:
+
+    When storing the Mamba states in FP16, add `--enable-mamba-cache-stochastic-rounding` to round SSM cache writes stochastically and reduce accumulation bias. It requires `--mamba-ssm-dtype float16` and CUDA; with the default `--mamba-backend triton` it additionally requires SM100. Use `--mamba-cache-philox-rounds` to control the number of Philox rounds (`0` uses the backend default).
+
+- **TP support**:
+
+    To set tp size, use `--tp <4|8|16>`. Recommended pairings:
+    - BF16: `--tp 16` on H100/H200, `--tp 8` on B200/B300
+    - NVFP4: `--tp 4` or `--tp 8` on B200/B300, `--tp 4` on GB200/GB300
+
+- **Multi-node BF16 on H100**:
+
+    The 16×H100 BF16 setup spans two nodes. Use `--dist-init-addr <head-node-ip>:5000 --nnodes 2 --node-rank <0|1>` on each node and keep `--tp 16`.
+
+- **DP attention**:
+
+    By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--dp <N> --enable-dp-attention`) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--dp` must divide `--tp`.
+
+- **Expert parallel (EP)**:
+
+    This MoE only supports `ep_size == 1` (off) or `ep_size == tp_size`. To enable expert parallelism, set `--ep <tp>` with the same value as `--tp`.
+
+- **Multi-token prediction (MTP)**:
+
+    Enable MTP speculative decoding (the toggle above) for low latency.
+
+- **FP8 KV cache**:
+
+    To enable fp8 kv cache, set `--kv-cache-dtype fp8_e4m3`. This is enabled by default on the NVFP4 checkpoint.
+
+- **Reasoning parser**:
+
+    Set `--reasoning-parser nemotron_3` to enable structured reasoning traces (`reasoning_content` field in the response).
+
+- **Tool calling**:
+
+    Set `--tool-call-parser qwen3_coder` to enable tool calling support.
+
+## 4. Model Invocation
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \
+  --trust-remote-code \
+  --tp 8 \
+  --mamba-radix-cache-strategy extra_buffer \
+  --attention-backend trtllm_mha \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser nemotron_3
+```
+
+### 4.1 Basic Usage (OpenAI-Compatible API)
+
+SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
+    ],
+    temperature=0.6,
+    max_tokens=1024,
+)
+print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
+print("\n")
+```
+
+Output:
+```text Output
+Reasoning: The user wants 3 bullet points about SGLang. Let me recall what I know about SGLang — it's a high-performance serving framework for large language models with a focus on structured generation and efficient KV cache reuse...(more tokens)
+
+Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn agent loops and few-shot workloads.
+- **OpenAI-compatible API and structured generation** — Drop-in replacement for the OpenAI client, with first-class support for constrained decoding (JSON schema, regex) and OpenAI-style tool calling.
+- **High-throughput serving on NVIDIA GPUs** — Continuous batching, chunked prefill, FP8/NVFP4 quantization, and optimized CUDA kernels deliver state-of-the-art throughput across H100, H200, B200, and GB200.
+```
+
+Streaming chat completion:
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "What are the first 5 prime numbers?"}
+    ],
+    temperature=0.7,
+    max_tokens=1024,
+    stream=True,
+)
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if delta and delta.content:
+        print(delta.content, end="", flush=True)
+```
+
+Output:
+```text Output
+The first 5 prime numbers are:
+**2, 3, 5, 7, 11**.
+
+### Explanation:
+- A **prime number** is a natural number greater than 1 whose only positive divisors are 1 and itself.
+- **2** is the smallest prime and the only even prime.
+- **3, 5, 7, 11** are each divisible only by 1 and themselves.
+- **1** is not prime by definition (it has only one positive divisor).
+- **4, 6, 8, 9, 10** are composite.
+```
+
+### 4.2 Reasoning
+
+The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+# Reasoning on (default)
+print("Reasoning on")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Plan a 3-step approach to debug a flaky integration test. Keep the thinking process short."}
+    ],
+    temperature=1,
+    max_tokens=1024,
+)
+print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
+print("\n")
+# Reasoning off
+print("Reasoning off")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Give me 3 facts about SGLang."}
+    ],
+    temperature=0,
+    max_tokens=256,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+print(f"Content: \n{resp.choices[0].message.content[:200]}...")
+```
+
+Output:
+```text Output
+Reasoning on
+Reasoning:
+The user wants a short reasoning chain plus a 3-step debug plan for a flaky integration test. I'll think briefly about common causes (timing/race, shared state, external service variance) and pick a t...
+Content:
+1. **Reproduce deterministically** — run the test in a loop (e.g. 50–100x) with logging at the suspected race points to confirm the failure rate and surface ordering.
+2. **Isolate state** — re-run with...
+
+Reasoning off
+Content:
+Here are 3 facts about SGLang:
+
+1. **High-performance LLM serving system** developed at UC Berkeley with contributions from a broad open-source community, focused on throughput and latency at scale.
+...
+```
+
+### 4.3 Tool Calling
+
+Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. The server must be launched with `--tool-call-parser qwen3_coder`.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+# Tool calling via OpenAI tools schema
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "search_codebase",
+            "description": "Search the project codebase for a symbol or pattern.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "query": {
+                        "type": "string",
+                        "description": "The symbol, function name, or regex to search for"
+                    },
+                    "path": {
+                        "type": "string",
+                        "description": "Optional sub-path to restrict the search to"
+                    }
+                },
+                "required": ["query"]
+            }
+        }
+    }
+]
+
+completion = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a coding agent. Use tools to inspect the repo before answering."},
+        {"role": "user", "content": "Where is the `RadixCache` class defined?"}
+    ],
+    tools=TOOLS,
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=512,
+    stream=False
+)
+
+print(completion.choices[0].message.reasoning_content)
+print(completion.choices[0].message.tool_calls)
+```
+
+Output:
+```text Output
+The user is asking where the RadixCache class is defined. I should search the codebase for the symbol "RadixCache" to find the file and line. I'll call search_codebase with that query.
+
+[ChatCompletionMessageFunctionToolCall(id='call_8a7f2c4e1b9d4a3e8c2f1d6b', function=Function(arguments='{"query": "class RadixCache"}', name='search_codebase'), type='function', index=0)]
+```
+
+### 4.4 Controlling Reasoning Budget
+
+The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.
+
+If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.
+
+```python Example
+from typing import Any, Dict, List
+import openai
+from transformers import AutoTokenizer
+
+class ThinkingBudgetClient:
+    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
+        self.base_url = base_url
+        self.api_key = api_key
+        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
+        self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)
+
+    def chat_completion(
+        self,
+        model: str,
+        messages: List[Dict[str, Any]],
+        reasoning_budget: int = 512,
+        max_tokens: int = 1024,
+        **kwargs,
+    ) -> Dict[str, Any]:
+        assert (
+            max_tokens > reasoning_budget
+        ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"
+
+        # 1. first call chat completion to get reasoning content
+        response = self.client.chat.completions.create(
+            model=model,
+            messages=messages,
+            max_tokens=reasoning_budget,
+            **kwargs
+        )
+
+        reasoning_content = response.choices[0].message.reasoning_content or ""
+
+        if "</think>" not in reasoning_content:
+            # reasoning content is too long, closed with a period (.)
+            reasoning_content = f"{reasoning_content}.\n</think>\n\n"
+
+        reasoning_tokens_used = len(
+            self.tokenizer.encode(reasoning_content, add_special_tokens=False)
+        )
+        remaining_tokens = max_tokens - reasoning_tokens_used
+
+        assert (
+            remaining_tokens > 0
+        ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."
+
+        # 2. append reasoning content to messages and call completion
+        messages.append({"role": "assistant", "content": reasoning_content})
+        prompt = self.tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            continue_final_message=True,
+        )
+
+        response = self.client.completions.create(
+            model=model,
+            prompt=prompt,
+            max_tokens=remaining_tokens,
+            **kwargs
+        )
+
+        response_data = {
+            "reasoning_content": reasoning_content.strip().strip("</think>").strip(),
+            "content": response.choices[0].text,
+            "finish_reason": response.choices[0].finish_reason,
+        }
+        return response_data
+```
+
+Usage example with `reasoning_budget=256`:
+
+```python Example
+SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"
+
+# Client
+client = ThinkingBudgetClient(
+    base_url="http://127.0.0.1:30000/v1",
+    api_key="null",
+    tokenizer_name_or_path=SERVED_MODEL_NAME
+)
+
+resp = client.chat_completion(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Outline a research plan to evaluate the throughput of two MoE serving strategies."}
+    ],
+    temperature=1,
+    max_tokens=1024,
+    reasoning_budget=256
+)
+print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
+```
+
+Output:
+```text Output
+Reasoning: The user wants a research plan to compare throughput of two MoE serving strategies. I should outline goals, baselines, datasets, metrics (tokens/s, TTFT, ITL, MFU), variables to sweep (TP, batch size, sequence length, concurrency), and statistical handling. Keep it concise since reasoning_budget is 256...
+Content:
+
+**Research plan**
+
+1. **Define goal & metrics** — peak token throughput (input+output), TTFT, P99 ITL, MFU; measured at fixed accuracy.
+2. **Choose baselines** — Strategy A (TP-only) vs Strategy B (TP + expert-parallel). Hold model checkpoint, precision, and KV-cache dtype constant.
+3. **Sweep** — `{batch ∈ 1,4,16,64, concurrency ∈ 16,64,256, seq_len ∈ 1k,8k,32k}` per strategy.
+4. **Workload** — `sglang.bench_serving --dataset-name random` with matched input/output budgets.
+5. **Analysis** — per-config throughput table + roofline overlay; bootstrap CIs over 3 reruns to bound noise.
+```
+
+---
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+- Hardware: GB200 (4x)
+- Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
+  --trust-remote-code \
+  --tp 4 \
+  --mamba-radix-cache-strategy extra_buffer \
+  --attention-backend trtllm_mha \
+  --max-running-requests 1024
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  1184.58
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total generated tokens:                  2087288
+Total generated tokens (retokenized):    1990224
+Request throughput (req/s):              3.46
+Input token throughput (tok/s):          1757.35
+Output token throughput (tok/s):         1762.05
+Peak output token throughput (tok/s):    3150.00
+Peak concurrent requests:                266
+Total token throughput (tok/s):          3519.40
+Concurrency:                             249.55
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   72169.95
+Median E2E Latency (ms):                 71994.47
+P90 E2E Latency (ms):                    99898.56
+P99 E2E Latency (ms):                    107119.61
+---------------Time to First Token----------------
+Mean TTFT (ms):                          40057.33
+Median TTFT (ms):                        41375.93
+P99 TTFT (ms):                           46377.89
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          63.15
+Median TPOT (ms):                        63.65
+P99 TPOT (ms):                           78.16
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           63.14
+Median ITL (ms):                         35.92
+P95 ITL (ms):                            178.10
+P99 ITL (ms):                            182.10
+Max ITL (ms):                            2466.36
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+**Environment**
+- Hardware: GB200 (4x)
+- Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+**Launch Model**
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
+  --trust-remote-code \
+  --tp 4 \
+  --mamba-radix-cache-strategy extra_buffer \
+  --attention-backend trtllm_mha \
+  --reasoning-parser nemotron_3
+```
+
+**Run Benchmark**
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py
+```
+
+**Test Results:**
+```text Output
+Accuracy: 0.970
+Invalid: 0.000
+Latency: 29.129 s
+Output throughput: 745.333 token/s
+```
+
+#### 5.2.2 MMLU Benchmark
+
+**Run Benchmark**
+```bash Command
+python3 benchmark/mmlu/bench_sglang.py
+```
+
+**Test Results:**
+```text Output
+TBD
+```
+
+---