docs(cookbook): add IBM Granite 4.2 cookbook (#36286)

2026-08-25 22:54:55 +08:00
parent 46d9427b91
commit b760f7fb19
6 changed files with 545 additions and 0 deletions
@@ -0,0 +1,231 @@
+---
+title: Granite 4.2
+description: "Deploy Granite 4.2 3B, 8B, and 30B dense models with SGLang on NVIDIA H200 and B200, including thinking modes and structured tool calling."
+tag: NEW
+---
+
+## Deployment
+
+<a id="install" />
+
+<Accordion title="Install SGLang">
+
+For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
+
+<Tabs>
+
+<Tab title="Python (pip / uv)">
+
+```bash Command
+pip install --upgrade pip
+pip install uv
+uv pip install --prerelease=allow sglang
+```
+
+Then run the **Python** output of the command panel below in that environment.
+
+</Tab>
+
+<Tab title="Docker">
+
+```bash Command
+docker pull lmsysorg/sglang:dev
+```
+
+For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
+
+</Tab>
+
+</Tabs>
+
+</Accordion>
+
+Pick a Granite 4.2 checkpoint to generate the launch command. The verified matrix covers BF16 serving on one NVIDIA H200 or B200 with tensor parallelism 1.
+
+import { Deployment } from "/src/snippets/_deployment.jsx";
+import { config } from "/src/snippets/configs/ibm-granite/granite-4.2.jsx";
+import { benchmarks } from "/src/snippets/configs/ibm-granite/granite-4.2-benchmarks.jsx";
+
+<Deployment config={config} benchmarks={benchmarks} />
+
+<Note>
+  The H200 speed results use `lmsysorg/sglang:dev` at SGLang commit `d59c1ddf7` and the B200 results at commit `d10a656ad8`; the launch recipes were verified end to end on both GPUs against the release checkpoints. Each speed point uses 80 fixed-length random requests at 8,192 input and 1,024 output tokens, 8 warmup requests, a flushed cache, greedy sampling, and ignore-EOS.
+</Note>
+
+## Playground
+
+The Playground layers SGLang features on top of the verified recipe. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end.
+
+import { Playground } from "/src/snippets/_playground.jsx";
+
+<Playground config={config} />
+
+## 1. Model introduction
+
+**Granite 4.2** is IBM's dense decoder-only language model family with 3B, 8B, and 30B checkpoints. Each checkpoint uses BF16 weights, has a configured context length of 131,072 tokens, and supports default thinking, non-thinking, low-effort thinking, and structured tool calls through its chat template. The repositories declare the Apache-2.0 license.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Variant</th>
+      <th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Total params</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Position in family</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/ibm-granite/granite-4.2-3b">Granite 4.2 3B</a></strong></td>
+      <td style={{padding: "9px 12px", textAlign: "right"}}>3B</td>
+      <td style={{padding: "9px 12px"}}>Smallest checkpoint</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/ibm-granite/granite-4.2-8b">Granite 4.2 8B</a></strong></td>
+      <td style={{padding: "9px 12px", textAlign: "right"}}>8B</td>
+      <td style={{padding: "9px 12px"}}>Mid-size checkpoint</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/ibm-granite/granite-4.2-30b">Granite 4.2 30B</a></strong></td>
+      <td style={{padding: "9px 12px", textAlign: "right"}}>30B</td>
+      <td style={{padding: "9px 12px"}}>Largest checkpoint</td>
+    </tr>
+  </tbody>
+</table>
+
+**Recommended generation:** IBM recommends `temperature=1.0` and `top_p=0.95` for general chat, reasoning, and tool calling. The release checkpoints ship these values in `generation_config.json`; send them per request when you want to be explicit.
+
+**Resources:** [Granite 4.2 3B](https://huggingface.co/ibm-granite/granite-4.2-3b) · [Granite 4.2 8B](https://huggingface.co/ibm-granite/granite-4.2-8b) · [Granite 4.2 30B](https://huggingface.co/ibm-granite/granite-4.2-30b).
+
+## 2. Configuration tips
+
+- **Thinking is enabled by default.** Set `chat_template_kwargs.enable_thinking` to `false` for a direct answer. Set `enable_thinking` and `low_effort` to `true` for a shorter reasoning trace.
+- **Give thinking enough tokens.** At `temperature=1.0` the default thinking mode can run past 1,000 tokens on multi-step problems. Use a `max_tokens` of at least 2,048 for thinking requests so the answer is not cut off.
+- **Reasoning parser.** Launch with `--reasoning-parser nemotron_3` so OpenAI-compatible responses separate the trace into `message.reasoning_content` and the answer into `message.content`. Without a parser flag the reasoning markup stays inline in `message.content`. `--reasoning-parser auto` also resolves to `nemotron_3` for these checkpoints.
+- **Tool-call parser.** Launch with `--tool-call-parser qwen3_coder` so tool requests are returned through `message.tool_calls`. Without it, raw `<tool_call>` markup stays in `message.content`. `--tool-call-parser auto` also resolves to `qwen3_coder` for these checkpoints.
+- **Single-GPU sizing.** All three BF16 checkpoints loaded and completed chat requests with `--tp 1 --mem-fraction-static 0.8` on one H200 and on one B200. Increase TP only after validating the new topology.
+- **Image selection.** The validated path uses `lmsysorg/sglang:dev`. A stable image tested during validation had an incompatible dependency set before model loading, so use the recipe's image until a newer tagged release is confirmed.
+
+## 3. Advanced usage
+
+The outputs below are verbatim captures from Granite 4.2 3B on the verified server. Sampling is stochastic, so a repeated request can produce different wording.
+
+### 3.1 Thinking modes
+
+The `nemotron_3` reasoning parser keeps reasoning and final content in separate fields. Granite 4.2 accepts three chat-template modes: default thinking, non-thinking, and low-effort thinking.
+
+<Accordion title="Thinking modes example (Python)">
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+model = "ibm-granite/granite-4.2-3b"
+
+modes = {
+    "thinking": {"enable_thinking": True},
+    "non-thinking": {"enable_thinking": False},
+    "low-effort": {"enable_thinking": True, "low_effort": True},
+}
+
+for name, chat_template_kwargs in modes.items():
+    response = client.chat.completions.create(
+        model=model,
+        messages=[
+            {"role": "user", "content": "What is 17 * 23? Answer briefly."}
+        ],
+        extra_body={"chat_template_kwargs": chat_template_kwargs},
+    )
+    message = response.choices[0].message
+    print(f"[{name}]")
+    print("Reasoning:", getattr(message, "reasoning_content", None))
+    print("Answer:", message.content)
+```
+
+</Accordion>
+
+<Accordion title="Example output">
+
+```text Output
+[thinking]
+Reasoning: Okay, the user asked "What is 17 * 23? Answer briefly." I need to calculate 17 multiplied by 23.
+
+Let me do the multiplication. 17 times 23.
+
+I can break it down: 17 * 20 = 340, and 17 * 3 = 51. Then add them: 340 + 51 = 391.
+
+Alternatively, 23 * 17: 23*10=230, 23*7=161, 230+161=391. Same result.
+
+So the answer is 391.
+
+The user wants a brief answer, so just state the number.
+
+Answer:
+391
+[non-thinking]
+Reasoning: None
+Answer: 391
+[low-effort]
+Reasoning: Compute 17*23 = 17*20=340, plus 17*3=51 => 391.
+
+Answer:
+391
+```
+
+</Accordion>
+
+### 3.2 Tool calling
+
+The `qwen3_coder` parser converts the model's tool markup into OpenAI-compatible structured calls.
+
+<Accordion title="Tool calling example (Python)">
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+tools = [{
+    "type": "function",
+    "function": {
+        "name": "get_weather",
+        "description": "Get the current weather for a city.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "city": {"type": "string", "description": "The city name"},
+            },
+            "required": ["city"],
+        },
+    },
+}]
+
+response = client.chat.completions.create(
+    model="ibm-granite/granite-4.2-3b",
+    messages=[{"role": "user", "content": "What is the weather in Boston right now?"}],
+    tools=tools,
+    tool_choice="auto",
+)
+
+choice = response.choices[0]
+message = choice.message
+print("Reasoning:", getattr(message, "reasoning_content", None))
+print("Content:", message.content)
+for call in message.tool_calls or []:
+    print("Tool:", call.function.name)
+    print("Arguments:", call.function.arguments)
+print("Finish reason:", choice.finish_reason)
+```
+
+</Accordion>
+
+<Accordion title="Example output">
+
+```text Output
+Reasoning: Okay, the user is asking for the weather in Boston right now. I need to use the available tool called get_weather. The tool requires the city parameter. Since the user specified Boston, I'll call get_weather with city set to Boston.
+
+Content: None
+Tool: get_weather
+Arguments: {"city": "Boston"}
+Finish reason: tool_calls
+```
+
+</Accordion>
@@ -61,6 +61,12 @@ metatags:
    href="/cookbook/autoregressive/Google/Gemma4"
    img="/cards/logos/google.png"
  />
+  <Card
+    title="IBM"
+    mode="card"
+    href="/cookbook/autoregressive/IBM/Granite-4.2"
+    img="/cards/logos/ibm.png"
+  />
  <Card
    title="LiquidAI"
    mode="card"
@@ -1302,6 +1302,12 @@
                      "cookbook/autoregressive/Google/DiffusionGemma"
                    ]
                  },
+                  {
+                    "group": "IBM",
+                    "pages": [
+                      "cookbook/autoregressive/IBM/Granite-4.2"
+                    ]
+                  },
                  {
                    "group": "LiquidAI",
                    "pages": [
@@ -0,0 +1,110 @@
+export const benchmarks = [
+  {
+    match: { hw: "h200", variant: "3b", quant: "bf16", strategy: "balanced", nodes: "single" },
+    sglang_version: "d59c1ddf7",
+    speed: [
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: 131.17,
+        tpot_ms: 3.51,
+        tokens_per_sec_per_gpu: 2472.96,
+      },
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: 1092.54,
+        tpot_ms: 7.23,
+        tokens_per_sec_per_gpu: 17416.41,
+      },
+    ],
+  },
+  {
+    match: { hw: "h200", variant: "8b", quant: "bf16", strategy: "balanced", nodes: "single" },
+    sglang_version: "d59c1ddf7",
+    speed: [
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: 260.59,
+        tpot_ms: 6.28,
+        tokens_per_sec_per_gpu: 1378.86,
+      },
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: 2168.05,
+        tpot_ms: 13.06,
+        tokens_per_sec_per_gpu: 9484.18,
+      },
+    ],
+  },
+  {
+    match: { hw: "h200", variant: "30b", quant: "bf16", strategy: "balanced", nodes: "single" },
+    sglang_version: "d59c1ddf7",
+    speed: [
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: 807.38,
+        tpot_ms: 17.14,
+        tokens_per_sec_per_gpu: 502.74,
+      },
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: 6860.18,
+        tpot_ms: 31.19,
+        tokens_per_sec_per_gpu: 3803.83,
+      },
+    ],
+  },
+  {
+    match: { hw: "b200", variant: "3b", quant: "bf16", strategy: "balanced", nodes: "single" },
+    sglang_version: "d10a656ad8",
+    speed: [
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: 74.93,
+        tpot_ms: 2.76,
+        tokens_per_sec_per_gpu: 3179.78,
+      },
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: 559.10,
+        tpot_ms: 4.66,
+        tokens_per_sec_per_gpu: 27563.95,
+      },
+    ],
+  },
+  {
+    match: { hw: "b200", variant: "8b", quant: "bf16", strategy: "balanced", nodes: "single" },
+    sglang_version: "d10a656ad8",
+    speed: [
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: 132.85,
+        tpot_ms: 4.44,
+        tokens_per_sec_per_gpu: 1969.35,
+      },
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: 1060.55,
+        tpot_ms: 8.08,
+        tokens_per_sec_per_gpu: 15800.59,
+      },
+    ],
+  },
+  {
+    match: { hw: "b200", variant: "30b", quant: "bf16", strategy: "balanced", nodes: "single" },
+    sglang_version: "d10a656ad8",
+    speed: [
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: 386.03,
+        tpot_ms: 12.01,
+        tokens_per_sec_per_gpu: 727.24,
+      },
+      {
+        workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: 3391.21,
+        tpot_ms: 18.78,
+        tokens_per_sec_per_gpu: 6522.86,
+      },
+    ],
+  },
+];
@@ -0,0 +1,192 @@
+export const config = {
+  modelName: "Granite 4.2",
+
+  latencyPercentile: "P50",
+
+  supportedHardware: ["h200", "b200"],
+
+  variants: [
+    { id: "3b", label: "3B", subtitle: "Dense" },
+    { id: "8b", label: "8B", subtitle: "Dense" },
+    { id: "30b", label: "30B", subtitle: "Dense" },
+  ],
+  quantizations: [
+    { id: "bf16", label: "BF16" },
+  ],
+  strategies: [
+    { id: "balanced", label: "Balanced" },
+  ],
+  nodesOptions: [
+    { id: "single", label: "Single Node" },
+  ],
+
+  modelNames: {
+    "3b|bf16": "ibm-granite/granite-4.2-3b",
+    "8b|bf16": "ibm-granite/granite-4.2-8b",
+    "30b|bf16": "ibm-granite/granite-4.2-30b",
+  },
+
+  placeholders: {
+    HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
+    PORT: { target: "command", label: "Bind port", default: "30000" },
+    CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
+    CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
+  },
+
+  curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
+-H 'Content-Type: application/json' \\
+-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
+
+  benchmarkCommands: {
+    speed:
+`python -m sglang.benchmark.serving \\
+  --backend sglang-oai \\
+  --host {{CURL_HOST}} --port {{CURL_PORT}} \\
+  --model {{MODEL_NAME}} --tokenizer {{MODEL_NAME}} \\
+  --dataset-name {{DATASET}} \\
+  --random-input-len {{ISL}} --random-output-len {{OSL}} \\
+  --random-range-ratio 1.0 \\
+  --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\
+  --warmup-requests 8 --flush-cache \\
+  --temperature 0.0 --top-p 1.0 \\
+  --seed 123 --disable-tqdm --output-details`,
+    numPromptsByConc: { 1: 80, 16: 80 },
+  },
+
+  dockerImages: {
+    h200: "lmsysorg/sglang:dev",
+    b200: "lmsysorg/sglang:dev",
+  },
+
+  github: {
+    cookbookModel: "ibm-granite/granite-4.2-3b",
+  },
+
+  playgroundFeatures: {
+    attention: {
+      knobs: [
+        { id: "tp", label: "TP", values: [null, 1, 2, 4, 8] },
+      ],
+    },
+    parsers: {
+      items: [
+        { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser nemotron_3" },
+        { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser qwen3_coder" },
+      ],
+    },
+    pdDisagg: {
+      modes: [
+        { id: "off", label: "Off" },
+        { id: "prefill", label: "Prefill role" },
+        { id: "decode", label: "Decode role" },
+      ],
+      transferBackends: [
+        { id: "mooncake", label: "Mooncake" },
+        { id: "nixl", label: "NiXL" },
+      ],
+      ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
+    },
+    hicache: {
+      backends: [
+        { id: null, label: "Auto" },
+        { id: "file", label: "File" },
+        { id: "mooncake", label: "Mooncake" },
+        { id: "hf3fs", label: "HF3FS" },
+        { id: "nixl", label: "NiXL" },
+      ],
+      writePolicies: [
+        { id: "auto", label: "Auto" },
+        { id: "write_through", label: "Write-through" },
+        { id: "write_back", label: "Write-back" },
+        { id: "write_through_selective", label: "Write-through (selective)" },
+      ],
+    },
+  },
+
+  cells: [
+    {
+      match: { hw: "h200", variant: "3b", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true,
+      env: [],
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--tp 1",
+        "--mem-fraction-static 0.8",
+        "--reasoning-parser nemotron_3",
+        "--tool-call-parser qwen3_coder",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    {
+      match: { hw: "h200", variant: "8b", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true,
+      env: [],
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--tp 1",
+        "--mem-fraction-static 0.8",
+        "--reasoning-parser nemotron_3",
+        "--tool-call-parser qwen3_coder",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    {
+      match: { hw: "h200", variant: "30b", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true,
+      env: [],
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--tp 1",
+        "--mem-fraction-static 0.8",
+        "--reasoning-parser nemotron_3",
+        "--tool-call-parser qwen3_coder",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    {
+      match: { hw: "b200", variant: "3b", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true,
+      env: [],
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--tp 1",
+        "--mem-fraction-static 0.8",
+        "--reasoning-parser nemotron_3",
+        "--tool-call-parser qwen3_coder",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    {
+      match: { hw: "b200", variant: "8b", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true,
+      env: [],
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--tp 1",
+        "--mem-fraction-static 0.8",
+        "--reasoning-parser nemotron_3",
+        "--tool-call-parser qwen3_coder",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    {
+      match: { hw: "b200", variant: "30b", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true,
+      env: [],
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--tp 1",
+        "--mem-fraction-static 0.8",
+        "--reasoning-parser nemotron_3",
+        "--tool-call-parser qwen3_coder",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+  ],
+};